cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Engineering & Streaming

Forum Posts

Volker
by New Contributor III
  • 1016 Views
  • 4 replies
  • 1 kudos

Asset Bundles cannot run job with single node job cluster

Hello community,we are deploying a job using asset bundles and the job should run on a single node job cluster. Here is the DAB job definition:resources: jobs: example_job: name: example_job tasks: - task_key: main_task ...

  • 1016 Views
  • 4 replies
  • 1 kudos
Latest Reply
Volker
New Contributor III
  • 1 kudos

Sorry for the late reply, this helped, thank you! 

  • 1 kudos
3 More Replies
SamAdams
by New Contributor III
  • 64 Views
  • 1 replies
  • 0 kudos

Redacted check constraint condition in Delta Table

Hello! I have a delta table with a check constraint - it's one of many that a config-driven ETL pipeline of mine generates. When someone edits the config file and deploys the change, I'd like for the check constraint to be updated as well if it's dif...

  • 64 Views
  • 1 replies
  • 0 kudos
Latest Reply
SamAdams
New Contributor III
  • 0 kudos

Figured this out with the help of @SamDataWalk 's post https://community.databricks.com/t5/data-engineering/databricks-bug-with-show-tblproperties-redacted-azure-databricks/m-p/93546It happens because Databricks thinks certain keywords in the constra...

  • 0 kudos
SamDataWalk
by New Contributor III
  • 1795 Views
  • 5 replies
  • 2 kudos

Resolved! Databricks bug with show tblproperties - redacted - Azure databricks

I am struggling to report what is a fairly fundamental bug. Can anyone help? Ideally someone from Databricks themselves. Or others who can confirm they can replicate it.There is a bug where databricks seems to be hiding “any” properties which have th...

  • 1795 Views
  • 5 replies
  • 2 kudos
Latest Reply
SamAdams
New Contributor III
  • 2 kudos

Like your example that redaction behavior seemed to pick up on the column name: a condition that included a column named "URL" was redacted, but one that included a "modifiedDateTime" was not

  • 2 kudos
4 More Replies
DBX123
by New Contributor
  • 82 Views
  • 1 replies
  • 1 kudos

Is it possible to have an alert when a row is added to a table?

I currently have a table that periodically adds rows in (sometimes daily, sometimes over a month). I was hoping to have an alert for when a row is added into this. The table has date fields of when rows are loaded in.I have an alert working, but it j...

  • 82 Views
  • 1 replies
  • 1 kudos
Latest Reply
Walter_C
Honored Contributor
  • 1 kudos

You could set your alert with query as: -- Custom alert to monitor for new rows added to a table SELECT COUNT(*) AS new_rows FROM your_table WHERE event_time > current_timestamp() - interval '1' hour In this example, the query checks for rows added t...

  • 1 kudos
EDDatabricks
by Contributor
  • 1238 Views
  • 2 replies
  • 1 kudos

How to enforce a cleanup policy on job cluster logs

We have a number of jobs on our databricks workspaces. All job clusters are configured with a dbfs location to save the respective logs (configured from Job cluster -> "Advanced options" -> "Logging").However, the logs are retained in the dbfs indefi...

  • 1238 Views
  • 2 replies
  • 1 kudos
Latest Reply
Atanu
Esteemed Contributor
  • 1 kudos

@EDDatabricks Thanks for reaching out to us. I think you should explore our Purge option is enabled or not. https://docs.databricks.com/en/administration-guide/workspace/settings/storage.html

  • 1 kudos
1 More Replies
Sahil0007
by New Contributor
  • 97 Views
  • 1 replies
  • 0 kudos

Access issue in iPad Air m2

I am trying to run databricks community edition in iPad Air m2. It’s showing me blank page while logging, but in my android phone it’s working fine. Is there any compatibility issue with ios ? Please help me

  • 97 Views
  • 1 replies
  • 0 kudos
Latest Reply
Walter_C
Honored Contributor
  • 0 kudos

We have received similar reports which seems to be related to an IOS issue right now, it is important to mention that mobile browsers are not supported by Databricks and we cannot guarantee functionality on them, please refer to https://docs.databric...

  • 0 kudos
merca
by Valued Contributor II
  • 7769 Views
  • 8 replies
  • 4 kudos

Value array {{QUERY_RESULT_ROWS}} in Databricks SQL alerts custom template

Please include in documentation an example how to incorporate the `QUERY_RESULT_ROWS` variable in the custom template.

  • 7769 Views
  • 8 replies
  • 4 kudos
Latest Reply
CJK053000
New Contributor
  • 4 kudos

@merca I have been using this same solution in alerts and it has been working great until yesterday, 10/9/2024. Now the results are showing up blank in my alert notification. Are you experiencing the same?

  • 4 kudos
7 More Replies
dyusuf
by New Contributor
  • 1174 Views
  • 2 replies
  • 2 kudos

Unity Catalog

Can we set up unity catalog on databricks community edition?If yes, please share the process.Thanks

  • 1174 Views
  • 2 replies
  • 2 kudos
Latest Reply
szymon_dybczak
Contributor III
  • 2 kudos

Another option you can try it to setup open source version of unity catalog with apache spark if you don't have possibility to create Azure trial account/

  • 2 kudos
1 More Replies
DataRabbit
by New Contributor II
  • 11530 Views
  • 3 replies
  • 0 kudos

Resolved! py4j.security.Py4JSecurityException: Constructor public org.apache.spark.ml.feature.VectorAssembler(java.lang.String) is not whitelisted.

Hello, i have a problem.When I try to run the MLlib Assembler (from pyspark.ml.feature import VectorAssembler) I get this error and I don't know what to do anymore. Please help.

  • 11530 Views
  • 3 replies
  • 0 kudos
Latest Reply
DBXC
Contributor
  • 0 kudos

How could we whitelist this error below with DBR 13.3 and above? Py4JError: An error occurred while calling None.org.apache.spark.ml.recommendation.ALS. Trace: py4j.security.Py4JSecurityException: Constructor public org.apache.spark.ml.recommendation...

  • 0 kudos
2 More Replies
alejandrofm
by Valued Contributor
  • 2633 Views
  • 2 replies
  • 0 kudos

Can't enable CLI 2.1 on CI

Hi! this is my CI configuration, I added the databricks jobs configure --version=2.1 command but it stills showing this error, any idea of what can I be doing wrong?Error:Resetting Databricks Job with job_id 1036...WARN: Your CLI is configured to use...

  • 2633 Views
  • 2 replies
  • 0 kudos
Latest Reply
karthik-kandiko
New Contributor
  • 0 kudos

any updated on this one, i am having a similar issue 

  • 0 kudos
1 More Replies
kunex
by New Contributor
  • 114 Views
  • 3 replies
  • 2 kudos

Resolved! Asset Bundles Job If Condition

HiI am trying to use Asset Bundles to deploy a job with email notifications active only on production environment.I tried to use such if statement but it does not seem to do anything.resources: jobs: Master_Load: name: Master Load e...

  • 114 Views
  • 3 replies
  • 2 kudos
Latest Reply
kunex
New Contributor
  • 2 kudos

Okay i made it work with a variable, it was pretty easy. The variable accepted multiline text easily.Thanks!databricks.ymltargets: dev: mode: development ... variables: notifications: default: - user1@smth.com ...

  • 2 kudos
2 More Replies
antr
by New Contributor
  • 110 Views
  • 3 replies
  • 0 kudos

What does DLT INITIALIZING phase do?

In Delta Live Tables, the INITIALIZING phase takes sometimes a minute, sometimes 5 minutes. I'd like to learn what is it doing in the background, and can it be optimized in any way.

  • 110 Views
  • 3 replies
  • 0 kudos
Latest Reply
szymon_dybczak
Contributor III
  • 0 kudos

Hi @antr, In Delta Live Tables (DLT), a feature of Databricks, the "Initializing" phase refers to the first step in the lifecycle of a DLT pipeline run. During this phase, the platform sets up the necessary resources, configurations, and dependencies...

  • 0 kudos
2 More Replies
noorbasha534
by New Contributor II
  • 71 Views
  • 1 replies
  • 0 kudos

Databricks as a "pure" data streaming software like Confluent

DearsI was wondering if anyone has leveraged Databricks as a "pure" data streaming software in place of Confluent, Flink, Kafka etc.I see the reference architectures placing Databricks on the data processing side mostly once data is made available by...

  • 71 Views
  • 1 replies
  • 0 kudos
Latest Reply
szymon_dybczak
Contributor III
  • 0 kudos

Hi @noorbasha534 ,It depends on what you're asking for. Kafka is primarily a messaging system, optimized for handling high-throughput, distributed message logs. Databricks can read from Kafka as a data source but doesn't replace Kafka's role in messa...

  • 0 kudos
RyoAriyama
by New Contributor
  • 212 Views
  • 1 replies
  • 0 kudos

Can't create table in unity catalog.

Hi all.I have created a Databricks workspace on AWS. I can log into the workspace and successfully perform select operations on files in S3, but I am unable to create tables.The error when creating the table is as follows. "Your request failed with s...

  • 212 Views
  • 1 replies
  • 0 kudos
Latest Reply
RyoAriyama
New Contributor
  • 0 kudos

I attach the screenshot when I encounter the error.  

  • 0 kudos

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels