- 1016 Views
- 4 replies
- 1 kudos
Asset Bundles cannot run job with single node job cluster
Hello community,we are deploying a job using asset bundles and the job should run on a single node job cluster. Here is the DAB job definition:resources: jobs: example_job: name: example_job tasks: - task_key: main_task ...
- 1016 Views
- 4 replies
- 1 kudos
- 64 Views
- 1 replies
- 0 kudos
Redacted check constraint condition in Delta Table
Hello! I have a delta table with a check constraint - it's one of many that a config-driven ETL pipeline of mine generates. When someone edits the config file and deploys the change, I'd like for the check constraint to be updated as well if it's dif...
- 64 Views
- 1 replies
- 0 kudos
- 0 kudos
Figured this out with the help of @SamDataWalk 's post https://community.databricks.com/t5/data-engineering/databricks-bug-with-show-tblproperties-redacted-azure-databricks/m-p/93546It happens because Databricks thinks certain keywords in the constra...
- 0 kudos
- 1795 Views
- 5 replies
- 2 kudos
Resolved! Databricks bug with show tblproperties - redacted - Azure databricks
I am struggling to report what is a fairly fundamental bug. Can anyone help? Ideally someone from Databricks themselves. Or others who can confirm they can replicate it.There is a bug where databricks seems to be hiding “any” properties which have th...
- 1795 Views
- 5 replies
- 2 kudos
- 2 kudos
Like your example that redaction behavior seemed to pick up on the column name: a condition that included a column named "URL" was redacted, but one that included a "modifiedDateTime" was not
- 2 kudos
- 43 Views
- 0 replies
- 0 kudos
Changing git's author field when committing through Databricks
I have a git folder to a Bitbucket repo. Whenever I commit something, the commit uses my Bitbucket username (the unique name) in the field author, making it less readable when I'm reading a list of commits.For example, commits end up like this: commi...
- 43 Views
- 0 replies
- 0 kudos
- 82 Views
- 1 replies
- 1 kudos
Is it possible to have an alert when a row is added to a table?
I currently have a table that periodically adds rows in (sometimes daily, sometimes over a month). I was hoping to have an alert for when a row is added into this. The table has date fields of when rows are loaded in.I have an alert working, but it j...
- 82 Views
- 1 replies
- 1 kudos
- 1 kudos
You could set your alert with query as: -- Custom alert to monitor for new rows added to a table SELECT COUNT(*) AS new_rows FROM your_table WHERE event_time > current_timestamp() - interval '1' hour In this example, the query checks for rows added t...
- 1 kudos
- 1238 Views
- 2 replies
- 1 kudos
How to enforce a cleanup policy on job cluster logs
We have a number of jobs on our databricks workspaces. All job clusters are configured with a dbfs location to save the respective logs (configured from Job cluster -> "Advanced options" -> "Logging").However, the logs are retained in the dbfs indefi...
- 1238 Views
- 2 replies
- 1 kudos
- 1 kudos
@EDDatabricks Thanks for reaching out to us. I think you should explore our Purge option is enabled or not. https://docs.databricks.com/en/administration-guide/workspace/settings/storage.html
- 1 kudos
- 97 Views
- 1 replies
- 0 kudos
Access issue in iPad Air m2
I am trying to run databricks community edition in iPad Air m2. It’s showing me blank page while logging, but in my android phone it’s working fine. Is there any compatibility issue with ios ? Please help me
- 97 Views
- 1 replies
- 0 kudos
- 0 kudos
We have received similar reports which seems to be related to an IOS issue right now, it is important to mention that mobile browsers are not supported by Databricks and we cannot guarantee functionality on them, please refer to https://docs.databric...
- 0 kudos
- 7769 Views
- 8 replies
- 4 kudos
Value array {{QUERY_RESULT_ROWS}} in Databricks SQL alerts custom template
Please include in documentation an example how to incorporate the `QUERY_RESULT_ROWS` variable in the custom template.
- 7769 Views
- 8 replies
- 4 kudos
- 4 kudos
@merca I have been using this same solution in alerts and it has been working great until yesterday, 10/9/2024. Now the results are showing up blank in my alert notification. Are you experiencing the same?
- 4 kudos
- 1174 Views
- 2 replies
- 2 kudos
Unity Catalog
Can we set up unity catalog on databricks community edition?If yes, please share the process.Thanks
- 1174 Views
- 2 replies
- 2 kudos
- 2 kudos
Another option you can try it to setup open source version of unity catalog with apache spark if you don't have possibility to create Azure trial account/
- 2 kudos
- 11530 Views
- 3 replies
- 0 kudos
Resolved! py4j.security.Py4JSecurityException: Constructor public org.apache.spark.ml.feature.VectorAssembler(java.lang.String) is not whitelisted.
Hello, i have a problem.When I try to run the MLlib Assembler (from pyspark.ml.feature import VectorAssembler) I get this error and I don't know what to do anymore. Please help.
- 11530 Views
- 3 replies
- 0 kudos
- 0 kudos
How could we whitelist this error below with DBR 13.3 and above? Py4JError: An error occurred while calling None.org.apache.spark.ml.recommendation.ALS. Trace: py4j.security.Py4JSecurityException: Constructor public org.apache.spark.ml.recommendation...
- 0 kudos
- 2633 Views
- 2 replies
- 0 kudos
Can't enable CLI 2.1 on CI
Hi! this is my CI configuration, I added the databricks jobs configure --version=2.1 command but it stills showing this error, any idea of what can I be doing wrong?Error:Resetting Databricks Job with job_id 1036...WARN: Your CLI is configured to use...
- 2633 Views
- 2 replies
- 0 kudos
- 114 Views
- 3 replies
- 2 kudos
Resolved! Asset Bundles Job If Condition
HiI am trying to use Asset Bundles to deploy a job with email notifications active only on production environment.I tried to use such if statement but it does not seem to do anything.resources: jobs: Master_Load: name: Master Load e...
- 114 Views
- 3 replies
- 2 kudos
- 2 kudos
Okay i made it work with a variable, it was pretty easy. The variable accepted multiline text easily.Thanks!databricks.ymltargets: dev: mode: development ... variables: notifications: default: - user1@smth.com ...
- 2 kudos
- 110 Views
- 3 replies
- 0 kudos
What does DLT INITIALIZING phase do?
In Delta Live Tables, the INITIALIZING phase takes sometimes a minute, sometimes 5 minutes. I'd like to learn what is it doing in the background, and can it be optimized in any way.
- 110 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @antr, In Delta Live Tables (DLT), a feature of Databricks, the "Initializing" phase refers to the first step in the lifecycle of a DLT pipeline run. During this phase, the platform sets up the necessary resources, configurations, and dependencies...
- 0 kudos
- 71 Views
- 1 replies
- 0 kudos
Databricks as a "pure" data streaming software like Confluent
DearsI was wondering if anyone has leveraged Databricks as a "pure" data streaming software in place of Confluent, Flink, Kafka etc.I see the reference architectures placing Databricks on the data processing side mostly once data is made available by...
- 71 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @noorbasha534 ,It depends on what you're asking for. Kafka is primarily a messaging system, optimized for handling high-throughput, distributed message logs. Databricks can read from Kafka as a data source but doesn't replace Kafka's role in messa...
- 0 kudos
- 212 Views
- 1 replies
- 0 kudos
Can't create table in unity catalog.
Hi all.I have created a Databricks workspace on AWS. I can log into the workspace and successfully perform select operations on files in S3, but I am unable to create tables.The error when creating the table is as follows. "Your request failed with s...
- 212 Views
- 1 replies
- 0 kudos
- 0 kudos
I attach the screenshot when I encounter the error.
- 0 kudos