- 2021 Views
- 2 replies
- 2 kudos
Table Trigger - Too many logfiles
Hi, we have implemented a job that runs on a trigger of a table update. The job worked perfectly, until the source table now have accumulated too many log files and the job isn't running anymore. Only the error message below:Storage location /abcd/_d...
- 2021 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi @184754 Interesting topic, as the docs says:"Log files are deleted automatically and asynchronously after checkpoint operations and are not governed by VACUUM. While the default retention period of log files is 30 days, running VACUUM on a table r...
- 2 kudos
- 3197 Views
- 2 replies
- 1 kudos
Cloning of Workflow from One env to different env using Job API
Hi Team,One of my team members recently shared one requirement: he wants to migrate the 10 Workflows from the sandbox to the dev environment to run his model in dev env.I wanted to move all these workflows in an automated way and one of the solutions...
- 3197 Views
- 2 replies
- 1 kudos
- 1520 Views
- 2 replies
- 2 kudos
Is there any option in databricks to estimate cost of a query before execution
Hi Team,I want to check if there is any option in data bricks which can help to estimate cost of a query before execution?I mean calculate DBU before actual query execution based on the logical plan? Regards
- 1520 Views
- 2 replies
- 2 kudos
- 2 kudos
Is there any way to track the progress or ETA?Do we have access to ideas portal? Where can we search this reference number DB-I-5730?
- 2 kudos
- 6147 Views
- 2 replies
- 2 kudos
Ways to write fast millions of rows inside a new delta table
Hello everyone,I am facing an issue with writing 100–500 million rows (partitioned by a column) into a newly created Delta table. I have set up a cluster with 256 GB of memory and 64 cores. However, the following code takes a considerable amount of t...
- 6147 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi @jeremy98 This is what I would suggest to test:1) remove repartition step or reduce number or partitions (start with number of cores and then try to increase it x2, x3) repartition(num_partitions*4, partition_col) I know repartitioning helps to di...
- 2 kudos
- 1700 Views
- 1 replies
- 0 kudos
Resolved! How do I use DataBricks SQL query to convert a field value % back into a wildcard?
Hi. If I've posted to the wrong area, please let me know.I am using SQL to join two tables. One table has the wildcard '%' stored as text/string/varchar. I need to join the value of TableA.column1 to TableB.column1 based on the wildcard in the str...
- 1700 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi,Could you please try the query below and let me know if it meets your requirements? SELECT * FROM TableA A LEFT JOIN TableB B ON A.Column1 LIKE REPLACE(B.Column1, '%', '%%')Replace helps us in treating the %' stored in TableB.Column1 as a wildcar...
- 0 kudos
- 5619 Views
- 4 replies
- 1 kudos
Error: no streaming listener attached to the spark app is the error we are observing post accessing streaming statistics API. Please help us with this issue ASAP. Thanks.
Issue: Spark structured streaming applicationAfter adding the listener jar file in the cluster init script, the listener is working (From what I see in the stdout/log4j logs)But when I try to hit the 'Content-Type: application/json' http://host:port/...
- 5619 Views
- 4 replies
- 1 kudos
- 5645 Views
- 3 replies
- 1 kudos
I am unable to attach a streaming listener to a spark streaming job. Error: no streaming listener attached to the spark application is the error we are observing post accessing streaming statistics API. Please help us with this issue ASAP. Thanks.
Issue:After adding the listener jar file in the cluster init script, the listener is working (From what I see in the stdout/log4j logs)But when I try to hit the 'Content-Type: application/json' http://host:port/api/v1/applications/app-id/streaming/st...
- 5645 Views
- 3 replies
- 1 kudos
- 2181 Views
- 2 replies
- 0 kudos
Resolved! Delta Live Tables: How does it identify new files?
Hi,I'm importing large numbers of parquet files (ca 5200 files per day, they each land in a separate folder) into Azure ADLS storage.I have a DLT streaming table reading from the root folder.I noticed a massive spike in storage account costs due to f...
- 2181 Views
- 2 replies
- 0 kudos
- 0 kudos
To resolve the issue of excessive directory scanning, I have changed the folder structure to separate historical files from current files and reduce the number of folders and files that the Databrick process monitors.
- 0 kudos
- 3255 Views
- 3 replies
- 0 kudos
Databricks Asset Bundle - Not fully deploying in Azure Pipeline
Hello Community, I'm encountering a challenging issue with my Azure Pipeline and I'm hoping someone here might have some insights. I'm attempting to deploy a Databricks bundle that includes both notebooks and workflow YAML files. When deploying the ...
- 3255 Views
- 3 replies
- 0 kudos
- 0 kudos
Hello,Same issue here, files and wheel are deployed and present on the databricks workspace but the jobs are not created
- 0 kudos
- 20642 Views
- 8 replies
- 5 kudos
Resolved! Error on Azure-Databricks write output to blob storage account
Hello,After implementing the use of Secret Scope to store Secrets in an azure key vault, i faced a problem.When writting an output to the blob i get the following error:shaded.databricks.org.apache.hadoop.fs.azure.AzureException: Unable to access con...
- 20642 Views
- 8 replies
- 5 kudos
- 5 kudos
Hi all,Is it correct that Azure-Databricks only support to write data to Azure Data Lake Gen2 and does not support for Azure Storage Blob (StorageV2 - general purpose) ?In my case, I can read the data from Azure Storage Blob (StorageV2 - general purp...
- 5 kudos
- 3883 Views
- 1 replies
- 5 kudos
Notebook cell output results limit increased- 10,000 rows or 2 MB. Hi all, Now, databricks start showing the first 10000 rows instead of 1000 rows.Tha...
Notebook cell output results limit increased- 10,000 rows or 2 MB.Hi all,Now, databricks start showing the first 10000 rows instead of 1000 rows.That will reduce the time of re-execution while working on fewer sizes of data that have rows between 100...
- 3883 Views
- 1 replies
- 5 kudos
- 5 kudos
Hi Ajay,Is there any way to increase this limit?Thanks, Fatima
- 5 kudos
- 12963 Views
- 3 replies
- 4 kudos
"Fatal error: The Python kernel is unresponsive." DBR 14.3
Running almost any notebook with a merge statement in Databricks with DBR 14.3 I get the following error and the notebook exists:"Fatal error: The Python kernel is unresponsive."I would provide more code, but like I said, it is pretty much anything w...
- 12963 Views
- 3 replies
- 4 kudos
- 4 kudos
Same thing, not finding any documentation out there around "spark.databricks.driver.python.pythonHealthCheckTimeoutSec". @ac0 or @Ayushi_Suthar any more details you found on this?
- 4 kudos
- 1539 Views
- 2 replies
- 2 kudos
Resolved! PKEY Upserting Pattern With Older Runtimes
Hi there,I'm aware that nowadays newer runtimes of Databricks support some great features, including primary and foreign key constraints. I'm wondering, if we have clusters that are running older runtime versions, are there Upserting patterns that ha...
- 1539 Views
- 2 replies
- 2 kudos
- 2 kudos
For clusters running older Databricks runtime versions, such as 13.3, you can still implement upserting patterns effectively, even though they may not support the latest features like primary and foreign key constraints available in newer runtimes. O...
- 2 kudos
- 4537 Views
- 6 replies
- 1 kudos
Resolved! Debugging a workflow
Hi All,With the new availability of debugging notebooks are really great. But how do you debug a workflow? Any suggestions on recommended practices on how to debug workflows? Best R, Thomas
- 4537 Views
- 6 replies
- 1 kudos
- 1 kudos
@Sega2 Please refer to this doc and confirm if it's helpfulhttps://docs.databricks.com/en/jobs/repair-job-failures.html#re-run-failed-and-skipped-tasks
- 1 kudos
- 3179 Views
- 3 replies
- 0 kudos
How to Schedule stop and start for a Continuous DLT pipeline
I have a use case where in there is a DLT pipeline in continuous mode, the requirement is to run this pipeline on scheduled basis i.e everyday it starts at 8am and stops at 8pm. I see a schedule option to start the job but don't see any option to sto...
- 3179 Views
- 3 replies
- 0 kudos
- 0 kudos
You can terminate any Job with REST API.I recommend to use Python SDK jobs.cancel_all_runs() methodhttps://databricks-sdk-py.readthedocs.io/en/latest/workspace/jobs/jobs.html#databricks.sdk.service.jobs.JobsExt.cancel_all_runs
- 0 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 586 | |
| 349 | |
| 287 |
