- 742 Views
- 5 replies
- 0 kudos
Lake Bridge ETL Rehouse into AWS Data bricks options ?
Lake Bridge ETL Rehouse into AWS Data bricks options ?==========================================Hi Community experts?Thanks for replies to my threads.We reviewed the Lake Bridge thread opened here. The functionality claimed, it can convert on-prem ET...
- 742 Views
- 5 replies
- 0 kudos
- 0 kudos
Thanks for weighing in. For the same question in another data engineering discussion board not giving a comfort feeling about this . They project a nightmare scenarios.
- 0 kudos
- 22915 Views
- 14 replies
- 10 kudos
How to overcome missing query parameters in Databricks SQL?
Hi, there! I'm trying to build up my first dashboard based on Dataabricks SQL. As far as I can see if you define a query parameter you can't skip it further. I'm looking for any option where I can make my parameter optional. For instance, I have a ta...
- 22915 Views
- 14 replies
- 10 kudos
- 10 kudos
Hello guys, I'm not sure if you could solve this issue but here is how I've handled it:SELECT *FROM my_tableWHERE (CASE WHEN LEN(:my_parameter) > 0 THEN my_column = :my_parameter ELSE my_column = my_column END)I hope this can help!
- 10 kudos
- 586 Views
- 1 replies
- 2 kudos
Resolved! Reg. virtual learning festival coupon
Hi team, I've attended DAIS 2025 Virtual Learning Festival (11 June - 2 July) and received coupon.1. Does the coupon applicable for 'Databricks certified associate developer for apache spark' as well?2. I'm preparing to take exam for spark certificat...
- 586 Views
- 1 replies
- 2 kudos
- 2 kudos
Hey @SugathithyanM thanks for the additional mention here, please see our conversation for reference. For any others also interested: 1. Yes, the certification voucher code is applicable on any Databricks Certification exam, including the Apache Spar...
- 2 kudos
- 945 Views
- 1 replies
- 0 kudos
Autoloader on ADLS blobs with archival enabled
Hi All, I'm trying to change our Ingestion process to use Autoloader to identify new files landing in a directory on ADLS. The ADLS directory has access tier enabled to archive files older than a certain time period. When I'm trying to set up Autoloa...
- 945 Views
- 1 replies
- 0 kudos
- 0 kudos
Facing the same issue when trying to use autoloader with useNotifications. Did you ever found a workaround?
- 0 kudos
- 900 Views
- 1 replies
- 1 kudos
asset bundle
Hello Guys,Its possible to use asset bundle databricks to set tbproperties of table?Cordially,
- 900 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @seefoods ,There is no out of the box to do this in DAB. Look at supported resources that you can configure with DABs:Databricks Asset Bundles resources | Databricks DocumentationAs a workaround, you can just create notebook that will set tblprope...
- 1 kudos
- 1728 Views
- 9 replies
- 1 kudos
Resolved! View Failing due to filed not recognise
Hi, I am facing problem where my existing view has stopped working due to unrecognised filed which is a alias field .I am using the same definition and spark configuration.DBR : 14.3 LTS, Spark 3.5.0Any one faced similar problem recently ?
- 1728 Views
- 9 replies
- 1 kudos
- 1 kudos
HI @Khaja_Zaffer yes it was actaully issue with privacera version . after invastaigation with privacera team they suggested to upgrade the privacera version to 9.0.35.1.
- 1 kudos
- 1534 Views
- 1 replies
- 2 kudos
Driver terminated abnormally due to FORCE_KILL
We have a job running on a job cluster where sometimes the driver dies:> The spark driver has stopped unexpectedly and is restarting. Your notebook will be automatically reattached.But the metrics don't suggest an explanation for this situation.In th...
- 1534 Views
- 1 replies
- 2 kudos
- 2 kudos
That error is usually related to driver load. Try upsizing the driver one size and see if it still happens. Otherwise, for troubleshooting, driver problems are surfaced to the cluster's event log, like DRIVER_NOT_RESPONDING and DRIVER_UNAVAILABLE. Yo...
- 2 kudos
- 727 Views
- 2 replies
- 2 kudos
Resolved! Python os.listdir() behavior difference between 15.4LTS and 16.4LTS DBRs
We found that when using os.listdir() in Databricks notebooks to list files stored in the Workspace (i.e., alongside the notebook, not in DBFS), file extensions were missing in Databricks Runtime 14.3 LTS and 15.4 LTS, but appeared correctly in 16.4 ...
- 727 Views
- 2 replies
- 2 kudos
- 2 kudos
This is expected and changed in DBR16.2: In Databricks Runtime 16.2 and above, notebooks are supported as workspace files.
- 2 kudos
- 929 Views
- 2 replies
- 0 kudos
Databricks Workflow Automatically Marked as Failed When Autoloader Stream Fails in a Task
Issue: I want my Databricks Task/Workflow, which is running a pytest test, to not be automatically marked as "Failed" when an Autoloader stream shuts down due to an issue. It seems that if an Autoloader / Structured Streaming stream fails, it will au...
- 929 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @r_g_s_cn ,When a streaming query (like Auto Loader) fails in Databricks, especially due to a schema mismatch, the job or task is automatically marked as FAILED, even if you catch the exception in your code. That’s because the failure is detected ...
- 0 kudos
- 693 Views
- 2 replies
- 1 kudos
Resolved! Autoloader Functionality Question: Pull API data directly?
Hi there, when referencing Common data loading patterns > Enable flexible semi-structured data pipelines , I noticed this interesting code snippet:spark.readStream.format("cloudFiles") \ .option("cloudFiles.format", "json") \ # will ensure that t...
- 693 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @ChristianRRL ,Unfortunately, they chose quite confusing name. Autloader only supports one type of source -> cloudFiles.And cloudFiles is nothing but your cloud object storage. So in this example they have a datalake directory /api/request where t...
- 1 kudos
- 832 Views
- 4 replies
- 1 kudos
Autoloader Console Output Issue
In reference prior post: Re: Autoloader Error Loading and Displaying - Databricks Community - 122579I am attempting to output results to the console (notebook cell), but am not seeing anything (other than the dataframe schema). Is this expected? I am...
- 832 Views
- 4 replies
- 1 kudos
- 1 kudos
Hi @ChristianRRL ,Did you run this code before? Maybe all your source files has been already written to checkpoint. Try to upload new json file and run it again. Also, you can check drivers logs. Sometimes you can find them error messages.
- 1 kudos
- 678 Views
- 1 replies
- 1 kudos
Resolved! Considering Autoloader for Bronze to Silver transformations
I’m currently implementing Auto Loader to ingest data from the source into the Bronze layer—essentially mapping the raw data into Delta tables. Now, I’ve also been considering using Auto Loader for Bronze-to-Silver transformations.Are there any pros ...
- 678 Views
- 1 replies
- 1 kudos
- 1 kudos
Auto Loader is for loading raw files, not loading Delta Lake or Apache Iceberg tables, see more here. Instead, stream from a Delta Lake table.
- 1 kudos
- 957 Views
- 1 replies
- 1 kudos
PySpark Lazy Evaluation
PySpark Lazy Evaluation - Why does my logging function seem to execute without an explicit action in Databricks?Hello everyone,I was scrolling and found some Medium post on a PySpark (https://medium.com/@sudeepwrites/pyspark-secrets-no-one-talks-abou...
- 957 Views
- 1 replies
- 1 kudos
- 1 kudos
I don't have full access to that article, but here's something that might help clarify things! While Spark uses lazy evaluation (meaning it waits to execute until absolutely necessary), Python works with eager evaluation. This means that when you ru...
- 1 kudos
- 1160 Views
- 2 replies
- 0 kudos
Resolved! Streaming Failure on Full Refresh Tables while using Serverless
On the afternoon of the 2025-07-30 my team began to experience issues with pipeline tasks that were set to full refresh and full refresh only. These pipelines were defined to use serverless, and the only way we were able to get them back online was t...
- 1160 Views
- 2 replies
- 0 kudos
- 0 kudos
Thanks for the reply - after consulting with our Databricks rep we determined it was a bug released by Databricks with a recent update to serverless. The Databricks team has resolved the issue and we have switched back to serverless.
- 0 kudos
- 651 Views
- 2 replies
- 4 kudos
Resolved! Autoloader fails when creating external Delta table in same notebook
Hi everyone,I’ve set up Databricks Autoloader to ingest data from ADLS into a Delta table. The table is defined as an external Delta table, with its location pointing to a path in ADLS.Here’s the flow I’m using:On the first run for a given data sourc...
- 651 Views
- 2 replies
- 4 kudos
- 4 kudos
Thank you for your response!I've tried something similar, added time.sleep(10) between table creation and autoloader initialization, but it did not work.What worked was separating the table creation and the autoloader initialization into different ce...
- 4 kudos
| User | Count |
|---|---|
| 1637 | |
| 791 | |
| 534 | |
| 349 | |
| 287 |
