- 933 Views
- 1 replies
- 1 kudos
Resolved! Unable To Complete Lab Due To Missing Lab Data
I’m a Databricks Academy Labs Subscription customer currently working through the self-paced learning plan “Deploy Workloads with Lakeflow Jobs” in preparation for the Databricks DE Associate certification. However, I have encountered an unexpected t...
- 933 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @jessicaY26, Sorry you're experiencing an issue with the labs. Unfortunately, this is not something the community can fix. I recommend submitting a support ticket. Visit the Databricks Training & Certification page, scroll to the Resources section...
- 1 kudos
- 2543 Views
- 3 replies
- 2 kudos
should have the option to mark succeeded with failures as a failure rather than a success
Hi we are having an issue with the way succeeded with failures is handled. We will get emails telling us that we have a failure, which is correct, but then the pipeline actually treats it like a success and keeps going, but actually we would like to ...
- 2543 Views
- 3 replies
- 2 kudos
- 2 kudos
@Advika Currently, if we put run_if: ALL_DONE on task level, it is flagged as succeededwithfailures. But there is another workflow under which this workflow runs. In master workflow, it is marked as succeeded only not succeededwithfailures.My orches...
- 2 kudos
- 5158 Views
- 15 replies
- 0 kudos
DLT table reading not performing file pruning on partition column
I have created bronze table and partitioned on processing date which is date column. In silver table i am putting filter on basis of processing date column to read last 2 days data but it is reading 37 million data but i have only 24722 in last 2 day...
- 5158 Views
- 15 replies
- 0 kudos
- 0 kudos
Hi @Anish_2, Looking at your pipeline DAG, the issue is that you have two separate APPLY CHANGES INTO flows both targeting the same silver table (ag_vlc_hist), one from ag_swt_vlchistory_historical and one from ag_swt_vlchistory. When you define mult...
- 0 kudos
- 1202 Views
- 3 replies
- 0 kudos
Job tasks failing with error "Failed to fetch SQL file" when file exists
I have a job with anywhere from 500-1000 sql tasks where the sql task is using a sql warehouse instance and running a sql script stored in a warehouse path like /Workspace/folder/file.sql. The sql task will fail with the error:Run failed with error m...
- 1202 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @antgei , Thanks for sharing your experience and the workaround. You raise a valid point -- the platform should ensure task files are fully synced before attempting execution, regardless of API rate limiting on the backend. When a job has hundreds...
- 0 kudos
- 3705 Views
- 3 replies
- 0 kudos
How to read and optimize Physical plans in Spark to optimize for TBs and PBs of data workflows
One of the Amazon interviews I attended, which was for a Big data engineer asked me for this particular skill of reading and understanding physical plans in spark to optimize MASSIVE dataloads. But I though spark automatically does all these optimiza...
- 3705 Views
- 3 replies
- 0 kudos
- 0 kudos
Greetings @praveenm00 , good question, and you're right that AQE handles a lot automatically. But understanding physical plans is still worth the investment, especially at TB/PB scale, because AQE works within constraints. It can't fix a bad query s...
- 0 kudos
- 1239 Views
- 2 replies
- 3 kudos
Resolved! Question on cluster sizing as per SLA - No resources in DE certification
How do we optimally size clusters and set configs for any given SLA in production workloads. It would have been great to have a real-life project or implementation to understand in detail. I wish Databricks had a good resource in their certification ...
- 1239 Views
- 2 replies
- 3 kudos
- 3 kudos
Greetings @praveenm00 , Good question, and honestly a fair callout on the cert — it covers cluster config conceptually but never puts you in front of a real sizing problem and there is a good reason for this - it is hard and depends on many factors....
- 3 kudos
- 1318 Views
- 5 replies
- 2 kudos
Resolved! Do not deploy all notebook to the given environment
Hello Community!what is the best way to avoid deploying some notebooks from the asset bundle to higher environments?Given I have following resources structure:resources/ ├── jobs/ │ ├── notebook_a.yml │ ├── notebook_b.yml ← dev onl...
- 1318 Views
- 5 replies
- 2 kudos
- 2141 Views
- 1 replies
- 1 kudos
Resolved! Iceberg native table Streaming in databricks
Hi ! I’ve been exploring the new Managed Iceberg tables integration and noticed a potential discrepancy between the documentation and actual behavior regarding streaming/incremental workloads.According to the official limitations, managed Iceberg tab...
- 2141 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @xwu, Given that managed Iceberg and many of its features are still in Public Preview and explicitly "subject to change," you should treat this as a preview or advanced usage, not as a contractually supported workaround. In other words, it is not ...
- 1 kudos
- 916 Views
- 2 replies
- 0 kudos
Resolved! Why this notebook is returning an error only when called by another notebook?
When I uncomment the last two lines of Called_Notebook.py and run it manually by itself, it correctly returns the output as:Status: SUCCESSCircle area: 50.26544But when I comment out the last two lines of Called_Notebook.py and run it from the Caller...
- 916 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @pradeep_singh, your suggestion worked. Thank you for sharing your knowledge. Worth noticing that not including dbutils.notebook.exit(f"{Value to return}") raised the error in the exception block of the function inside the Called_Notebook - and th...
- 0 kudos
- 979 Views
- 3 replies
- 0 kudos
Getting driver error for my job when migrating to Unity
I am in the process of migrating our jobs from the legacy hive metastore to Unity. I have modified my existing job to read and write from a different bucket as part of the migration. The only change I have made to my job config is to enable this sett...
- 979 Views
- 3 replies
- 0 kudos
- 563 Views
- 1 replies
- 1 kudos
matching sas proc survey means for quantiles in databricks
Hi, I currently have the following code in databricks that i am using to calculate survey estimates and quantiles. I wish to match (or get as close) to SAS results using proc survey means for quantiles as possible (I am able to match proportions fine...
- 563 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @Jake3, I didn’t expect to jump into this thread given my complete lack of SAS knowledge and the rather serious-looking statistics you’re working with. But I decided to treat your post as a chance to see what Genie Code could do with it... and it...
- 1 kudos
- 1947 Views
- 1 replies
- 1 kudos
Resolved! Azure Databricks S3 External Location
Hi,I have recently created a new Azure Databricks account and several workspaces. I am needing to ingest data from an S3 bucket and am trying to follow the documentation detailed here:https://learn.microsoft.com/en-us/azure/databricks/connect/unity-c...
- 1947 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @tsmith-11, Having checked internally and from the screenshot, this doesn’t look like a configuration issue on your side but rather that the cross‑cloud S3 feature isn’t enabled on your Azure Databricks account/metastore yet. You should see an AWS...
- 1 kudos
- 787 Views
- 3 replies
- 1 kudos
Refresh streaming table error
Refresh streaming table sql succeeds for the first time.The subsequent refresh statements fails with TABLE_OR_VIEW_NOT_FOUND error.The streaming table is still available in the same catalog and schema
- 787 Views
- 3 replies
- 1 kudos
- 1 kudos
HiHere is the query i use REFRESH STREAMING TABLE `cxxxxx`.`tgt_dev`.`ldp_csv` It is successful when i execute the first time.If i run the same query after 30 min, it throws error TABLE_OR_VIEW_NOT_FOUND
- 1 kudos
- 1104 Views
- 2 replies
- 0 kudos
Resolved! Creating a sync table from a workspace catalog to a project
We have a table in a workspace we would like to sync to a project. And we can fine choose the database project but we cannot see the database in the first section (Destination), see attached file.
- 1104 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @Sega2, Thanks for sharing the screenshot — this helps clarify what's happening. There are two likely reasons why you're not seeing your database in the Destination section: 1. The source table must be in Unity Catalog Synced tables only support ...
- 0 kudos
- 8764 Views
- 7 replies
- 4 kudos
Databricks SQL Warehouse, Tableau and spark.driver.maxResultSize error
I'm attempting to create a tableau extract on tableau server with a connection to databricks large sql warehouse. The extract process fails due to spark.driver.maxResultSize error.Using a databricks interactive cluster in the data science & engineer...
- 8764 Views
- 7 replies
- 4 kudos
- 4 kudos
This is a common headache, I've definitely hit this wall myself when dealing with large datasets! My go-to is usually optimizing queries to return less data initially, maybe aggregating more in SQL. Have you considered checking your click speed on a ...
- 4 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 584 | |
| 349 | |
| 287 |
