- 242 Views
- 1 replies
- 0 kudos
Getting error hadoop_azure_shaded.com.microsoft.azure.storage.StorageException: The specified blob d
I am exporting parquet files (partitioned by id) in append mode. However, I encounter errors occasionally, while other times the job completes successfully.Apache Spark Exception: Exception thrown in awaitResult: hadoop_azure_shaded.com.microsoft.azu...
- 242 Views
- 1 replies
- 0 kudos
- 0 kudos
Its generally due to race conditions when Spark checks for existing partition files before writing combined with Azure Blob Storage's eventual consistency mode.You can follow below1. Switch to Delta Lake - You can use Delta Lake format instead of Par...
- 0 kudos
- 1046 Views
- 4 replies
- 0 kudos
Resolved! DLT pipelines failing out of memory (serverless)
I have a Data Lake Transformation (DLT) pipeline that runs weekly. Normally, it takes 8 minutes to complete, but since last Friday (June 19), it has been running for hours until it encounters an out-of-memory error. This pipeline is responsible for c...
- 1046 Views
- 4 replies
- 0 kudos
- 0 kudos
Hey, I found that another engineer added joins to the metric view definitions to a federated query table. I converted it to a streaming table to use CDF and now time reduced. I need to improve joins to reduce latency as much as possible, but I’m not ...
- 0 kudos
- 245 Views
- 1 replies
- 0 kudos
Aspiring Data Engineer offering free project support in exchange for mentorship 🚀
Hi everyone,I am looking to accelerate my growth as a Data Engineer and am seeking hands-on guidance from an experienced professional. To gain real-world experience, I am offering my technical services completely free of charge.What I bring to the ta...
- 245 Views
- 1 replies
- 0 kudos
- 0 kudos
check my github where i make some small project while learning https://github.com/hasansayyed13?tab=repositories
- 0 kudos
- 374 Views
- 2 replies
- 0 kudos
Trigger a full refresh in a lakeflow connect pipeline with a job
Hello everyone, I'm implementing a lakeflow connect pipeline that is orchestrated by a lakeflow job. My question is: Is there a way ti configure a job parameter so that when ir receives the vale "true" a full refresh Is performed in the pipeline and ...
- 374 Views
- 2 replies
- 0 kudos
- 0 kudos
You can solve it using the conditional tasks in the jobs. You can use If/Else conditional tasks in the main job configuration to run full/incremental based on the parameter. It allows you to route to different pipeline tasks based on the parameter va...
- 0 kudos
- 1285 Views
- 2 replies
- 0 kudos
Resolved! How does Databricks handle registration and discovery of custom PySpark data sources in SDPs?
I'm working with Databricks declarative pipelines and have defined a custom PySpark data source (CDS) in its own standalone Python module. I include this module as part of the pipeline resources. What I find interesting is that, even without explicit...
- 1285 Views
- 2 replies
- 0 kudos
- 0 kudos
That is a great observation! You aren't actually triggering a hidden "auto-discovery" feature for custom data sources. Instead, what you are seeing is a byproduct of how Spark Declarative Pipelines (SDPs) evaluate pipeline resources.To answer your sp...
- 0 kudos
- 296 Views
- 1 replies
- 0 kudos
Documentation issue: Invalid JSON example in Lakeflow Connect multi-destination pipeline
Hi Community,I am trying to implement multi-destination pipelines and came across this code in Databricks documentation: Create multi-destination pipelines | Databricks on AWSKey 'table' is repeated in all examples - I believe JSON objects should not...
- 296 Views
- 1 replies
- 0 kudos
- 0 kudos
You are absolutely correct! Great catch.The JSON example provided in the documentation is structurally invalid. Because standard JSON parsers do not support duplicate keys within the same dictionary object, the second "table" key will silently overwr...
- 0 kudos
- 6978 Views
- 6 replies
- 1 kudos
Recover Lost Notebook
Hi Team, I was using Databricks community edition for learning purpose. I had an account https://community.cloud.databricks.com/?o=6822095545287159 where I stored all my learning notebooks. Unfortunately, this account suddenly stopped working, and I ...
- 6978 Views
- 6 replies
- 1 kudos
- 1 kudos
The workspace id you have shared seems to be related to a workspace which is still in running state, if you missed the login access to this workspace then our team you have reached over email would be able to assist.I will add the following doc for s...
- 1 kudos
- 420 Views
- 2 replies
- 0 kudos
How to promote Lakeflow Connect and Spark Declarative Pipeline to a higher environment
I have five data ingestion jobs using Lakeflow Connect from SQL Server to Databricks, along with a Spark Declarative Pipeline to load data from bronze to silver. I need guidance on promoting these pipelines and jobs from Dev to Stage and Prod, and ho...
- 420 Views
- 2 replies
- 0 kudos
- 0 kudos
Promoting data workflows across environments is a critical step, and setting it up correctly from the start will save you a lot of operational headaches.The recommended and most robust methodology for managing the lifecycle of both Lakeflow Connect j...
- 0 kudos
- 690 Views
- 2 replies
- 2 kudos
DAB best practices suggestion
We're currently setting up Databricks Asset Bundles (DAB) with a CI/CD pipeline using Azure DevOps.Our planned development workflow is as follows:Main branch → Developer creates a feature branch → Implement changes → Create a Pull Request → Senior de...
- 690 Views
- 2 replies
- 2 kudos
- 2 kudos
The workflow itself is solid. A few things worth tightening at each stage, based on running this in production with Azure DevOps:At the PR stage, run databricks bundle validate as a required check before merge is allowed. It catches wrong field names...
- 2 kudos
- 1080 Views
- 3 replies
- 0 kudos
Resolved! Unity Catalog External Location with Amazon S3 Access Points,session policy behavior and workarounds
ContextI'm working on integration patterns between enterprise NAS storage (Amazon FSx for NetApp ONTAP) and Databricks via S3 Access Points. S3 Access Points provide S3 API access to file data without copying — a common pattern for organizations with...
- 1080 Views
- 3 replies
- 0 kudos
- 415 Views
- 3 replies
- 1 kudos
Views in DR environment
Hi Team,We are currently using the Databricks Deep clone feature to clone our tables to Databricks DR environment. When we deploy our jobs, they run in production and the tables get cloned to the DR. But the views dont get cloned as deepclone doesnt ...
- 415 Views
- 3 replies
- 1 kudos
- 1 kudos
Thank you AshwinWe have implemented CI/CD pipeline to deploy the jobs, but it doesn't run the jobs.The jobs which load the data/create the tables need to run at specific times and so that is handled by a separate scheduling tool. In such a scenario, ...
- 1 kudos
- 328 Views
- 1 replies
- 1 kudos
Streaming Amazon DocumentDB to Databricks in near real time - what's the best approach?
Hi everyone,I'm looking for advice from anyone who has implemented near real-time ingestion from Amazon DocumentDB into Databricks.Our current architecture is:Application → Amazon DocumentDBPython AWS Lambda functions capture changes from DocumentDBL...
- 328 Views
- 1 replies
- 1 kudos
- 1 kudos
The best pattern I can think of is to put a streaming bus between DocumentDB and Databricks and consume it with Structured Streaming. You are most of the way there already. Lowest-disruption path, since you already capture changes in Lambda: Repoint...
- 1 kudos
- 1250 Views
- 4 replies
- 1 kudos
Resolved! Unable to see a folder in DBFS
Hi Team,We have a few scripts stored in a folder on a DBFS path. Recently, we've noticed that when we navigate to this path manually, the folder appears to be empty, and we are unable to see the scripts.However, the jobs that reference and access the...
- 1250 Views
- 4 replies
- 1 kudos
- 1 kudos
Hi @Shivani_Komma99, Thanks for flagging this. Based on the behaviour you described, this appears to be consistent with a DBFS browser UI issue/regression rather than a problem with the underlying files, especially since the files are still accessibl...
- 1 kudos
- 668 Views
- 3 replies
- 0 kudos
Bundle deployment overwrites artifacts while jobs are running - best practices?
Hi everyone,I'm using #Declarative Automation Bundles (DAB) to deploy data pipelines, and I've run into an issue with concurrent job runs and deploymentWhat happened:I started a job that depends on a wheel file built by the bundle (timestamped artifa...
- 668 Views
- 3 replies
- 0 kudos
- 0 kudos
Do you have source_linked_deployment set as false? That's probably causing it.
- 0 kudos
- 1723 Views
- 14 replies
- 6 kudos
Resolved! PySpark AnalysisException: Ambiguous reference to field t when parsing nested JSON
I'm working on a personal data engineering project using Kafka, Spark Structured Streaming, and Docker.The application consumes two Kafka topics that originate from an external market-data websocket source:a trade streama candlestick (kline/OHLCV) st...
- 1723 Views
- 14 replies
- 6 kudos
- 6 kudos
Spark Structured Streaming writes to file sinks and generally it uses a phased commit by writing temporary files to the output directory followed by writing metadata with references and a final commit by moving/renaming temp files to final names. You...
- 6 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 582 | |
| 349 | |
| 287 |
