First some context: I'm setting up a Genie Space and have multiple benchmark questions with business validated SQL ground truths defined. Currently no configuration defined, apart from metadata and SQL joins.I haven't found any other way of reporting...
I am exporting parquet files (partitioned by id) in append mode. However, I encounter errors occasionally, while other times the job completes successfully.Apache Spark Exception: Exception thrown in awaitResult: hadoop_azure_shaded.com.microsoft.azu...
Its generally due to race conditions when Spark checks for existing partition files before writing combined with Azure Blob Storage's eventual consistency mode.You can follow below1. Switch to Delta Lake - You can use Delta Lake format instead of Par...
I have a Data Lake Transformation (DLT) pipeline that runs weekly. Normally, it takes 8 minutes to complete, but since last Friday (June 19), it has been running for hours until it encounters an out-of-memory error. This pipeline is responsible for c...
Hey, I found that another engineer added joins to the metric view definitions to a federated query table. I converted it to a streaming table to use CDF and now time reduced. I need to improve joins to reduce latency as much as possible, but I’m not ...
If any code changes are done and moved to bit bucket based on env now i want to fetch all the code changes and move all these changes to a table in adls/unity catalog table.Anyone please help with the approaches we are using bit bucket for repository
Hi everyone,I am looking to accelerate my growth as a Data Engineer and am seeking hands-on guidance from an experienced professional. To gain real-world experience, I am offering my technical services completely free of charge.What I bring to the ta...
check my github where i make some small project while learning https://github.com/hasansayyed13?tab=repositories
Hello everyone, I'm implementing a lakeflow connect pipeline that is orchestrated by a lakeflow job. My question is: Is there a way ti configure a job parameter so that when ir receives the vale "true" a full refresh Is performed in the pipeline and ...
You can solve it using the conditional tasks in the jobs. You can use If/Else conditional tasks in the main job configuration to run full/incremental based on the parameter. It allows you to route to different pipeline tasks based on the parameter va...
I'm working with Databricks declarative pipelines and have defined a custom PySpark data source (CDS) in its own standalone Python module. I include this module as part of the pipeline resources. What I find interesting is that, even without explicit...
That is a great observation! You aren't actually triggering a hidden "auto-discovery" feature for custom data sources. Instead, what you are seeing is a byproduct of how Spark Declarative Pipelines (SDPs) evaluate pipeline resources.To answer your sp...
Hi Community,I am trying to implement multi-destination pipelines and came across this code in Databricks documentation: Create multi-destination pipelines | Databricks on AWSKey 'table' is repeated in all examples - I believe JSON objects should not...
You are absolutely correct! Great catch.The JSON example provided in the documentation is structurally invalid. Because standard JSON parsers do not support duplicate keys within the same dictionary object, the second "table" key will silently overwr...
Hi Team, I was using Databricks community edition for learning purpose. I had an account https://community.cloud.databricks.com/?o=6822095545287159 where I stored all my learning notebooks. Unfortunately, this account suddenly stopped working, and I ...
The workspace id you have shared seems to be related to a workspace which is still in running state, if you missed the login access to this workspace then our team you have reached over email would be able to assist.I will add the following doc for s...
I have five data ingestion jobs using Lakeflow Connect from SQL Server to Databricks, along with a Spark Declarative Pipeline to load data from bronze to silver. I need guidance on promoting these pipelines and jobs from Dev to Stage and Prod, and ho...
Promoting data workflows across environments is a critical step, and setting it up correctly from the start will save you a lot of operational headaches.The recommended and most robust methodology for managing the lifecycle of both Lakeflow Connect j...
We want to leverage these connectors to pull in marketing spend data. But the docs seem to say that the destination must be unique based on accounts. For Tik, we have a hundred accounts... each account will have a destination table for each object. ...
I share your pain, I am sitting on 710 landing tables for this reason (of our services is templated website hosting and this is all Google Tag data). I tried all kinds of tricks in the UI to try and land data into a single landing table but it would...
We have a python module (WidgetUtil.py) that sits in the same folder as our notebook. For the past few years we have been using a simple import statement to use it. Starting with DBR18.0 the imports fails intermittently (25% of the time) when running...
Hello, we also have the same issue. The error is sporadic (on the next executions it passes, then it fails again). It is either "cannot import name ..." or "no module ...".We did not change anything and we have this issue since a few hours ago. We ar...
We're currently setting up Databricks Asset Bundles (DAB) with a CI/CD pipeline using Azure DevOps.Our planned development workflow is as follows:Main branch → Developer creates a feature branch → Implement changes → Create a Pull Request → Senior de...
The workflow itself is solid. A few things worth tightening at each stage, based on running this in production with Azure DevOps:At the PR stage, run databricks bundle validate as a required check before merge is allowed. It catches wrong field names...
ContextI'm working on integration patterns between enterprise NAS storage (Amazon FSx for NetApp ONTAP) and Databricks via S3 Access Points. S3 Access Points provide S3 API access to file data without copying — a common pattern for organizations with...
Hi Team,We are currently using the Databricks Deep clone feature to clone our tables to Databricks DR environment. When we deploy our jobs, they run in production and the tables get cloned to the DR. But the views dont get cloned as deepclone doesnt ...
Thank you AshwinWe have implemented CI/CD pipeline to deploy the jobs, but it doesn't run the jobs.The jobs which load the data/create the tables need to run at specific times and so that is handled by a separate scheduling tool. In such a scenario, ...
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 576 | |
| 349 | |
| 287 |
