- 2421 Views
- 3 replies
- 1 kudos
Azure Asset Bundle deploy removes the continous: true configuration
I have this pipeline configuration that I'm deploying using Azure Asset Bundles:ingest-pipeline.test.yml```resources: pipelines: ingest-pipeline-test: name: ingest-pipeline-test-2 clusters: - label: default node_type_id:...
- 2421 Views
- 3 replies
- 1 kudos
- 1 kudos
I just got confirmation internally that as of now Continuous is disabled in mode: development
- 1 kudos
- 5443 Views
- 2 replies
- 0 kudos
Delta UserMetadata attribute using Serverless Compute
To automate the configuration of Spark on serverless compute, Databricks has removed support for manually setting most Spark configurations. I've used userMetadata attribute to add context for all workloads that write to delta tablesI have the follow...
- 5443 Views
- 2 replies
- 0 kudos
- 0 kudos
@RobsonNLPT I see, so you're currently getting a "Configuration spark.databricks.delta.commitInfo.userMetadata is not available" correct? I did an internal research and there doesn't seem to be a workaround for this yet, but it'll be good if you co...
- 0 kudos
- 5265 Views
- 1 replies
- 0 kudos
Databricks Connect Vscode. Cannot find package installed on cluster
I am using Databricks Connect v2 to connect to a UC enabled cluster. I have a package I have made and installed in a wheel file on the cluster. When using vscode to import the package and use it I get a module not found error when running cell by cel...
- 5265 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @brady_tyson just checking—are you still facing this issue with using your custom package and Databricks Connect? If so, here are a few questions to collect some data points about your setup: Is Databricks Connect properly installed and configur...
- 0 kudos
- 2503 Views
- 1 replies
- 0 kudos
Problem with streaming jobs (foreachBatch) with USER_ISOLATION compute cluster
We have been trying to run a streaming job on an all-purpose compute (4 cores, 16 gb) in the “user_isolation”, recommended by databricks to run with/for unity catalog. The job reads CDC files produced by a table refreshed every hour and produces aro...
- 2503 Views
- 1 replies
- 0 kudos
- 0 kudos
@prakharcode Thank you for sharing the detailed information about your issue. Before diving into solutions, I want to confirm if this is still an ongoing problem you're facing. Regarding the difference in job performance between "NO_ISOLATION" mode ...
- 0 kudos
- 5251 Views
- 1 replies
- 0 kudos
Why does soda not initialize?
Hey everyone, im using autoloader x soda.I'm new to both,The idea is to ingest with quality checks in my silver table for every batch in a continuous ingestion.I tried to configure soda as str just like the docs show, but its seems that it keeps on t...
- 5251 Views
- 1 replies
- 0 kudos
- 0 kudos
@sakuraDev is this still an ongoing issue? If so, could you please share the error stacktrace as a file attachment? Thanks.
- 0 kudos
- 7876 Views
- 3 replies
- 0 kudos
Python worker exited unexpectedly (crashed)
I have a failing pipeline which results in the following failure:org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 2053.0 failed 4 times, most recent failure: Lost task 0.3 in stage 2053.0 (TID 4594) (10.171.199.129 e...
- 7876 Views
- 3 replies
- 0 kudos
- 0 kudos
@zsh24 , just checking if you were able to address the problem or need further guidance?
- 0 kudos
- 22736 Views
- 2 replies
- 0 kudos
How to parallelly merge data into partitions of databricks delta table using PySpark/Spark streaming?
I have a PySpark streaming pipeline which reads data from a Kafka topic, data undergoes thru various transformations and finally gets merged into a databricks delta table. In the beginning we were loading data into the delta table by using the merge ...
- 22736 Views
- 2 replies
- 0 kudos
- 0 kudos
@bobbysidhartha :When merging data into a partitioned Delta table in parallel, it is important to ensure that each job only accesses and modifies the files in its own partition to avoid concurrency issues. One way to achieve this is to use partition...
- 0 kudos
- 4929 Views
- 3 replies
- 0 kudos
DLT Merge tables into Delta
We are trying to load a Delta table from streaming tables using DLT. This target table needs a MERGE of 3 source tables. But when we use the DLT command with merge it says Merge is not supported. Is this anything related to DLT version? Please help u...
- 4929 Views
- 3 replies
- 0 kudos
- 0 kudos
Hey @JothyGanesan Please take a look at the Apply Changes API - https://docs.databricks.com/en/delta-live-tables/cdc.htmlThis is a replacement of MERGE INTO in Databricks.Cheers!
- 0 kudos
- 1474 Views
- 1 replies
- 0 kudos
Delta Live Tables: large use
Does anyone use Delta Live Table on large scale in production pipelines ? Are they satisfied with the product ?Recently, I´ve started a PoC to evaluate the DLT and notice some concerns:- Excessive use of compute resources when you check the cluster m...
- 1474 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Taja,I agree that DLT pipelines doesn't accept a single node cluster to begin with but you can always choose the instance type for both your driver and the worker nodes.As far as `waiting for resources` time is concerned, I've seen that DLT takes...
- 0 kudos
- 2412 Views
- 3 replies
- 0 kudos
DELTA_INVALID_SOURCE_VERSION issue on spark structure streaming
I am doing a structure streaming and getting this error on databricks, the source table already have 2 versions(0,1). It is still not able to find Query {'_id': UUID('fe7a563e-f487-4d0e-beb0-efe794ab4708'), '_runId': UUID('bf0e94b5-b6ce-42bb-9bc7-15...
- 2412 Views
- 3 replies
- 0 kudos
- 0 kudos
Are you using checkpoints? If so, make sure the permisions to that location are ok, alternatively delete all the checkpoints, you have created in that location and try again. This was my case.
- 0 kudos
- 750 Views
- 0 replies
- 1 kudos
Data Engineering Journey on Databricks
For any new Data Engineering aspirant, it has always been a difficult where to start the learning journey. I faced this challenge a decade ago. In order to help new aspirants I created a series of medium article for new learners. I hope it brings mor...
- 750 Views
- 0 replies
- 1 kudos
- 5654 Views
- 3 replies
- 1 kudos
Resolved! Get job ID from Asset Bundles
When using Asset Bundles to deploy jobs, how does one get the job ID of the resources that are created?I would like to deploy some jobs through asset bundles, get the job IDs, and then trigger these jobs programmatically outside the CI/CD pipeline us...
- 5654 Views
- 3 replies
- 1 kudos
- 1 kudos
Refer this answer and this can be a solution to above scenario -> https://community.databricks.com/t5/data-engineering/getting-job-id-dynamically-to-create-another-job-to-refer-as-job/m-p/102860/highlight/true#M41252
- 1 kudos
- 2652 Views
- 1 replies
- 0 kudos
Unable to convert to date from datetime string with AM and PM
Any help to understand why it's showing 'null' instead of the date value? It's showing null only for 12:00:00 AM and for any other values it's showing date correctlyTO_DATE("12/30/2022 12:00:00 AM", "MM/dd/yyyy HH:mm:ss a") AS tsDate
- 2652 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @David_Billa, Can you try with: TO_TIMESTAMP("12/30/2022 12:00:00 AM", "MM/dd/yyyy hh:mm:ss a") AS tsDate The issue you are encountering with the TO_DATE function returning null for the value "12:00:00 AM" is likely due to the format string not ma...
- 0 kudos
- 40979 Views
- 7 replies
- 13 kudos
How to convert string to datetime with correct timezone?
I have a field stored as a string in the format "12/30/2022 10:30:00 AM"If I use the function TO_DATE, I only get the date part... I want the full date and time.If I use the function TO_TIMESTAMP, I get the date and time, but it's assumed to be UTC, ...
- 40979 Views
- 7 replies
- 13 kudos
- 13 kudos
use from_utc_timestamp(to_timestam("<string>", <format>),<timezone>)
- 13 kudos
- 4843 Views
- 3 replies
- 0 kudos
Resolved! DLT: Schema mismatch error
HiI am encountering the following error when writing a DLT pipeline. Here is my workflow:Read a bronze delta tableCheck Data Quality RulesWrite clean records to a silver table with defined schema. I use TRY_CAST for columns where there is mismatch be...
- 4843 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @Svish ,You have one line that differs:JOB_CERTREP_CONTRACT_INT: string (nullable = true) vs. JOB_CERTREP_CONTRACT_NUMBER: string (nullable = true)
- 0 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 584 | |
| 349 | |
| 287 |
