- 10318 Views
- 10 replies
- 2 kudos
ConcurrentAppendException After Delta Table was enable Liquid Clustering and Row level concurrency
Everytime I run parallel job it always failed with this error: ConcurrentAppendException: Files were added to the root of the table by a concurrent update. Please try the operation again.I did a lot of reseaches also create liquid clustering table an...
- 10318 Views
- 10 replies
- 2 kudos
- 2 kudos
Hi, Any proper or permanent solution to this?I too faced this problem, but handled it through retry on ConcurrentAppendException!Isn't ForEachBatch hard-coded to writeSerializable? And is it really possible to avoid ConcurrentAppendException when two...
- 2 kudos
- 18181 Views
- 8 replies
- 0 kudos
using databricks sql warehouse as web app backend
Hi,I wanted to ask if anyone is using Databricks SQL Warehouse as backend for small to large scale web application? What are your thoughts about it, specially what Databricks team thinks of it ?Kind Regards,A
- 18181 Views
- 8 replies
- 0 kudos
- 0 kudos
Yes, Databricks SQL Warehouse can be used as a web app backend by exposing it through REST APIs or JDBC/ODBC to serve queries and data to your application.
- 0 kudos
- 2731 Views
- 1 replies
- 0 kudos
Resolved! Databricks authentication within cluster running custom docker container
I have setup an asset bundle that deploys a job, with a job cluster within a pool, utilising a Docker image.The docker image pulls and the dependencies install which was as battle in itself, but now I'm struggling to get a Databricks WorkspaceClient ...
- 2731 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @fleetwoodmatt Thanks for the detailed context. Recommended Approach: Cluster-Scoped Service Principal via Spark Config Store secrets in Databricks Secret ScopeInject secrets as cluster env vars in your bundle spark_env_vars:DATABRICKS_HOST: "...
- 0 kudos
- 1049 Views
- 1 replies
- 0 kudos
Resolved! Entity Matching on struct column
Databricks Genie allows configurable prompt matching, for example entity matching:Entity matching provides curated lists of distinct values for up to 120 columns where users are likely to reference specific entries, such as states and product categor...
- 1049 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Malthe , Entity matching supports simple data types only (no structs). The recommended approach is to flatten the table structure before passing it to Genie. Hope it helps,. Best regards,
- 0 kudos
- 9609 Views
- 8 replies
- 2 kudos
ODBC on Windows -- Where to specify Catalog name?
We are attempting to connect a Windows ODBC application to Unity Catalog.The Configure the Databricks ODBC and JDBC drivers documentation has a section titled "ODBC configuration and connection parameters" that mentions a configuration parameter call...
- 9609 Views
- 8 replies
- 2 kudos
- 2 kudos
Open the ODBC DSN Setup, go to Advanced Options, and define databricks.catalog=<your_catalog_name> under Server Side Properties.
- 2 kudos
- 1579 Views
- 1 replies
- 0 kudos
Resolved! Implement waiting logic in Databricks jobs
I want to build a job or a set of jobs that load data from several different source systems (with approximately 600 tasks in total). Data from the source systems arrive at different points in time every day, and a query needs to be executed to see if...
- 1579 Views
- 1 replies
- 0 kudos
- 0 kudos
below wud be my approach rather than making on big monolithic job , cost and not maintainable at all.Each source system has its own ingestion job (or a pipeline).Each ingestion job runs on a schedule or via trigger. we can also make use of autoloader...
- 0 kudos
- 936 Views
- 2 replies
- 1 kudos
Resolved! job id fetching in parent job
hi,i want to fetch job id of child job which executes its own workflow ,into parent job which executes its own workflow but depends on job child jobs job id?how to do that?
- 936 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @bricksuser looks like a common fan-out orchestration patter. your parent job should triger child job programatically and then capture maintain its run_id. you can use databricks jobs api endpoints to start child job.Use POST /api/2.1/jobs/run-now...
- 1 kudos
- 7914 Views
- 5 replies
- 3 kudos
Resolved! SparkContext lost when running %sh script.py
I need to execute a .py file in Databricks from a notebook (with arguments which for simplicity i exclude here). For this i am using:%sh script.pyscript.py:from pyspark import SparkContext def main(): sc = SparkContext.getOrCreate() print(sc...
- 7914 Views
- 5 replies
- 3 kudos
- 3 kudos
hey @madrhr ..i am checking on this...its not working for me. Can u help with screenshots of implementation
- 3 kudos
- 2103 Views
- 7 replies
- 1 kudos
Resolved! copy into - statement executed - no records inserted
I am able to run the copy command but records are not inserting and there is no error raised on execution.
- 2103 Views
- 7 replies
- 1 kudos
- 1 kudos
Records not inserted with the COPY INTO statement usually indicate either no matching source data or unmet conditions in the query.
- 1 kudos
- 5157 Views
- 3 replies
- 1 kudos
Resolved! Spark UI simulator is not available online
About 2 weeks ago I started course on "Optimizing Apache Spark on Databricks" from official Databricks academy. It is heavily based on Spark UI simulator experiments that were available here: https://www.databricks.training/spark-ui-simulator and for...
- 5157 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi @LandanG The link is still not working. Any idea when it will be live again?http://www.databricks.training/spark-ui-simulator/GitHub - databricks-academy/spark-ui-simulator: Apache Spark UI simulator with 30+ educational experiments, pre-recorded ...
- 1 kudos
- 8559 Views
- 3 replies
- 3 kudos
Resolved! Delta live tables - foreign keys
I'm creating ingestions using delta live tables, the dlt support the use of schema, with constraints like foreign keys. The problem is: how can i create foreign keys between the same pipeline, that has no read/write relation, but has foreign key rela...
- 8559 Views
- 3 replies
- 3 kudos
- 3 kudos
Was there any update from Databricks to @User12350 comment above? We have the exact issue - you cannot add foreign key constraint on materialized view (even if its not enforced).
- 3 kudos
- 1085 Views
- 2 replies
- 0 kudos
Resolved! Which of my Lakeflow pipelines are using the same gateway?
Does anyone know of a way to see what Lakeflow pipelines are using the same gateway? We have a gateway connected to a SQL Server that serves multiple individual pipelines but I cannot find a way to see what those are. I've tried system tables. Any in...
- 1085 Views
- 2 replies
- 0 kudos
- 0 kudos
Thanks @szymon_dybczak - this worked perfectly. This is what I ended up throwing together to see what I was looking for:from databricks.sdk import WorkspaceClient url = f"{DATABRICKS_HOST}/api/2.0/pipelines" headers = wc.config.authenticate() wc = ...
- 0 kudos
- 15340 Views
- 8 replies
- 7 kudos
Reading Iceberg table present in S3 from databricks console using spark given none error .
Hi Team , I am facing issue while reading iceberg table from S3 and getting none error when read the data . below steps I followed .Added Iceberg Spark connector library to your Databricks cluster. 2. Cluster Configuration to Enable Iceberg ...
- 15340 Views
- 8 replies
- 7 kudos
- 7 kudos
https://community.databricks.com/t5/technical-blog/secure-migration-of-aws-glue-iceberg-tables-to-databricks-unity/ba-p/148414
- 7 kudos
- 1096 Views
- 4 replies
- 1 kudos
Resolved! Table listener
Hello Community,I would like to ask whether it’s possible to define a job that checks for updates in a table at a specified frequency.Here is my use case:Data is uploaded to a table located in Catalog A, Schema B, Table C (a.b.c).I need to transfer t...
- 1096 Views
- 4 replies
- 1 kudos
- 1 kudos
Also I have a question about failure handling in case of triggered jobs. Let's say new data has come to the source table and job failed for any reason. If I rerun it or the next batch of data comes to source table, will the data from the failed job b...
- 1 kudos
- 1573 Views
- 1 replies
- 3 kudos
Resolved! Asset Bundles: Extending complex variables
Hi all,In my Asset Bundle I have some setting for a cluster like in the example at Substitutions and variables in Databricks Asset Bundles (section Define a complex variable).Now I want to add some additional attribute when using this variables, like...
- 1573 Views
- 1 replies
- 3 kudos
- 3 kudos
Hi @Daniel_dlh ,you can try to use YAML anchor, have a look at this example:before: databricks.yml variables: my_cluster: description: "Base cluster configuration" default: spark_version: "15.4.x-scala2.12" node_type_id: "Standa...
- 3 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 586 | |
| 349 | |
| 287 |
