- 4313 Views
- 5 replies
- 1 kudos
Resolved! Trigger bad records in databricks
I use bad records while reading a csv as follows:df = spark.read.format("csv") .schema(schema) .option("badRecordsPath", bad_records_path) Since bad records are not written immediately, I want to know how can trigger the write...
- 4313 Views
- 5 replies
- 1 kudos
- 1 kudos
I found the problem why the code didn't trigger the bad records write. I did empty the folder for bad records. After fixing that, it works. Thanks for the help Isi data_frame.write.format("delta").option("optimizeWrite", "true").mode( "ov...
- 1 kudos
- 2418 Views
- 3 replies
- 0 kudos
databricks_sql_connector not connecting
Hi All,I am trying to connect to sql warehouse using a databricks_oauth auth type using databricks-sql-connector.from databricks.sql import connect conn = connect( server_hostname="https://adb-xxxxxxxxxxxxxx.azuredatabricks.net/", http_path=...
- 2418 Views
- 3 replies
- 0 kudos
- 0 kudos
@Upendra_Dwivedi Have you installed the databricks-sdk dependency?https://docs.databricks.com/aws/en/dev-tools/python-sql-connector
- 0 kudos
- 14382 Views
- 6 replies
- 5 kudos
How I could read the Job id, run id and parameters in python cell?
I have tried following ways to get job parameters but none of the things are working.runId='{{run_id}}' jobId='{{job_id}}' filepath='{{filepath}}' print(runId," ",jobId," ",filepath) r1=dbutils.widgets.get('{{run_id}}') f1=dbutils.widgets.get('{{file...
- 14382 Views
- 6 replies
- 5 kudos
- 5 kudos
You should use {{job.id}} and {{job.run_id}} instead of with an underscore. This works for me.
- 5 kudos
- 1688 Views
- 2 replies
- 1 kudos
Resolved! DELTA_TXN_LOG_FAILED_INTEGRITY
Hi, Trying to use merge statement for a query and the error comes as shown below. I am able to describe history on the table but not able to perform any operations like vaccum, restore, optimize and even merge. Tried dropping the external delta tabl...
- 1688 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @ar45 , I am not sure what caused the corruption, but to resolve the issue, you can try removing the transaction log for the corrupt version 4574 (.json file for version 4574 under the _delta_log folder). If there are .crc files for version 4574, ...
- 1 kudos
- 3794 Views
- 4 replies
- 0 kudos
Internal errors when running SQLs
We are running Databricks on GCP with a classic SQL warehouse. Its on the current version (v 2025.15)We have a pipeline that runs DBT on top of the SQL warehouseSince the 9th of May, our queries have been failing intermittently with internal errors f...
- 3794 Views
- 4 replies
- 0 kudos
- 0 kudos
Hi @utkarshamone ,We faced a similar issue and I wanted to share our findings, which might help clarify what’s going on.We’re using a Classic SQL Warehouse size L (v2025.15), and executing a dbt pipeline on top of it.Our dbt jobs started to fail with...
- 0 kudos
- 9255 Views
- 4 replies
- 1 kudos
Resolved! How to install a JAR library via a global init script?
I have a JAR I want to be installed as a library on all clusters. I have tried both wget /databricks/jars/ some_repoandcp /dbfs/FileStore/jars/name_of_jar.jar /databricks/jars/clusters start up but the JAR is not installed as a library. I am aware th...
- 9255 Views
- 4 replies
- 1 kudos
- 1 kudos
Ensure your init script installs the JAR correctly on cluster startup.
- 1 kudos
- 2829 Views
- 1 replies
- 0 kudos
JVM Heap Leak When Iterating Over Large Number of Tables Using DESCRIBE DETAIL
Problem:I'm trying to generate a consolidated metadata table for all tables within a Databricks database (I do not have admin privileges). The process works fine for the first few thousand tables, but as it progresses, the driver node eventually cras...
- 2829 Views
- 1 replies
- 0 kudos
- 0 kudos
It's best to iterate over information_schema's TABLES table instead of listing yourself.
- 0 kudos
- 890 Views
- 1 replies
- 0 kudos
Data Engineering Associate -13+ Years of SAP SD/OTC Experience -Data Engineering Associate
Hi DataBricks ,This is Chandra ,I am adapting the world of data with the help of Data bricks .I need your help and advises to successfully get adapt the Databricks Engineer profile approach .I have enrolled myself in the Learning platform ,I need yo...
- 890 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @ChandraR! Happy to help you get started on your Databricks journey! To begin, it's important to get familiar with the Databricks ecosystem, including key components like the Lakehouse architecture, Delta Lake, Apache Spark, and Unity Catalog. ...
- 0 kudos
- 2122 Views
- 1 replies
- 0 kudos
Column Name Case sensitivity in DLT pipeline
I've a DLT pipeline that processes messages from event grid. The schema of the message has two columns in different cases - "employee_id" and "employee_ID",I tried setting spark.sql.caseSensitive to true in my DLT notebook as well in DLT configurati...
- 2122 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @AmanSehgal, DLT treat column names as case-insensitive, even if spark.sql.caseSensitive is set to true. That’s why employee_id and employee_ID are seen as duplicates and cause the error. To fix this, you’ll need to rename one of the columns so yo...
- 0 kudos
- 983 Views
- 1 replies
- 0 kudos
Resolved! Introduction to Spark Lab
Hello, I got an error while accessing the Introduction to Spark Lab. Please see the error message below and resolve.", line 155, in do response = retryable(self._perform)(method, File "/voc/scripts/python/venv/lib/python3.10/site-packages/databricks/...
- 983 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @sunday-okey! Apologies for the inconvenience.The issue has been resolved. Please try restarting the lab, it should be working as expected now.
- 0 kudos
- 2423 Views
- 4 replies
- 0 kudos
path-based access to a table with row filters or column masks is not supported
I have a delta table which I am applying masking to some columns, however, every time I want to refresh the table (overwrite) I cannot I receive this error:If I do what Assistant recommend me (If you remove the .option("path", DeltaZones))It worked b...
- 2423 Views
- 4 replies
- 0 kudos
- 844 Views
- 1 replies
- 0 kudos
I got suspended from the Databrick certified Associate developer for Apache Spark.
I need immediate assistance to reschedule my exam. By mistakenly I have taken note book to do rough work. Due to using notebook i got suspended from the exam i think so. Please help me out with this issue.mail id- malothsiddunaik133@gmail.comThankyou...
- 844 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @Siddartha01! It looks like this post duplicates the one you recently posted. A response has already been provided to the Original thread. I recommend continuing the discussion in that thread to keep the conversation focused and organised.
- 0 kudos
- 1607 Views
- 1 replies
- 1 kudos
Resolved! Delta sharing recipient auth status
I'm creating recipients, and send them activation link via email. All recipients are external (they don't have databricks account). Lets say I've created 300 recipients and I want to know who downloaded creds fail successfully, and got authenticated....
- 1607 Views
- 1 replies
- 1 kudos
- 1 kudos
Hey @drag7ter You’re absolutely right, currently, there is no official API field that explicitly returns the recipient activation status (I just have tested it), even though the Databricks API documentation references a field called activated boolea...
- 1 kudos
- 7089 Views
- 7 replies
- 6 kudos
dlt Streaming Checkpoint Not Found
I am using Delta Live Tables and have my pipeline defined using the code below. My understanding is that a checkpoint is automatically set when using Delta Live Tables. I am using the Unity Catalog and Schema settings in the pipeline as the storage d...
- 7089 Views
- 7 replies
- 6 kudos
- 3066 Views
- 4 replies
- 1 kudos
Resolved! Define time interval for when a cluster can be active
Hullo good Databricks people.I have a small dedicated cluster being used for Direct Query (PBI) which has a long termination period. I'd like for it to only be active during business hours though, and to set a restriction so that it's not possible to...
- 3066 Views
- 4 replies
- 1 kudos
- 1 kudos
So the hybrid solution turned out to be the smartest, at least for my case, with a simple time check that defines the connection information according to whether the query is sent outside business hours or not.
- 1 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 581 | |
| 349 | |
| 287 |
