- 984 Views
- 1 replies
- 0 kudos
Not able to login to Partner Academy
I am not able to login to databricks partner academy. Also I raised a ticket (ticket id: 00670650) for the support team. Can someone please help me in resolving the issue?
- 984 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @abhishekv5055! The support team takes around 24-48 hours to review and respond to submitted tickets. They will follow up with you directly through the ticket you’ve raised. We appreciate your patience in the meantime.
- 0 kudos
- 2022 Views
- 3 replies
- 0 kudos
Databricks Asset Bundles Firewall Issue
HelloI'm trying to use Databricks Asset Bundles within a network that has limited access to the internet. When I try to deploy I get the error message "error downloading Terraform: Get "https://releases.hashicorp.com/terraform/1.5.5/index.json". Is...
- 2022 Views
- 3 replies
- 0 kudos
- 0 kudos
HelloAfter testing the following urls need to be whitelisted on the firewall... - [Terraform Registry](https://registry.terraform.io) - [Terrafrom Checkpoint API](https://checkpoint-api.hashicorp.com) - [Terraform Releases](https://releases.hashi...
- 0 kudos
- 510 Views
- 1 replies
- 0 kudos
S3 Private Connection from Databricks Serverless Workspace
I'm looking for reference to privately connect to an S3 bucket from a Serverless Workspace deployed on the same region.
- 510 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @mgcasas! To enable private access from a Serverless Workspace to an S3 bucket in the same region, you can use an AWS Gateway Endpoint. Specifically, create an S3 Gateway Endpoint in your AWS VPC to allow direct and secure connectivity without ...
- 0 kudos
- 1086 Views
- 2 replies
- 0 kudos
IP address configuration for databricks workspace
Hi All Im running the below code in notebook but always returns 400 error even though everything seems correct, Can you please help me how to see the current IP address list or enable the IP access list for the databricks workspaceyour help will be a...
- 1086 Views
- 2 replies
- 0 kudos
- 0 kudos
HI @Shua42 - I tried installing the CLI and tried providing the necessary commands from the below documentation, but workspace commands are not working as workspace -conf is not working , can you help here Link - Configure IP access lists for worksp...
- 0 kudos
- 2548 Views
- 5 replies
- 1 kudos
Creating an Azure-Keyvault-backed secret scope with terraform
We want to create an Azure-Keyvault-backed secret scope with terraform - while we are able to do it via the UI with the URL https://adb-xxxxxxxx.x.azuredatabricks.net/?o=xxxxxxxxxxxxxx#secrets/createScope, I'm unable to do it with Terraform. resourc...
- 2548 Views
- 5 replies
- 1 kudos
- 1 kudos
I am also having the same issue. I am deploying the Azure backed secrets across 3 different workspaces in my TF deployment. Strangley enough it works for 2/3 of my deployments but then I get the same error: Scope with Azure KeyVault must have userAAD...
- 1 kudos
- 2792 Views
- 1 replies
- 0 kudos
NoSuchMethodError with Delta-spark and Databricks Runtime 16.4 LTS
I'm running into an exception when trying to run a Java Spark Jar using the delta-spark library as a job on a Databricks Runtime 16.4 LTS cluster on Azure. I've tried various versions of the Delta Spark library from 3.0.0 to the latest 3.3.1, but alw...
- 2792 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @dperkins The error is likely due to a mismatch between your delta-spark library and the Databricks Runtime. The runtime uses Spark 3.5.2 and Scala 2.13 (with Delta Lake 3.3.1), so ensure your JAR is built with these versions.It’s best to remove t...
- 0 kudos
- 667 Views
- 1 replies
- 0 kudos
Delta Live Table - Delta Table Sink Error after the first run
Hello,I'm trying to create a delta table sink to store delete requests coming to our system that are ingested in the bronze layer successfully with an autoloader. As I want to have a delete control table that needs to be updated after the data deleti...
- 667 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Luca_dall , It looks like this is trying to create the delta_sink_flow table each time the pipeline is run, where the creation of it should be managed by DLT. You can try removing the create_sink call and just run the append_flow to handle the ta...
- 0 kudos
- 3679 Views
- 4 replies
- 0 kudos
best practice for optimizedWrites and Optimize
What is the best practice for a delta pipeline with very high throughput to avoid small files problem and also reduce the need for external OPTIMIZE frequently?
- 3679 Views
- 4 replies
- 0 kudos
- 0 kudos
Hi All,Can anyone who has solved this challenge confirm if the below increases write latency and avoids creating smaller file, based a POC I did, I dont see that behaviour replicable, so I am just wondering. Many thanks.
- 0 kudos
- 1010 Views
- 2 replies
- 1 kudos
Resolved! Avoiding metadata information when sending data to GCS
Hi all,I have use case where I need to push the table data to GCS bucket,query = "${QUERY}" df = spark.sql(query) gcs_path = "${GCS_PATH}" df.write.option("maxRecordsPerFile", int("${MAX_RECORDS_PER_FILE}")).mode("${MODE}").json(gcs_path)This can ...
- 1010 Views
- 2 replies
- 1 kudos
- 1 kudos
Thanks a lot @cgrant . This removed '_started_...' , '_committed_..', but still generated _SUCCESS file.spark.conf.set("mapreduce.fileoutputcommitter.marksuccessfuljobs", "false")this removed _SUCCESS files also.
- 1 kudos
- 1970 Views
- 5 replies
- 1 kudos
Resolved! Trigger bad records in databricks
I use bad records while reading a csv as follows:df = spark.read.format("csv") .schema(schema) .option("badRecordsPath", bad_records_path) Since bad records are not written immediately, I want to know how can trigger the write...
- 1970 Views
- 5 replies
- 1 kudos
- 1 kudos
I found the problem why the code didn't trigger the bad records write. I did empty the folder for bad records. After fixing that, it works. Thanks for the help Isi data_frame.write.format("delta").option("optimizeWrite", "true").mode( "ov...
- 1 kudos
- 1493 Views
- 3 replies
- 0 kudos
databricks_sql_connector not connecting
Hi All,I am trying to connect to sql warehouse using a databricks_oauth auth type using databricks-sql-connector.from databricks.sql import connect conn = connect( server_hostname="https://adb-xxxxxxxxxxxxxx.azuredatabricks.net/", http_path=...
- 1493 Views
- 3 replies
- 0 kudos
- 0 kudos
@Upendra_Dwivedi Have you installed the databricks-sdk dependency?https://docs.databricks.com/aws/en/dev-tools/python-sql-connector
- 0 kudos
- 12458 Views
- 6 replies
- 5 kudos
How I could read the Job id, run id and parameters in python cell?
I have tried following ways to get job parameters but none of the things are working.runId='{{run_id}}' jobId='{{job_id}}' filepath='{{filepath}}' print(runId," ",jobId," ",filepath) r1=dbutils.widgets.get('{{run_id}}') f1=dbutils.widgets.get('{{file...
- 12458 Views
- 6 replies
- 5 kudos
- 5 kudos
You should use {{job.id}} and {{job.run_id}} instead of with an underscore. This works for me.
- 5 kudos
- 979 Views
- 2 replies
- 1 kudos
Resolved! DELTA_TXN_LOG_FAILED_INTEGRITY
Hi, Trying to use merge statement for a query and the error comes as shown below. I am able to describe history on the table but not able to perform any operations like vaccum, restore, optimize and even merge. Tried dropping the external delta tabl...
- 979 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @ar45 , I am not sure what caused the corruption, but to resolve the issue, you can try removing the transaction log for the corrupt version 4574 (.json file for version 4574 under the _delta_log folder). If there are .crc files for version 4574, ...
- 1 kudos
- 2096 Views
- 4 replies
- 0 kudos
Internal errors when running SQLs
We are running Databricks on GCP with a classic SQL warehouse. Its on the current version (v 2025.15)We have a pipeline that runs DBT on top of the SQL warehouseSince the 9th of May, our queries have been failing intermittently with internal errors f...
- 2096 Views
- 4 replies
- 0 kudos
- 0 kudos
Hi @utkarshamone ,We faced a similar issue and I wanted to share our findings, which might help clarify what’s going on.We’re using a Classic SQL Warehouse size L (v2025.15), and executing a dbt pipeline on top of it.Our dbt jobs started to fail with...
- 0 kudos
- 7904 Views
- 4 replies
- 1 kudos
Resolved! How to install a JAR library via a global init script?
I have a JAR I want to be installed as a library on all clusters. I have tried both wget /databricks/jars/ some_repoandcp /dbfs/FileStore/jars/name_of_jar.jar /databricks/jars/clusters start up but the JAR is not installed as a library. I am aware th...
- 7904 Views
- 4 replies
- 1 kudos
- 1 kudos
Ensure your init script installs the JAR correctly on cluster startup.
- 1 kudos
| User | Count |
|---|---|
| 1632 | |
| 791 | |
| 512 | |
| 349 | |
| 287 |
