- 65276 Views
- 21 replies
- 7 kudos
Resolved! Is it possible to get Job Run ID of notebook run by dbutils.notbook.run?
When running a notebook using dbutils.notebook.run from a master-notebook, an url to that running notebook is printed, i.e.: Notebook job #223150 Notebook job #223151 Are there any ways to capture that Job Run ID (#223150 or #223151)? We have 50 or ...
- 65276 Views
- 21 replies
- 7 kudos
- 7 kudos
Thanks for the response @Manoj5 - I had to use this "safeToJson()" option too because all of the previous suggestions in this thread were erroring out for me with a message like "py4j.security.Py4JSecurityException: Method public java.lang.String com...
- 7 kudos
- 3435 Views
- 2 replies
- 2 kudos
Resolved! Column comments in DLT python notebook
The SQL API specification in the DLT docs shows an option for adding column comments when creating a table. Is there an equivalent way to do this when creating a DLT pipeline with a python notebook? The Python API specification in the DLT docs does n...
- 3435 Views
- 2 replies
- 2 kudos
- 2 kudos
this works in a readStream writeStream scenario for us, but the exact same code fails when put in a DLT
- 2 kudos
- 10 Views
- 1 replies
- 0 kudos
How to Stop Driver Node from Overloading When Using ThreadPoolExecutor in Databricks
Hi everyone,I'm using a ThreadPoolExecutor in Databricks to run multiple notebooks at the same time. The problem is that it seems like all the processing happens on the driver node, while the executor nodes are idle. This causes the driver to run out...
- 10 Views
- 1 replies
- 0 kudos
- 0 kudos
Greetings @alex307 and thank you for sending your question. When using ThreadPoolExecutor to run multiple notebooks concurrently in Databricks, the workload is being executed on the driver node rather than distributed across Spark executors. This res...
- 0 kudos
- 30 Views
- 2 replies
- 0 kudos
Scaling Declarative Streaming Pipelines for CDC from On-Prem Database to Lakehouse
We have a scenario where we need to mirror thousands of tables from on-premises Db2 databases to an Azure Lakehouse. The goal is to create mirror Delta tables in the Lakehouse.Since LakeFlow Connect currently does not support direct mirroring from on...
- 30 Views
- 2 replies
- 0 kudos
- 0 kudos
Yes, a databricks labs project seems perfect for your scenario. https://databrickslabs.github.io/dlt-meta/index.html
- 0 kudos
- 2388 Views
- 2 replies
- 2 kudos
Best sequence of using Vacuum, optimize, fsck repair and refresh commands.
I have a delta table whose size will increases gradually now we have around 1.5 crores of rows while running vacuum command on that table i am getting the below error.ERROR: Job aborted due to stage failure: Task 7 in stage 491.0 failed 4 times, most...
- 2388 Views
- 2 replies
- 2 kudos
- 2 kudos
In my opinion Best order: Optimize → Vacuum → FSCK Repair → Refresh.Your error is likely a timeout — try more cluster resources or a longer retention period.
- 2 kudos
- 30 Views
- 1 replies
- 0 kudos
DABs with multi github sources
I want to deploy a dabs that has dev using a github branch and prod using a github release tag. I can't seem to find a way to make this part dynamic based on the target. Things I've tried:- Setting the git varaible in the databricks.yml- making the g...
- 30 Views
- 1 replies
- 0 kudos
- 0 kudos
You may want to look into SHA-Based Versioning. For more details, look here: https://towardsdev.com/ci-cd-strategies-for-databricks-asset-bundles-e4aaf921823e
- 0 kudos
- 83 Views
- 3 replies
- 1 kudos
online tables to synced table, why is it creating a different service principal everytime?
Hello!We started to move our online tables to synced_tables. We just couldnt figure out why it is creating a new service principal everytime we ran the same code we use for online tables?try: fe.create_feature_spec(name=feature_spec_name ...
- 83 Views
- 3 replies
- 1 kudos
- 1 kudos
Greetings @hgm251 , here are some things to consider. Things are working as designed: when you create a new Feature Serving or Model Serving endpoint, Databricks automatically provisions a dedicated service principal for that endpoint, and a fresh...
- 1 kudos
- 3382 Views
- 2 replies
- 2 kudos
Resolved! DLT Streaming With Watermark fails, suggesting I should add watermarks
Hi all,I have the following Problem: I have two streaming tables containing time-series measurements from different sensor data, each feed by multiple sensors. (Imagine: Multiple Temperature Sensors for the first table, and multiple humidity sensors ...
- 3382 Views
- 2 replies
- 2 kudos
- 2 kudos
To resolve the DLT streaming aggregation error about unsupported output modes and watermarks in Databricks, you need to carefully set watermarks on the original event timestamp rather than on computed columns like "time_window" and carefully consider...
- 2 kudos
- 3245 Views
- 1 replies
- 0 kudos
Transaction Log Failed Integrity Checks
I have started to receive the following error message - that the transaction log has failed integrity checks - when attempting to optimize and run compaction on a table. It also occurs when I attempt to alter this table.This blocks my pipeline from r...
- 3245 Views
- 1 replies
- 0 kudos
- 0 kudos
Your issue—encountering "the transaction log has failed integrity checks" in Databricks Delta Lake—indicates metadata corruption or an inconsistency in the Delta transaction log (_delta_log). This commonly disrupts DML operations like OPTIMIZE, DELET...
- 0 kudos
- 3624 Views
- 1 replies
- 1 kudos
Streamlit Databricks App Compute Scaling
I have a streamlit Databricks app and I’m looking to increase the compute resources. According to the documentation and the current settings, the app is limited to 2 vCPUs and 6 GB of memory. Is there a way to adjust these limits or add more resource...
- 3624 Views
- 1 replies
- 1 kudos
- 1 kudos
You can increase compute resources for your Streamlit Databricks app, but this requires explicitly configuring the compute size in the Databricks app management UI or via deployment configuration—environment variables like DATABRICKS_CLUSTER_ID alone...
- 1 kudos
- 3297 Views
- 1 replies
- 0 kudos
AI BI Genie throwing internal error
For any prompt I am getting INTERNAL_ERROR: AI service did not respond with a valid answer
- 3297 Views
- 1 replies
- 0 kudos
- 0 kudos
The "INTERNAL_ERROR: AI service did not respond with a valid answer" in Databricks AI/BI Genie typically means the Genie service failed to process your query, often due to one of a few common issues. This can include problems with the table existence...
- 0 kudos
- 3438 Views
- 1 replies
- 0 kudos
Finding all folder paths in a blob store connected via UC external connetion
Hi All,I need to easily find all the paths in a blob store to find the files and load them. I have tried using Azure Blob storage connection in python and I have a solution that works it is very slow. I was speaking to a data engineer, and he suggest...
- 3438 Views
- 1 replies
- 0 kudos
- 0 kudos
The most efficient way to list all file paths in an Azure Blob Storage container from Databricks, especially when Hierarchical Namespace (HNS) is not enabled, is to use Azure SDKs targeting the blob flat namespace directly rather than filesystem prot...
- 0 kudos
- 3597 Views
- 2 replies
- 1 kudos
Debugger freezes when calling spark.sql with dbx connect
I have just created a simple bundle with databricks, and is using Databricks connect to debug locally. This is my script:from pyspark.sql import SparkSession, DataFrame def get_taxis(spark: SparkSession) -> DataFrame: return spark.read.table("samp...
- 3597 Views
- 2 replies
- 1 kudos
- 1 kudos
The issue you're experiencing—where your script freezes in VS Code when running spark.sql locally using Databricks Connect, but works correctly when deployed—can result from several common causes related to Databricks Connect configuration, networkin...
- 1 kudos
- 3542 Views
- 2 replies
- 0 kudos
How to Define Constants at Bundle Level in Databricks Asset Bundles for Use in Notebooks?
I'm working with Databricks Asset Bundles and need to define constants at the bundle level based on the target environment. These constants will be used inside Databricks notebooks.For example, I want a constant gold_catalog to take different values ...
- 3542 Views
- 2 replies
- 0 kudos
- 0 kudos
Yes, you can define environment-specific constants at the bundle level in Databricks Asset Bundles and make them accessible inside Databricks notebooks, without relying on task-level parameters. This can be done using environment variables, bundle co...
- 0 kudos
- 3474 Views
- 1 replies
- 0 kudos
Accessing Databricks Delta table in ADF using system-defined managed identity
I am using Lookup activity in ADF which will read the delta table values from databricks. Currently using the system-defined managed identity of the ADF to connect Databricks delta table. I am unable to see my unity catalog database names in the look...
- 3474 Views
- 1 replies
- 0 kudos
- 0 kudos
You are experiencing an issue in Azure Data Factory (ADF) where the Lookup activity does not show your Unity Catalog databases in the configuration dropdown, even though connectivity from ADF to Databricks is successful and you have followed all reco...
- 0 kudos
| User | Count |
|---|---|
| 1619 | |
| 790 | |
| 483 | |
| 349 | |
| 287 |
