- 1436 Views
- 1 replies
- 0 kudos
How to copy file from UC volume to external location folder
Hi there, How to copy file from UC volume to external location folder Thanks&Regards, zmsoft
- 1436 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @zmsoft! To copy a file from a UC volume to an external location, you can use: dbutils.fs.cp( "UC_volume_path", "external_location_path" ) Ensure the external location is preconfigured in Unity Catalog and you have the necessary permission...
- 0 kudos
- 2189 Views
- 4 replies
- 0 kudos
Schema update Issue in DLT
I have a pipeline in databricks with this flowSQL SERVER (Source) -> Staging (Parquet) -> Bronze (DLT) -> Silver(DLT) -> Gold (DLT)The pipeline is up and running smoothly for months but recently, there was a schema update at my source level and one o...
- 2189 Views
- 4 replies
- 0 kudos
- 0 kudos
Hi @Alberto_Umana, is there any word on how to fix my data and bring all the records back to the pipeline schema?
- 0 kudos
- 1808 Views
- 1 replies
- 0 kudos
Resolved! How do I use the azure databricks dlt pipeline to consume azure Event Center data
Hi there, How do I use the azure databricks dlt pipeline to consume azure Event Hub dataCode :TOPIC = "myeventhub" KAFKA_BROKER = "" GROUP_ID = "group_dev" raw_kafka_events = (spark.readStream .format("kafka") .option("subscribe", EH_NAME) .opt...
- 1808 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi there @zmsoft ,Did you have a look at this ref doc : https://docs.databricks.com/aws/en/dlt/event-hubsThis might help
- 0 kudos
- 1165 Views
- 1 replies
- 0 kudos
Getting Errors when reading data from Excel InternalError: pip is not installed for /local_disk
Hi all,We have a daily Databricks job that downloads excel files from SharePoint and read them, the job works fine until today (3March). We are getting the following error message when running the code to read the excel file:org.apache.spark.SparkExc...
- 1165 Views
- 1 replies
- 0 kudos
- 0 kudos
I think the issue comes from installing Office365-REST-Python-Client using dbutils.library.installPyPI, which seems to create a conflicting Python environment for Spark executors. Since notebook specific installs modify the environment dynamically, t...
- 0 kudos
- 1046 Views
- 1 replies
- 0 kudos
ETL Pipeline work fine, but when executed via Workflow it fails due to StorageAccessError
I have a fairly simple ETL Pipeline that uses dlt. It streams data from ADLS2 SA and creates materialized view using two tables. It works fine when i execute it on its own. Materialized view is properly refreshed.Now I wanted to add this as a task to...
- 1046 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @xx123,Could you please provide more details on the cluster configuration?. I am guessing the cluster policy you might be using for deploying the job in workflow / and when you are testing might be different. Please try to use same cluster policy ...
- 0 kudos
- 2871 Views
- 3 replies
- 0 kudos
I've data in s3/Iceberg tables. How to read it using databricks SparkSQL ?
I tried this method: df = spark.read.format("iceberg").load("s3-bucket-path")But got an error: Multiple sources found for iceberg (com.databricks.sql.transaction.tahoe.uniform.sources.IcebergBrowseOnlyDataSource, org.apache.iceberg.spark.source.Icebe...
- 2871 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @Alberto_Umana, I tried the steps you've provided but still I'm not able to read data which is in iceberg format. It would be useful if I get any other suggestions.
- 0 kudos
- 4780 Views
- 1 replies
- 0 kudos
Databricks Managed RG Storage cost is High
Hi Community,How to calculate the databricks storage cost and where to see the data which is stored and charged in databricks.I'm trying to understand the storage cost on a managed resource group and i'm clueless about the data and where it is stored...
- 4780 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi,How are you doing today? To understand Databricks storage costs in Azure, you can check where your data is stored and how it’s being charged. Managed tables, DBFS files, and Unity Catalog volumes are usually stored in an Azure Data Lake Storage (A...
- 0 kudos
- 1151 Views
- 1 replies
- 1 kudos
Need to run update statement from databricks using azure sql pyodbc connection
Hi All, I was Trying to run the update statement in data bricks notebook using pyodbc connection. while i was doing I was getting following error. I need assistance to solve this.Error: ('01000', "[01000] [unixODBC][Driver Manager]Can't open lib 'ODB...
- 1151 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi Narendra,How are you doing today? As per my understanding, It looks like your Databricks notebook can't find the ODBC Driver 17 for SQL Server. You can first check if the driver is installed by running !odbcinst -q -d in a notebook cell. If it's m...
- 1 kudos
- 2693 Views
- 3 replies
- 0 kudos
Resolved! Delta Live Tables are refreshed in parallel rather than sequentially
Hi experts,I have defined my DLT Pipeline as follows:-- Define a streaming table to ingest data from a volume CREATE OR REFRESH STREAMING TABLE pumpdata_bronze TBLPROPERTIES ("myCompanyPipeline.quality" = "bronze") AS SELECT * FROM cloud_files("abfss...
- 2693 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @BobCat62 ,So the thing is Now dlt has different modes dlt direct publishing mode , classic mode(legacy). Look here for mode details : https://docs.databricks.com/aws/en/release-notes/product/2025/january#dlt-now-supports-publishing-to-tables-in-m...
- 0 kudos
- 5507 Views
- 5 replies
- 1 kudos
databricks asset bundles: Unable to fetch variables from variable-overrides.json
Hi,I am using Databricks CLI 0.227.1 for creating a bundle project to deploy job.As per this , https://learn.microsoft.com/en-us/azure/databricks/dev-tools/bundles/variables I wanted to have variable-overrides.json to have my variables.I created a js...
- 5507 Views
- 5 replies
- 1 kudos
- 1701 Views
- 2 replies
- 1 kudos
Resolved! Keyvault reference for federated connection setup - Azure
I am trying to create a federated connection in unity catalog for an Oracle Database. The connection configuration GUI seems to ask for the password. Is it possible to put a keyvault reference here instead?  
- 1701 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @yorkuDE01,I suppose this could be done when you are trying to create / setup federated connection using API. But, I don't think so this could be possible via UI, where you can reference a key-vault scoped secret.But please refer the documentation...
- 1 kudos
- 5734 Views
- 8 replies
- 1 kudos
Cant read/write tables with shared cluster
Hi!I have a pipeline that i cant execute sucessfully in a shared cluster. Basically i read a query from multiple sources on my databricks instance, including streaming tables (thats the reason i have to use a shared cluster).But when comes to the par...
- 5734 Views
- 8 replies
- 1 kudos
- 1 kudos
Hi @Ramonrcn,If I understand your question, you should need to have MODIFY / ALL PRIVILEGES permission on top of the table inorder to drop or modify a table. And if you are performing this change using Managed Identity / IAM, the same permission ment...
- 1 kudos
- 5818 Views
- 1 replies
- 0 kudos
Databricks is not converting my .ipynb file to .py
I can't find a way to have Databricks automatically convert my python notebooks to .py files when committing in a Git directory. This seems to counter the documentation I've read in that it should automatically convert. I've double checked that my se...
- 5818 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @megan,The default format for notebooks in Databricks workspace now is .ipynb and not .py as mentioned in the image below. And you can navigate to settings -> Developer - > Default file format for notebooks and you can set the default format acros...
- 0 kudos
- 1042 Views
- 1 replies
- 0 kudos
Stream-stream join using MongoDB sink
I am performing stream-to-stream join in Databricks using MongoDB as a source (readStream()). Both sources collections receive data at same time. Initially I tried with using watermarks orderWithWatermark = order \ .selectExpr("order_id AS orderId",...
- 1042 Views
- 1 replies
- 0 kudos
- 0 kudos
There is not enough information in this high-level error message. Please expand the full stacktrace and feel free to post it here
- 0 kudos
- 1412 Views
- 1 replies
- 0 kudos
How can we import the exception "MetadataChangedException"?
I regularly getMetadataChangedException: [DELTA_METADATA_CHANGED] MetadataChangedException: The metadata of the Delta table has been changed by a concurrent update. Please try the operation again.What is the recommended way to import this specific ty...
- 1412 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi! It depends on whether you're using Scala or Python.If you're using Scala, you should be able to import `io.delta.exceptions.MetadataChangedException`, which you can see defined here: https://github.com/delta-io/delta/blob/master/spark/src/main/sc...
- 0 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 584 | |
| 349 | |
| 287 |
