- 2129 Views
- 1 replies
- 0 kudos
Importing data from S3 to Azure DataBricks Cluster with Unity Catalog in Shared Mode
Environment details:DataBricks on Azure, 13.3 LTS, Unity Catalog, Shared Cluster mode.Currently in the environment I'm in, we run imports from S3 with code like:spark.read.option('inferSchema', 'true').json(s3_path). When running on a cluster in Sha...
- 2129 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello Sha, We usually get such error while working with shared cluster mode assuming this your dev environment just to avoid errors, please use different clusters. However as a alternative solution in case if would like to keep the shared cluster the...
- 0 kudos
- 11058 Views
- 4 replies
- 0 kudos
CREATE TABLE does not overwrite location whereas CREATE OR REPLACE TABLE does
I am working on Azure Databricks, with Databricks Runtime version being - 14.3 LTS (includes Apache Spark 3.5.0, Scala 2.12). I am facing the following issue.Suppose I have a view named v1 and a database f1_processed created from the following comman...
- 11058 Views
- 4 replies
- 0 kudos
- 0 kudos
Hi @Dhruv-22 , Based on the information you shared above, the "CREATE OR REPLACE" and "CREATE" commands in Databricks do have different behaviours, particularly when it comes to handling tables with specific target locations. The "CREATE OR REPLACE"...
- 0 kudos
- 11042 Views
- 1 replies
- 2 kudos
REDACTED_POSSIBLE_SECRET_ACCESS_KEY as part of column value result form aes_encrypt
Hi, i've encountered an error using base64/aes_encrypt, as result the string saved contains 'REDACTED_POSSIBLE_SECRET_ACCESS_KEY' at the end destroying the original data, rendering it useless undecryptable, is there a way to avoid this replacement in...
- 11042 Views
- 1 replies
- 2 kudos
- 2 kudos
I had the same issue, and my usage was similar to OP:base64(aes_encrypt(<clear_text>, unbase64(secret(<scope>, <key>))))Databricks support suggested to not call secret within the insert/update operation that writes to the table. After updating the py...
- 2 kudos
- 11530 Views
- 3 replies
- 1 kudos
Resolved! REPLACE TABLE AS SELECT is not working with parquet whereas it works fine for delta
I am working on Azure Databricks, with Databricks Runtime version being - 14.3 LTS (includes Apache Spark 3.5.0, Scala 2.12). I am facing the following issue.Suppose I have a view named v1 and a database f1_processed created from the following comman...
- 11530 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi @Dhruv-22 We understand that you are facing the following error when using REPLACE TABLE AS SELECT on the Parquet Table but at this moment the REPLACE TABLE AS SELECT operation you're trying to perform is not supported for Parquet tables. Accord...
- 1 kudos
- 2617 Views
- 2 replies
- 0 kudos
Near Real time Solutioning on data from Core System which gets updated
We are trying to build solution , where customer data stored in one of RDBM database SQL server and we are moving this data to delta lake in medallion architecture and want to this to be near real time by using DLT pipeline.Problem is that source tab...
- 2617 Views
- 2 replies
- 0 kudos
- 0 kudos
came across this matrix while reading about DLT what is read from complete and write to Incremental means ?
- 0 kudos
- 2042 Views
- 2 replies
- 0 kudos
SQL warehourse l in AZURE databricks trial account
Can we create SQL warehourse 2xsmall cluster in AZURE databricks trial account
- 2042 Views
- 2 replies
- 0 kudos
- 9138 Views
- 1 replies
- 0 kudos
Resolved! "Run Job" without waiting for target job to finish?
We have configured a task in Job-A to run Job-B.However, the task in Job-A continues to 'run' until Job-B has completed.I can see this would be useful if we wanted to wait for Job-B and then perform another task, but we would actually like Job-A to e...
- 9138 Views
- 1 replies
- 0 kudos
- 0 kudos
you can create and run job use sdkdatabricks-sdk-py/examples/jobs/run_now_jobs_api_full_integration.py at main · databricks/databricks-sdk-py (github.com)
- 0 kudos
- 3136 Views
- 1 replies
- 2 kudos
Latest pushed code is not taken into account by Notebook
Hello, I cloned a repo my_repo in the Dataricks space Repos.Inside my_repo, I created a notebook new_experiment where I can import functions from my_repo, which is really handy. When I want to modify a function in my_repo, I open my local IDE, do the...
- 3136 Views
- 1 replies
- 2 kudos
- 2 kudos
Use %reload_ext autoreload instead, it will do your expected behavior.You just need to run it once, like %load_ext autoreload %autoreload 2
- 2 kudos
- 11245 Views
- 2 replies
- 1 kudos
Handling Exceptions from dbutils.fs in Python
I have a notebook that calls dbutils.fs.ls() for some derived file path in azure. Occasionally, this path may not exist, and in general I can't always guarantee that the path exists. When the path doesn't exist it throws an "ExecutionError" which app...
- 11245 Views
- 2 replies
- 1 kudos
- 1 kudos
Hey @jcoggs The problem looks legit though never occurred to me as I try to keep my mounts manually fed to the pipeline using a parameters or a variable by doing this you will have more control over your pipelines see if you could do the same in your...
- 1 kudos
- 2432 Views
- 1 replies
- 0 kudos
Databricks import api with lakeview dashboard "error_code":"INVALID_PARAMETER_VALUE"
Hi Community!I was trying to use the import api to replicate a lakehouse dashboard in a workspace but I keep bumping into INVALID_PARAMETER_VALUE error. After spending some time with getting the "content" property to (probably) correct base64 string ...
- 2432 Views
- 1 replies
- 0 kudos
- 0 kudos
There may be more information in the log. Look for "Cannot parse the zip archive."
- 0 kudos
- 31736 Views
- 12 replies
- 13 kudos
Option "delta.columnMapping.mode","name" introduces unexpected result
Hi, I am trying to write and create a delta table by enable "delta.columnMapping.mode","name", and the partition is date. But I found that when I enable this option, the partition folder name is not date any more while it is some random two letters.A...
- 31736 Views
- 12 replies
- 13 kudos
- 13 kudos
Hello, I'm a bit late to the party, but I'll put that for posterity:There's a way to rename your weird two letter named folders and still have your table working, but it violates the good practices guidelines suggested by Data Bricks, and I don't thi...
- 13 kudos
- 3004 Views
- 2 replies
- 1 kudos
Unable to write to Volume from DLT pipeline
Hi,I've a DLT pipeline running in Unity Catalog, and one of the task is to write content into a file within volume.I was able to write to file within volume using just PySpark, however when I do the same in DLT, I get an error:OSError: [Errno 30] Rea...
- 3004 Views
- 2 replies
- 1 kudos
- 1 kudos
It seems like a permission error. Can you check if the managed identity has correct permissions to write to the volume?DLT should be supported to write into volumes.
- 1 kudos
- 9357 Views
- 1 replies
- 0 kudos
What is the best practice for data model in silver layer in lakehouse
As per databricks https://www.databricks.com/glossary/medallion-architecture silver layers typically represent the "enterprise view" with improved quality than bronze (cleansed, deduplicated , augmented ) and mostly has 3NF like normalised data . The...
- 9357 Views
- 1 replies
- 0 kudos
- 0 kudos
Hey @param_sen Given your concerns about expensive joins and prioritizing analytics with flat raw data, here are some suggestions:Analyze the most common queries and reports you anticipate. Do they heavily rely on joins across dimensions? If not, the...
- 0 kudos
- 6472 Views
- 1 replies
- 1 kudos
Resolved! Asset Bundles Scala
Do asset bundles support Scala notebooks? I am trying to run a simple job that uses Scala notebook and getting an error: "run failed with error message Your administrator has only allowed sql and python commands on this cluster. This execution contai...
- 6472 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @VitaliiK,Thanks for bringing up your concerns; I am always happy to help The error means that the cluster is configured to only allow SQL and Python languages to be used in notebooks and the notebook you are trying to run contains a Scala langua...
- 1 kudos
- 4420 Views
- 3 replies
- 1 kudos
Resolved! autoloader stops working if I do not drop table each time
I first create a catalog and schema and ingest some data into it as follows:catalogName = 'neo'schemaName ='indicators'locationPath = 's3a://databricks-workspace-olaptrader-stack-1-bucket/unity-catalog/99999999xxx'sqlContext.sql(f"CREATE CATALOG IF N...
- 4420 Views
- 3 replies
- 1 kudos
- 1 kudos
My issue was due to the fact that I have over 300 columns and due to datatype mismatches, the rows were actually written to the table, but values were all null. That's why I didnt get any errors. I am doing manual datatype mapping now and I am able t...
- 1 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 581 | |
| 349 | |
| 287 |
