- 19464 Views
- 8 replies
- 5 kudos
Resolved! 'No file or Directory' error when using pandas.read_excel in Databricks
I am baffled by the behaviour of Databricks:Below you can see the contents of the directory using dbutils in Databricks. It shows the `test.xlsx` file clearly in directory (and I can even open it using `dbutils.fs.head`) But when I go to use panda.re...
- 19464 Views
- 8 replies
- 5 kudos
- 5 kudos
Hey, I encountered it recently. I can see you are using the shared cluster, try switching to a single user cluster and it will fix it.Can someone let me know why it wasn't working w a shared cluster?Thanks.
- 5 kudos
- 3846 Views
- 1 replies
- 0 kudos
Consume 2 kafka topic with different schemas on 1 cluster databricks
Hi everyone,I have a concern that is there any way to read stream from 2 different kafka topics with 2 different in 1 jobs or same cluster? or we need to create 2 separate jobs for it ? (Job will need to process continually)
- 3846 Views
- 1 replies
- 0 kudos
- 1598 Views
- 0 replies
- 0 kudos
Strategy to add new table base on silver data
I have a merge function for streaming foreachBatch kind ofmergedf(df,i): merge_func_1(df,i) merge_func_2(df,i)Then I want to add new merge_func_3 into it. Is there any best practices for this case? when streaming always runs, how can I process...
- 1598 Views
- 0 replies
- 0 kudos
- 16168 Views
- 1 replies
- 1 kudos
Inconsistent Results When Writing to Oracle DB with Spark's dropDuplicates and foreachPartition
It's more a spark question then a databricks question, I'm encountering an issue when writing data to an Oracle database using Apache Spark. My workflow involves removing duplicate rows from a DataFrame and then writing the deduplicated DataFrame to ...
- 16168 Views
- 1 replies
- 1 kudos
- 1 kudos
The difference in behaviour between using foreachPartition and data.write.jdbc(...) after dropDuplicates() could be due to how Spark handles data partitioning and operations on partitions. When you use foreachPartition, you are manually handling the ...
- 1 kudos
- 11528 Views
- 5 replies
- 3 kudos
"MERGE" always slower than "CREATE OR REPLACE"
OverviewTo update our Data Warehouse tables, we have tried two methods: "CREATE OR REPLACE" and "MERGE". With every query we've tried, "MERGE" is slower.My question is this: Has anyone successfully gotten a "MERGE" to perform faster than a "CREATE OR...
- 11528 Views
- 5 replies
- 3 kudos
- 3 kudos
Hi @Graham Can you please try Low Shuffle Merge [LSM] and see if it helps? LSM is a new MERGE algorithm that aims to maintain the existing data organization (including z-order clustering) for unmodified data, while simultaneously improving performan...
- 3 kudos
- 47995 Views
- 0 replies
- 0 kudos
Recover files from previous cluster execution
I saved a file with results by just opening a file via fopen("filename.csv", "a").Once the execution ended (and the cluster shutted down) I couldn't retrieve the file.I found that the file was stored in "/databricks/driver", and that folder empties w...
- 47995 Views
- 0 replies
- 0 kudos
- 1476 Views
- 0 replies
- 0 kudos
Failure to intialize congratulations
Hi team,When we reading the CSV file from azure blob using databricks we are not getting any key error and able to read the data from blob .But if we are trying to read XML file it failed with key issue invalid configuration . Error:Failure to inti...
- 1476 Views
- 0 replies
- 0 kudos
- 2820 Views
- 1 replies
- 0 kudos
Updated sys.path not working any more
We have a monorepo so our pyspark notebooks do not use namespace relative to the root of the repo. Thus the default sys.path of repo root and cwd does not work. We used to package a whl dependency but recently moved to having code update sys.path wit...
- 2820 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Liliana , Just a friendly follow-up. Have you had a chance to review my colleague's response to your inquiry? Did it prove helpful, or are you still in need of assistance? Your response would be greatly appreciated.
- 0 kudos
- 3826 Views
- 1 replies
- 0 kudos
Unable to instantiate Hive Meta Store Client
Databricks python sql script gives below error: Error in SQL statement: AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
- 3826 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @hal-qna, Just a friendly follow-up. Have you had a chance to review my colleague's response to your inquiry? Did it prove helpful, or are you still in need of assistance? Your response would be greatly appreciated.
- 0 kudos
- 7194 Views
- 2 replies
- 1 kudos
- 7194 Views
- 2 replies
- 1 kudos
- 1 kudos
I'll try to answer this in the simplest possible way 1. Spark is an imperative programming framework. You tell it what it to do, it does it. DLT is declarative - you describe what you want the datasets to be (i.e. the transforms), and it takes care ...
- 1 kudos
- 8348 Views
- 1 replies
- 0 kudos
Resolved! how to run @dlt pipeline in vscode
I want to test a pipeline created using dlt and python in vscode.
- 8348 Views
- 1 replies
- 0 kudos
- 0 kudos
Hey @rt-slowth check out this tutorial. You won't get debugging in VSCode yet, but this workflow is pretty nice.
- 0 kudos
- 2779 Views
- 1 replies
- 0 kudos
Data Encryption in DLT
Hi Team,We have a requirement to Encrypt PII data in Silver layer. What is the best way to implement this in DLT? and only users that has security privileges are able to decrypt the PII info.I have done this in the past using Structured Streaming but...
- 2779 Views
- 1 replies
- 0 kudos
- 0 kudos
Can you show me how to use the functions built in pyspark using DLT please.Also, trying to implement column/row level security in silver tables that is generated by DLT, but giving me the following error[RequestId=35024c5d-ad05-4f68-a4cb-f3a723f66e1c...
- 0 kudos
- 38541 Views
- 13 replies
- 3 kudos
Resolved! displayHTML can't seem to be used from Python code, only hand typed into a cell???
Trying to use displayHTML from w/in a Python module gets a Python exception:NameError: name 'displayHTML' is not definedand I've found no way around this. It seems to be something at the UI layer or something, not a Python function that can be refere...
- 38541 Views
- 13 replies
- 3 kudos
- 3 kudos
Holy Guacamole Batman! It works finally!!!! Wow, thanks @ptweir That's awesome! I can go back and update my doc (and code, to just use databricks the same, now, and Jupyter!) and it'll work by default. It's great they fixed it, shame they never told ...
- 3 kudos
- 7236 Views
- 2 replies
- 0 kudos
Resolved! Invalid configuration value detected for fs.azure.account.key only when trying to save RDD
Hello,We have encountered a weird issue in our (old) set-up that looks like a bug in the Unity Catalog. The storage account which we are trying to persist is configured via External Volumes.We have a pipeline that gets XML data and stores it in an RD...
- 7236 Views
- 2 replies
- 0 kudos
- 0 kudos
I will post here what worked resolving this error for us, in case someone else in the future encounters this.It turns out that this error appears in this case, when we were using the below command while the directory 'staging2' already exists. To avo...
- 0 kudos
- 14122 Views
- 3 replies
- 1 kudos
Resolved! How to kill the execution of a notebook on specyfic cell?
Let's say I want to check if a condition is false then stop the execution of the rest of the script. I tried with two approaches:1) raising exceptionif not data_input_cols.issubset(data.columns): raise Exception("Missing column or column's name mis...
- 14122 Views
- 3 replies
- 1 kudos
- 1 kudos
In Jupyter notebooks or similar environments, you can stop the execution of a notebook at a specific cell by raising an exception. However, you need to handle the exception properly to ensure the execution stops. The issue you're encountering could b...
- 1 kudos
| User | Count |
|---|---|
| 1619 | |
| 790 | |
| 486 | |
| 349 | |
| 287 |
