- 3951 Views
- 1 replies
- 0 kudos
Unable to instantiate Hive Meta Store Client
Databricks python sql script gives below error: Error in SQL statement: AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
- 3951 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @hal-qna, Just a friendly follow-up. Have you had a chance to review my colleague's response to your inquiry? Did it prove helpful, or are you still in need of assistance? Your response would be greatly appreciated.
- 0 kudos
- 7520 Views
- 2 replies
- 1 kudos
- 7520 Views
- 2 replies
- 1 kudos
- 1 kudos
I'll try to answer this in the simplest possible way 1. Spark is an imperative programming framework. You tell it what it to do, it does it. DLT is declarative - you describe what you want the datasets to be (i.e. the transforms), and it takes care ...
- 1 kudos
- 8538 Views
- 1 replies
- 0 kudos
Resolved! how to run @dlt pipeline in vscode
I want to test a pipeline created using dlt and python in vscode.
- 8538 Views
- 1 replies
- 0 kudos
- 0 kudos
Hey @rt-slowth check out this tutorial. You won't get debugging in VSCode yet, but this workflow is pretty nice.
- 0 kudos
- 2899 Views
- 1 replies
- 0 kudos
Data Encryption in DLT
Hi Team,We have a requirement to Encrypt PII data in Silver layer. What is the best way to implement this in DLT? and only users that has security privileges are able to decrypt the PII info.I have done this in the past using Structured Streaming but...
- 2899 Views
- 1 replies
- 0 kudos
- 0 kudos
Can you show me how to use the functions built in pyspark using DLT please.Also, trying to implement column/row level security in silver tables that is generated by DLT, but giving me the following error[RequestId=35024c5d-ad05-4f68-a4cb-f3a723f66e1c...
- 0 kudos
- 40955 Views
- 13 replies
- 3 kudos
Resolved! displayHTML can't seem to be used from Python code, only hand typed into a cell???
Trying to use displayHTML from w/in a Python module gets a Python exception:NameError: name 'displayHTML' is not definedand I've found no way around this. It seems to be something at the UI layer or something, not a Python function that can be refere...
- 40955 Views
- 13 replies
- 3 kudos
- 3 kudos
Holy Guacamole Batman! It works finally!!!! Wow, thanks @ptweir That's awesome! I can go back and update my doc (and code, to just use databricks the same, now, and Jupyter!) and it'll work by default. It's great they fixed it, shame they never told ...
- 3 kudos
- 7495 Views
- 2 replies
- 0 kudos
Resolved! Invalid configuration value detected for fs.azure.account.key only when trying to save RDD
Hello,We have encountered a weird issue in our (old) set-up that looks like a bug in the Unity Catalog. The storage account which we are trying to persist is configured via External Volumes.We have a pipeline that gets XML data and stores it in an RD...
- 7495 Views
- 2 replies
- 0 kudos
- 0 kudos
I will post here what worked resolving this error for us, in case someone else in the future encounters this.It turns out that this error appears in this case, when we were using the below command while the directory 'staging2' already exists. To avo...
- 0 kudos
- 14450 Views
- 3 replies
- 1 kudos
Resolved! How to kill the execution of a notebook on specyfic cell?
Let's say I want to check if a condition is false then stop the execution of the rest of the script. I tried with two approaches:1) raising exceptionif not data_input_cols.issubset(data.columns): raise Exception("Missing column or column's name mis...
- 14450 Views
- 3 replies
- 1 kudos
- 1 kudos
In Jupyter notebooks or similar environments, you can stop the execution of a notebook at a specific cell by raising an exception. However, you need to handle the exception properly to ensure the execution stops. The issue you're encountering could b...
- 1 kudos
- 10786 Views
- 9 replies
- 2 kudos
Resolved! How to version your workflows/jobs
WE would like to version control workflows/jobs over git, not the underlying notebooks but the job logic itselfis it possible?
- 10786 Views
- 9 replies
- 2 kudos
- 5696 Views
- 1 replies
- 0 kudos
Move Files from S3 to Local File System with Unity Catalog Enabled
Dear Databricks Community Experts,I am working on databricks on AWS with unity catalog.One usecase for me is to uncompress files with many extensions there on S3 Bucket.Below is my strategy:-Move files from S3 to Local file system (where spark driver...
- 5696 Views
- 1 replies
- 0 kudos
- 0 kudos
did you find a solution to this?
- 0 kudos
- 5404 Views
- 3 replies
- 0 kudos
Version control jobs
How do engineering teams out there version control their jobs? If there is a production issue, can I revert to an older version of the job?
- 5404 Views
- 3 replies
- 0 kudos
- 0 kudos
You can use version controlled source code for you databricks job and each time you need to rollback to older version of your job you need just to move to older version code. For version controlled source code you have multiple choises:- Use a noteb...
- 0 kudos
- 2984 Views
- 2 replies
- 2 kudos
Stream-stream window join after time window aggregation not working in 13.1
Hey,I'm trying to perform Time window aggregation in two different streams followed by stream-stream window join described here. I'm running Databricks Runtime 13.1, exactly as advised.However, when I'm reproducing the following code:clicksWindow = c...
- 2984 Views
- 2 replies
- 2 kudos
- 2 kudos
Hey,I'm currently facing the same problem, so I would to know if you've made any progress in resolving this issue.
- 2 kudos
- 4954 Views
- 1 replies
- 0 kudos
Combining DLT and workflow - MATERIALIZED_VIEW_OPERATION_NOT_ALLOWED
Hello everyone !I currently have a DLT pipeline that loads into several Delta LIVE tables (both streaming and materialized view).The end table of my DLT pipeline is a materialized view called "silver.my_view".In a later step I need to join/union/merg...
- 4954 Views
- 1 replies
- 0 kudos
- 10877 Views
- 2 replies
- 0 kudos
Divide a dataframe into multiple smaller dataframes based on values in multiple columns in Scala
I have to divide a dataframe into multiple smaller dataframes based on values in columns like - gender and state , the end goal is to pick up random samples from each dataframeI am trying to implement a sample as explained below, I am quite new to th...
- 10877 Views
- 2 replies
- 0 kudos
- 0 kudos
@raela I also have similar usecase. I am writing data to different databricks tables based on colum value.But I am getting insufficient disk space error and driver is getting killed. I am suspecting df.select(colName).distinct().collect()step is taki...
- 0 kudos
- 8184 Views
- 1 replies
- 2 kudos
IDENTITY columns generating every other number when merging
Hi,I'm doing merge to my Delta Table which has IDENTITY column:Id BIGINT GENERATED ALWAYS AS IDENTITYInserted data has in the id column every other number, like this:Is this expected behavior? Is there any workaround to make number increasing by 1?
- 8184 Views
- 1 replies
- 2 kudos
- 2 kudos
Were you able to resolve it? I'm facing the same issue.
- 2 kudos
- 5577 Views
- 0 replies
- 0 kudos
Merge delta tables with data more than 200 million
HI Everyone,Im trying to merge two delta tables who have data more than 200 million in each of them. These tables are properly optimized. But upon running the job, the job is taking a long time to execute and the memory spills are huger (1TB-3TB) rec...
- 5577 Views
- 0 replies
- 0 kudos
| User | Count |
|---|---|
| 1637 | |
| 791 | |
| 517 | |
| 349 | |
| 287 |
