- 13912 Views
- 2 replies
- 6 kudos
METASTORE_DOWN: Cannot connect to metastore
I am trying to view databases and tables, default as well user created but it looks like the cluster created is not able to connect. I am using databricks default hive metastore. Viewing cluster logs provide following ventMETASTORE_DOWN Metastore is...
- 13912 Views
- 2 replies
- 6 kudos
- 6 kudos
@rami​ :If the metastore is down, it means that the Databricks cluster is not able to connect to the metastore. Here are a few things you can try to resolve the issue:Check if the Hive metastore is up and running. You can try to connect to the metast...
- 6 kudos
- 3773 Views
- 2 replies
- 2 kudos
Resolved! Error 95 when importing one Notebook into another
When I follow the instructions Modularize your code using files I get the following error:I am on azure, use DBRT 12.2 LTS, use ADLS as storage, I am happy to provide more details if needed. My research suggest that the reason is that the dfbs fuse...
- 3773 Views
- 2 replies
- 2 kudos
- 2 kudos
import works for .py files..%run is for notebooks.is lib a .py file or a notebook?
- 2 kudos
- 4612 Views
- 3 replies
- 4 kudos
How do I define & run jobs that execute scripts that are copied inside a custom DataBricks container?
Hi all, we are building custom Databricks containers (https://docs.databricks.com/clusters/custom-containers.html). During the container build process we install dependencies and also python source code scripts. We now want to run some of these scrip...
- 4612 Views
- 3 replies
- 4 kudos
- 4 kudos
Hi @Thijs van den Berg​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best an...
- 4 kudos
- 4957 Views
- 2 replies
- 1 kudos
Resolved! Is it possible to write a pyspark dataframe to a custom log table in Log Analytics workspace?
I have a pyspark dataframe that contains information about the tables that I have on sql database (creation date, number of rows, etc)Sample data: { "Day":"2023-04-28", "Environment":"dev", "DatabaseName":"default", "TableName":"discount"...
- 4957 Views
- 2 replies
- 1 kudos
- 1 kudos
@Bruno Simoes​ :Yes, it is possible to write a PySpark DataFrame to a custom log table in Log Analytics workspace using the Azure Log Analytics Workspace API.Here's a high-level overview of the steps you can follow:Create an Azure Log Analytics Works...
- 1 kudos
- 1957 Views
- 1 replies
- 1 kudos
Connecting to Azure SQL database using R in Databricks and Azure Key Vaults
I'm trying to connect to an Azure SQL database using R in Databricks. I want to read the credentials stored in Azure secret key vaults rather than hard coding in R code. I've seen some examples of it being done in Scala, however i'm after a R solutio...
- 1957 Views
- 1 replies
- 1 kudos
- 1 kudos
Did you find a solution for this @Long Pham​ ?? I am having the same issue
- 1 kudos
- 5567 Views
- 2 replies
- 4 kudos
Resolved! Delta Table Merge statement is not accepting broadcast hint
I have a statement like this with pyspark:target_tbl.alias("target")\ .merge(stage_df.hint("broadcast").alias("source"), merge_join_expr)\ .whenMatchedUpdateAll()\ .whenNotMatchedInsertAll()\ .w...
- 5567 Views
- 2 replies
- 4 kudos
- 4 kudos
Hi @Nikolay Chalkanov​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best ans...
- 4 kudos
- 14671 Views
- 2 replies
- 0 kudos
S3 write to bucket - best performance tips
I'm writing big dataframes into deltas in s3 buckets. df.write\ .format("delta")\ .mode("append")\ .partitionBy(partitionColumns)\ .option("mergeSchema", "true")\ .save(target...
- 14671 Views
- 2 replies
- 0 kudos
- 0 kudos
@Pablo (Ariel)​ :There are several ways to improve the performance of writing data to S3 using Spark. Here are some tips and recommendations:Increase the size of the write buffer: By default, Spark writes data in 1 MB batches. You can increase the si...
- 0 kudos
- 9274 Views
- 1 replies
- 2 kudos
Resolved! Best Practices for PowerBI Connectivity w/ Partner Connect. Access Token w/ Service Principal, Databricks Username w/ Service account, or OAuth?
I'm aware all are possible methods but are all equal? Or is the matter trivial? Thank you so much!
- 9274 Views
- 1 replies
- 2 kudos
- 2 kudos
@Will Heyer​ :The best method for Power BI connectivity with Partner Connect depends on your specific use case and requirements. Here are some factors to consider for each method:Access Token with Service Principal: This method uses a client ID and s...
- 2 kudos
- 17683 Views
- 1 replies
- 4 kudos
Resolved! How to use multi-cursor and rectangular selection for notebooks and query editor in Linux ?
The documentation explains how to use multicursor in notebooks. However, it only says it for Windows and MacOS. The Windows way would work in Linux (Ubuntu) up to a few days ago but it does not work now anymore.
- 17683 Views
- 1 replies
- 4 kudos
- 4 kudos
@Davide Cagnoni​ :Multicursor support in Databricks notebooks is implemented using the Ace editor, which is a web-based code editor. Therefore, the behavior of multicursor support may depend on the specific browser and operating system you are using....
- 4 kudos
- 2808 Views
- 1 replies
- 1 kudos
Resolved! Clusters API 2.0 - Unable to execute cluster events api
Details: I keep getting "Missing required field: cluster_id" even though you can see it is supplied. Is this a bug? or I am missing something? I am testing this in postman. Error: {"error_code":"INVALID_PARAMETER_VALUE","message":"Missing required fi...
- 2808 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi guysI'm sorry to hear that the Clusters API 2.0 and cluster event execution are giving you trouble. I advise getting in touch with the support staff for guidance on quickly fixing the problem.
- 1 kudos
- 2573 Views
- 1 replies
- 1 kudos
Resolved! DLT best practices
Hi Team,Could you please recommend the best practices to implement the delta live tables?Regards,Phanindra
- 2573 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi Phani, what exactly are you looking for with best practices? At a high level:Always provide an external storage location (S3, ADLS, GCS) for your pipelineUse Auto Scaling! Python imports can be leverage to reuse code With regards to providing a st...
- 1 kudos
- 7296 Views
- 1 replies
- 2 kudos
Resolved! Azure Databricks PATs expire even before validity
Hi all,we have this issue in our environment - even thought we give 365 days validity for Databricks PATS generation, the PATs expire every now and then. Is there any problem with the command we use : curl --location --request POST 'https://<<HOST_NA...
- 7296 Views
- 1 replies
- 2 kudos
- 2 kudos
@NOOR BASHA SHAIK​ It looks you are providing 365 days, can you please post your response. if you won't provide any lifetime then it should be valid indefinitely. can you please add 90 days validity and test
- 2 kudos
- 1583 Views
- 1 replies
- 1 kudos
API to get Databricks Status AWS.
Hi, Do you have an api endpoint to call to get the databricks status for AWS?Thanks,
- 1583 Views
- 1 replies
- 1 kudos
- 1 kudos
@Chinu Lee​ you have webhook/slack that can be used to fetch status https://docs.databricks.com/resources/status.html#webhookare you specifically looking for your account workspace/above one
- 1 kudos
- 2145 Views
- 1 replies
- 3 kudos
Create (account wide) groups without account admin permissions
The use case is quite simple: each environment - databricks workspace (prod, test, dev) will be created by a separate service principal (which for isolation purpose should not have account wide admin permission) with terraform, but will belong to the...
- 2145 Views
- 1 replies
- 3 kudos
- 3 kudos
Another thing would be also to assign workspace to a metastore without account admin permission - for similar reason.
- 3 kudos
- 8422 Views
- 6 replies
- 2 kudos
Resolved! Delta Sharing - Unity Catalog difference
Delta Sharing and Unity catalog both have elements of data sharing. Can you please explain when one would use Delta sharing vs Unity Catalog?
- 8422 Views
- 6 replies
- 2 kudos
- 2 kudos
Based on the Databricks reply from the post below: "Unity Catalog does not currently support separating data by workspace or Azure subscription. As you noted, data from all catalogs within a region can be accessed by any workspace within that region,...
- 2 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 553 | |
| 349 | |
| 287 |
