- 4734 Views
- 4 replies
- 0 kudos
Where the "Driver logs" are stored by default, and How much default space to store it.
Recently, when I was using databricks for deep learning I ran into an issue, i.e., after a certain amount of time of execution the cluster would break and restart. The logs are as below:The Event logs displayed "Metastore is down; Driver is up but is...
- 4734 Views
- 4 replies
- 0 kudos
- 0 kudos
Hello Jaron, is it not possible for you to redirect the login to an ABFSS or S3 bucket?
- 0 kudos
- 8226 Views
- 2 replies
- 0 kudos
Callback bound method error
When executing a withColumn (running on DBR 14.3 LST) I get this error:Error in callback <bound method UserNamespaceCommandHook.post_run_cell of <dbruntime.DatasetInfo.UserNamespaceCommandHook object at 0x7feda2b2efb0>> (for post_run_cell):How shoul...
- 8226 Views
- 2 replies
- 0 kudos
- 0 kudos
We have the same issue using a shared cluster running DBR 14.3:Code executed: dfNew = dfTmp.withColumn(HashKeyColumnName, F.sha2(F.concat_ws("||", *ColumnList), 256))Error received: Error in callback <bound method UserNamespaceCommandHook.post_run_ce...
- 0 kudos
- 1378 Views
- 0 replies
- 0 kudos
Pystan3 with databricks runtime 13.3
I am getting a read-only error when running pystan3 build() in a UDF.I think the issue is related to the location that the code is being run from, which is read-only. I am looking to set a custom cache location inside the UDF. Based on this link I a...
- 1378 Views
- 0 replies
- 0 kudos
- 2301 Views
- 1 replies
- 0 kudos
Databricks JDBC driver attempting to pass access token (JWT) from an external IDP (OKTA)
Configured a Databricks workspace for SSO to an IDP (OKTA).Databricks JDBC driver 02.06.38.1068Attempting to connect to Databricks using a URL similar to following where access token is obtained from the IDP (OKTA). Using tool such as SQLSquirrel wit...
- 2301 Views
- 1 replies
- 0 kudos
- 0 kudos
Yes, latest version 02.06.38.1068Would like to know if other persons have successfully passed access tokens from an external IDP via the driver.
- 0 kudos
- 2237 Views
- 0 replies
- 0 kudos
Databricks vs. Competitors: Key Features
Hi everyone,Can anyone share insights on the key features that differentiate Databricks from its competitors? Looking forward to your thoughts!Thanks!
- 2237 Views
- 0 replies
- 0 kudos
- 5077 Views
- 4 replies
- 1 kudos
run datarbicks worflow as service pricipal (managed identity) reads from azure dev ops repo Failed
Hello,we are running a workflow as a service principal, that is a aad managed identity. This does result in the issue: run databricks workflow as service principal the reads from azure dev ops repo Failed to checkout Git repository: PERMISSION_DENIED...
- 5077 Views
- 4 replies
- 1 kudos
- 1 kudos
We managed to solve this problem, however it is not an elegant solution. Databricks should simplify this.The steps that have to be done are listed below. We are using user assigned managed identity (MI), but I assume this should work for Azure Servic...
- 1 kudos
- 3181 Views
- 1 replies
- 0 kudos
Cluster Auto Termination Best Practices
Are there any recommended practices to set cluster auto termination for cost optimization?
- 3181 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @dataailearner , Greetings of the day! Here are a few steps that you can follow for cost optimizations: 1. Choose the most efficient compute size: Databricks runs one executor per worker node. The total number of cores across all executors is a...
- 0 kudos
- 1836 Views
- 1 replies
- 0 kudos
When are DLT going to support multiple targets
Due to the limitations with all output data needing to be stored in one target we have stopped using DLT until more flexibility is added. If anyone has a workaround we are open to suggestions.
- 1836 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi Zavi,One potential workaround is to establish multiple DLT pipelines, with each pipeline specifically configured to point to a unique target. This approach effectively allows for a diverse range of output data to be stored across various targets.T...
- 0 kudos
- 4496 Views
- 1 replies
- 0 kudos
Databricks Custom model Serving endpoint Failing
Hello all, I have created a custom model serving endpoint in Azure databricks. This endpoint connects with the AzureopenAI model and Azure postgres connection.All of these Azure services are with Private endpoints. When I run this notebook,I am able ...
- 4496 Views
- 1 replies
- 0 kudos
- 1212 Views
- 1 replies
- 0 kudos
VSCode Databricks Extension
Hi all,I've been trying to sync my VSCode IDE with our Databricks GCP workspace using the Databricks extension. I am able to connect authenticate my account and workspace and find our clusters. However, when I try to sync a destination it throws a st...
- 1212 Views
- 1 replies
- 0 kudos
- 0 kudos
@Retired_mod thanks for you response.I am not running through a proxy. At least, not on purpose. How do I know if I am running through a proxy? And where can I find the values of <proxy_url> and <port> so that I can try restarting my VSCode.I have tr...
- 0 kudos
- 24460 Views
- 0 replies
- 0 kudos
Identity column has null values
I created a table in databricks using a dbt model pre hook CREATE TABLE IF NOT EXISTS accounts ( account_id BIGINT GENERATED ALWAYS AS IDENTITY, description STRING other columns)I use the same dbt model to merge values into this table in the post...
- 24460 Views
- 0 replies
- 0 kudos
- 1624 Views
- 0 replies
- 0 kudos
UCX Installation Error
While downloading and installing ucx from a shell code,I am facing the below error. Can anyone provide a solution[i] Creating isolated Virtualenv with Python: /c/Program Files/Python312/pythonActual environment location may have moved due to redirect...
- 1624 Views
- 0 replies
- 0 kudos
- 989 Views
- 0 replies
- 0 kudos
Fetching CPU and memory data using REST APIs
Hi,I am trying to fetch CPU and memory details from Databricks. Are there any APIs present to which I can connect using postman and fetch these details?
- 989 Views
- 0 replies
- 0 kudos
- 5314 Views
- 3 replies
- 1 kudos
Setting up Unity Catalog in Azure
Trying to create a metastore that will be connected to an external storage (ADLS) but we don't have the option to create a new metastore in 'Catalog' tab in the UI. Based on some research, we see that we'll have to go into "Manage Account" and then c...
- 5314 Views
- 3 replies
- 1 kudos
- 1 kudos
I have been wrestling with this question for days now. I seem to be the only one with this question so I am sure I am doing something wrong. I am trying to create a UC metastore but there is not an option in "Catalog" to create a metastore. This s...
- 1 kudos
- 1273 Views
- 0 replies
- 0 kudos
Failed deploying bundle via gitlab - Request failed for POST
I'm encountering an issue in my .gitlab-ci.yml file when attempting to execute databricks bundle deploy -t prod. The error message I receive is: Error: Request failed for POST <path>/state/deploy.lockInterestingly, when I run the same command locally...
- 1273 Views
- 0 replies
- 0 kudos
