- 2116 Views
- 4 replies
- 0 kudos
DLT Performance question with Unity Catalog
Dear Community MembersThis question is about debugging performance issue of DLT pipeline with unity catalog.I had a DLT pipeline in Azure Databricks running on local store i.g. hive_metastore. And the processes took about 2 hour with the auto scalain...
- 2116 Views
- 4 replies
- 0 kudos
- 0 kudos
Hey Harvey, I getting around the same performance problems as you:From around 25 minutes in a normal workspace to an 1 hour and 20mins in UC workspace. Which is roughly 3x slower.Did you manage to solve this? I've also noticed dbutil.fs.ls() is much ...
- 0 kudos
- 3094 Views
- 0 replies
- 0 kudos
Auto-Update API Data
Not sure if this has come up before, but I'm wondering if Databricks has any kind of functionality to "watch" an API call for changes?E.g. Currently I have a frequently running job that pulls data via an API call and overwrites the old data. This see...
- 3094 Views
- 0 replies
- 0 kudos
- 6500 Views
- 7 replies
- 3 kudos
System table with state UNAVAILABLE
Hello,When I check the system table's status, it seems that they are in UNAVAILABLE state. I would like to know if anyone have faced this issue ?Because of that, I can't enable the system table. {"schemas":[{"schema":"storage","state":"UNAVAILABLE"},...
- 6500 Views
- 7 replies
- 3 kudos
- 3313 Views
- 1 replies
- 0 kudos
Unable to connect Azure kafka server with public IP from databricks notebook
Hi Team,I am unable to connect (SSH connection) from Azure Databricks Notebook to Azure Kafka server.Kafka Server and Databricks both are under same resource group and region. Also in Inbound rule the port is added in Kafka server.Please help me to r...
- 3313 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi, This looks like issue with networking config. Could you please check on the routing configs, firewall routes etc to make sure destination IP to 9092 is added in the Azure console?
- 0 kudos
- 1092 Views
- 1 replies
- 0 kudos
user email invitation to workspace not received
HelloSince December 2023, I cannot anymore invite users to connect to my workspace as I used to. For no reason, the users I add through my admin dashboard do not receive the invitation email and thus the link to connect to the workspace. I tried my...
- 1092 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi, Could you also please try to add users through account console if Identity Federation is enabled? Refer: https://docs.databricks.com/en/administration-guide/users-groups/users.html#assign-a-user-to-a-workspace-using-the-account-console
- 0 kudos
- 10823 Views
- 11 replies
- 6 kudos
databricks-connect 13.1.0 limitations
Hi,Quite excited to see the new release of databricks-connect, I started writing unit tests running pyspark on a databricks cluster using databricks-connect.After some successful basic unit tests, I tested just more chained transformations on a dataf...
- 10823 Views
- 11 replies
- 6 kudos
- 6 kudos
I doubled the `spark.connect.grpc.maxInboundMessageSize` parameter to 256mb but that didn't appear to resolve anything.
- 6 kudos
- 762 Views
- 1 replies
- 0 kudos
Running sql command on Single User cluster vs Shared.
Hi, when i am running the below simple code over my Unity Catalog on a Shared cluster, it works very well.But on a Single User - i am getting : Failed to acquire a SAS token for list on /__unitystorage/schemas/1bb5b053-ac96-471b-8077-8288c56c8a20/tab...
- 762 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi, Could you please refer to the limitations here: https://docs.databricks.com/en/compute/access-mode-limitations.html . Please let us know if this helps.
- 0 kudos
- 2237 Views
- 1 replies
- 0 kudos
how to access data in one databricks in another databricks
I want to acces data in another databricks in my databricks, how to do that
- 2237 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello, many thanks for your question, to be able to provide you with a more precise response we required some additional information:1. When you refer databricks in my databricks are you refering to access data that is in one workspace to another wor...
- 0 kudos
- 5286 Views
- 3 replies
- 2 kudos
Resolved! Move multiple notebooks at the same time (programmatically)
If I want to move multiple (hundreds of) notebooks at the same time from one folder to another, what is the best way to do that? Other than going to each individual notebook and clicking "Move".Is there a way to programmatically move notebooks? Like ...
- 5286 Views
- 3 replies
- 2 kudos
- 2 kudos
You should be redirected to the KB page, but this is the information contained: Problem How to migrate Shared folders and the notebooks Cause Shared notebooks are not migrated into new workspace by default Solution Please find the script to migrate t...
- 2 kudos
- 2394 Views
- 1 replies
- 2 kudos
Databricks API using the personal access token
We can access the Azure databricks API using the personal access token which is created by us manually.The objective is that client don’t want to store the personal access token which may not be secure .Do we have option to generate the token during ...
- 2394 Views
- 1 replies
- 2 kudos
- 2 kudos
Hi @Phani1 ,Yes, now you can use databricks Create a user token API for create access token via automated API call.Please refer below doc - Create a user token | Token API | REST API reference | Azure Databricks
- 2 kudos
- 4307 Views
- 3 replies
- 0 kudos
Databricks Volume. Not able to read a file from Scala.
I used to use dbfs with mounted directories and now I want to switch to Volumes for storing my jars and application.conf for pipelines. I see the file my application.conf in Data Explorer > Catalog > Volumes, I also see the file with dbutils.fs.ls("/...
- 4307 Views
- 3 replies
- 0 kudos
- 0 kudos
Volumes mount are accessible using scala code only on a shared cluster. On single user mode this features is not supported yet. We use init scripts to move contents from Volumes to clusters local drive, when we need to access files from Native Scala ...
- 0 kudos
- 3487 Views
- 2 replies
- 1 kudos
Resolved! DLT Notebook and Pipeline Separation vs Consolidation
Super basic question. For DLT pipelines I see there's an option to add multiple "Paths". Is it generally best practice to completely separate `bronze` from `silver` notebooks? Or is it more recommended to bundle both raw `bronze` and clean `silver` d...
- 3487 Views
- 2 replies
- 1 kudos
- 7813 Views
- 0 replies
- 0 kudos
Cloudera SQL
Hi Team,Could you please help me how efficiently/quickly can convert Cloudera SQL and Hive SQL Scripts to Pyspark Script.Regards,Phanindra
- 7813 Views
- 0 replies
- 0 kudos
- 1187 Views
- 1 replies
- 0 kudos
Databricks setup with Azure storage
Hi ,We have an issue with our intial setup and design.We are using a single Azure, Premium, Block blob storage account with hierarchical namespace and LRS enabled.We have three containers within it, one for each environment – Dev -Test – ProdBut the ...
- 1187 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi, could you please check https://community.databricks.com/t5/data-governance/metastore-one-per-account-region-limitation/td-p/41097 and let us know if this discussion helps?
- 0 kudos
- 3020 Views
- 0 replies
- 0 kudos
How to handle complex json schema
I have a mounted external directory that is an s3 bucket with multiple subdirectories containing call log files in json format. The files are irregular and complex, when i try to use spark.read.json or spark.sql (SELECT *) i get the UNABLE_TO_INFER_...
- 3020 Views
- 0 replies
- 0 kudos
