cancel
Showing results for 
Search instead for 
Did you mean: 
Get Started Discussions
Start your journey with Databricks by joining discussions on getting started guides, tutorials, and introductory topics. Connect with beginners and experts alike to kickstart your Databricks experience.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

harvey-c
by New Contributor III
  • 3128 Views
  • 4 replies
  • 0 kudos

DLT Performance question with Unity Catalog

Dear Community MembersThis question is about debugging performance issue of DLT pipeline with unity catalog.I had a DLT pipeline in Azure Databricks running on local store i.g. hive_metastore. And the processes took about 2 hour with the auto scalain...

  • 3128 Views
  • 4 replies
  • 0 kudos
Latest Reply
Mystagon
New Contributor III
  • 0 kudos

Hey Harvey, I getting around the same performance problems as you:From around 25 minutes in a normal workspace to an 1 hour and 20mins in UC workspace. Which is roughly 3x slower.Did you manage to solve this? I've also noticed dbutil.fs.ls() is much ...

  • 0 kudos
3 More Replies
ChristianRRL
by Valued Contributor III
  • 4260 Views
  • 0 replies
  • 0 kudos

Auto-Update API Data

Not sure if this has come up before, but I'm wondering if Databricks has any kind of functionality to "watch" an API call for changes?E.g. Currently I have a frequently running job that pulls data via an API call and overwrites the old data. This see...

  • 4260 Views
  • 0 replies
  • 0 kudos
SamyA
by New Contributor III
  • 9854 Views
  • 7 replies
  • 4 kudos

System table with state UNAVAILABLE

Hello,When I check the system table's status, it seems that they are in UNAVAILABLE state. I would like to know if anyone have faced this issue ?Because of that, I can't enable the system table. {"schemas":[{"schema":"storage","state":"UNAVAILABLE"},...

  • 9854 Views
  • 7 replies
  • 4 kudos
Latest Reply
D365
New Contributor II
  • 4 kudos

May Be internal IssueFollow

  • 4 kudos
6 More Replies
abhijit007
by New Contributor
  • 4274 Views
  • 1 replies
  • 0 kudos

Unable to connect Azure kafka server with public IP from databricks notebook

Hi Team,I am unable to connect (SSH connection) from Azure Databricks Notebook to Azure Kafka server.Kafka Server and Databricks both are under same resource group and region. Also in Inbound rule the port is added in Kafka server.Please help me to r...

abhijit007_0-1704345593867.png abhijit007_1-1704346105457.png
  • 4274 Views
  • 1 replies
  • 0 kudos
Latest Reply
Debayan
Databricks Employee
  • 0 kudos

Hi, This looks like issue with networking config. Could you please check on the routing configs, firewall routes etc to make sure destination IP to 9092 is added in the Azure console? 

  • 0 kudos
311102
by New Contributor
  • 2343 Views
  • 1 replies
  • 0 kudos

user email invitation to workspace not received

HelloSince December 2023, I cannot anymore invite users to connect to my workspace as I used to. For no reason, the users I add through my admin dashboard do not receive the invitation email and thus the link to connect to the workspace.   I tried my...

  • 2343 Views
  • 1 replies
  • 0 kudos
Latest Reply
Debayan
Databricks Employee
  • 0 kudos

Hi, Could you also please try to add users through account console if Identity Federation is enabled? Refer: https://docs.databricks.com/en/administration-guide/users-groups/users.html#assign-a-user-to-a-workspace-using-the-account-console

  • 0 kudos
thibault
by Contributor III
  • 16272 Views
  • 11 replies
  • 6 kudos

databricks-connect 13.1.0 limitations

Hi,Quite excited to see the new release of databricks-connect, I started writing unit tests running pyspark on a databricks cluster using databricks-connect.After some successful basic unit tests, I tested just more chained transformations on a dataf...

  • 16272 Views
  • 11 replies
  • 6 kudos
Latest Reply
jackson-nline
New Contributor III
  • 6 kudos

I doubled the `spark.connect.grpc.maxInboundMessageSize` parameter to 256mb but that didn't appear to resolve anything.

  • 6 kudos
10 More Replies
udi_azulay
by New Contributor II
  • 1363 Views
  • 1 replies
  • 0 kudos

Running sql command on Single User cluster vs Shared.

Hi, when i am running the below simple code over my Unity Catalog on a Shared cluster, it works very well.But on a Single User - i am getting : Failed to acquire a SAS token for list on /__unitystorage/schemas/1bb5b053-ac96-471b-8077-8288c56c8a20/tab...

  • 1363 Views
  • 1 replies
  • 0 kudos
Latest Reply
Debayan
Databricks Employee
  • 0 kudos

Hi, Could you please refer to the limitations here: https://docs.databricks.com/en/compute/access-mode-limitations.html . Please let us know if this helps. 

  • 0 kudos
udi_azulay
by New Contributor II
  • 4820 Views
  • 1 replies
  • 0 kudos

local filesystem access is forbidden

Hi,When i running this command over my private cluster (Single User) it works well :dbutils.fs.cp(ituff_file, protocol_local_file_path) When i try to run it over a shared cluster, i am getting : java.lang.SecurityException: Cannot use com.databricks....

  • 4820 Views
  • 1 replies
  • 0 kudos
Latest Reply
Debayan
Databricks Employee
  • 0 kudos

Hi, Could you please refer to the limitations here: https://docs.databricks.com/en/compute/access-mode-limitations.html#shared-access-mode-limitations-on-unity-catalog  (Volumes do not support dbutils.fs commands distributed to executors.) , Also, fo...

  • 0 kudos
Mado
by Valued Contributor II
  • 4386 Views
  • 1 replies
  • 0 kudos

How to update a value in a column in a delta table with Map of Struct datatype?

I have a delta table in Databricks named "prod.silver.control_table". It has a few columns including "table_name" with string data type and "transform_options" with the below structure:  |-- transform_options: map (nullable = true) | |-- key: str...

Get Started Discussions
MAP
Struct
Update_table
  • 4386 Views
  • 1 replies
  • 0 kudos
Databricks_Work
by New Contributor II
  • 3007 Views
  • 1 replies
  • 0 kudos

how to access data in one databricks in another databricks

I want to acces data in another databricks in my databricks, how to do that

  • 3007 Views
  • 1 replies
  • 0 kudos
Latest Reply
Walter_C
Databricks Employee
  • 0 kudos

Hello, many thanks for your question, to be able to provide you with a more precise response we required some additional information:1. When you refer databricks in my databricks are you refering to access data that is in one workspace to another wor...

  • 0 kudos
Godhuli
by New Contributor II
  • 1776 Views
  • 0 replies
  • 0 kudos

Unable to login to Databricks Community edition

I signed up to Databricks Community Edition with gmail and verified my account and used it to create a notebook as well but having issue in re-logging in even though email and password provided are correct. I did  try  "Forget Password" step but I am...

  • 1776 Views
  • 0 replies
  • 0 kudos
arun949290
by New Contributor II
  • 3666 Views
  • 3 replies
  • 0 kudos

Unable to login to community edition

Hello there,I have successfully created a databricks account and went to login to the community edition with the exact same login credentials as my account, but it tells me that the email/password are invalid. I can login with these same exact creden...

  • 3666 Views
  • 3 replies
  • 0 kudos
Latest Reply
Erik_L
Contributor II
  • 0 kudos

Databricks Community Edition and Databricks are separate services. You have to create an account specific to community.

  • 0 kudos
2 More Replies
Omri
by New Contributor
  • 1182 Views
  • 1 replies
  • 0 kudos

Is it possible to create a scratchpad ui?

https://jupyter-contrib-nbextensions.readthedocs.io/en/latest/nbextensions/scratchpad/README.htmlIs something like this available on Databrick's notebooks ui?

  • 1182 Views
  • 1 replies
  • 0 kudos
Latest Reply
Debayan
Databricks Employee
  • 0 kudos

Hi, it would be great if you can submit the idea on this and the progress can be tracked by the ideas portal. https://docs.databricks.com/en/resources/ideas.html 

  • 0 kudos

Join Us as a Local Community Builder!

Passionate about hosting events and connecting people? Help us grow a vibrant local community—sign up today to get started!

Sign Up Now
Labels