- 5246 Views
- 1 replies
- 0 kudos
We are getting AttributeError: module 'numpy.typing' has no attribute 'NDArray' with tidal jobs
File "/databricks/python/lib/python3.8/site-packages/PIL/_typing.py", line 10, in <module>NumpyArray = npt.NDArray[Any]AttributeError: module 'numpy.typing' has no attribute 'NDArray' we are using 10.4 runtime version , please suggest
- 5246 Views
- 1 replies
- 0 kudos
- 0 kudos
The file path of the above error is suggesting it is from the Pillow library dependency: "/databricks/python/lib/python3.8/site-packages/PIL/_typing.py" Pillow's _typing.py checks for NumPy >=1.21 while 10.4 runtime has these versions installed ...
- 0 kudos
- 4387 Views
- 1 replies
- 1 kudos
Troubleshooting Cluster
We had a failure on a previously running fact table load (our biggest one) and it looked like an executor was failing due to a timeout error. As a test we upped the cluster size and changed the spark.executor.heartbeatinterval to 300s and the spark....
- 4387 Views
- 1 replies
- 1 kudos
- 1 kudos
The XXKDA error code is a general indicator for task scheduler issues or SPARK_JOB_CANCELLED.
- 1 kudos
- 674 Views
- 1 replies
- 0 kudos
Community Accounts Getting Deleted Automatically
This is the second time in the same week, the accounts are getting vanished without any prior information. All the saved work has no backup I guess and no one would take responsibility because it a community version.Not a good thing to happen twice, ...
- 674 Views
- 1 replies
- 0 kudos
- 0 kudos
I would suggest to follow steps in https://community.databricks.com/t5/support-faqs/databricks-community-sso-august-3rd-2024/ta-p/78459 , if issue persist please contact databricks-community@databricks.com
- 0 kudos
- 459 Views
- 1 replies
- 0 kudos
Account missing
I have been using my community for the past two weeks. Now, while trying to log in, it shows this: Can anyone help me fix this? Thanks in advance!
- 459 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello, @udhayakumar!You can find helpful resources for Databricks Community Edition here. If the available resource doesn’t resolve your concern, please submit a ticket with Databricks Support team for further assistance.
- 0 kudos
- 1223 Views
- 1 replies
- 0 kudos
Databricks SQL Warehouse Hung - Queries Stuck in Queued State & No Alerts Triggered
We have been facing critical challenges with Databricks SQL Warehouse for the last four weeks. We are using Databricks SQL Warehouse injection from IICS, and we have observed the following issues:SQL Warehouse Going into a Hung State – The SQL Wareho...
- 1223 Views
- 1 replies
- 0 kudos
- 0 kudos
Hey @sdheepak The first thing you need to identify is the type of SQL Warehouse you are using in Databricks:• Is it Serverless? If so, it is fully managed by Databricks, and you must contact Databricks support because you won’t have access to logs in...
- 0 kudos
- 940 Views
- 1 replies
- 0 kudos
Can we provide custom dns name for Databricks app?
Hi All,I want to access my Databricks app https://myapp.aws.databricksapps.com/ using https://myapp.mycompaney.com Is this possible? We tried DNS mapping but it is not working.
- 940 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @satniks_o , It should be possible to set up a DNS but requires SSL and other settings, how are you setting it up?
- 0 kudos
- 609 Views
- 1 replies
- 0 kudos
why doesn't databricks allow setting executor metrics
I have an all-purpose compute cluster that processes different data sets for various jobs. I am struggling to optimize executor metrics like below.spark.executor.memory 4gIs it allowed to override default executor metrics and specify such configurati...
- 609 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @marc88, As you mentioned in Spark config under Advance cluster options you can do it once cluster boots up it will be set at run level. Or you can draft a cluster policy and apply it across for job computes when creating your workflow.
- 0 kudos
- 692 Views
- 2 replies
- 0 kudos
Cross-tenant networking two orgs on Azure
Hi Databricks Community,For one client, we are looking for best practices, steps to establish cross-tenant networking and landing zones between two organizations. The current situation is that the source systems are in one organization, while the Azu...
- 692 Views
- 2 replies
- 0 kudos
- 0 kudos
Cross posting https://community.databricks.com/t5/administration-architecture/databricks-azure-cross-tenant-connection-to-storage-account/td-p/82491#:~:text=I%20tested%20the%20connection%20to,when%20tested%20from%20this%20VM.
- 0 kudos
- 2582 Views
- 3 replies
- 5 kudos
It's possible to sync account group/user to workspace without the need to do it manually?
I am using Databricks SCIM for my Databricks Account, so when i add a user or group in the SCIM connector, the user or group its created in Databricks Account. After this, i need to manually assign the user/group to the workspaces. My boss wants to o...
- 2582 Views
- 3 replies
- 5 kudos
- 884 Views
- 2 replies
- 1 kudos
Resolved! Table size information display
Hi,I have a problem with displaying information about the size of my tables.This information is visible several times, but after a while it disappears again.I need to undestand what is happen, and why this information is not available all the time on...
- 884 Views
- 2 replies
- 1 kudos
- 1 kudos
In general you need a cluster or warehouse to be active for those detail to be presented.
- 1 kudos
- 978 Views
- 1 replies
- 0 kudos
Driver: how much memory is actually available?
I have a cluster where Driver type is Standard_DS3_v2 (14GB Memory and 4 Cores). When I use free -h command in Web terminal (see attached screenshot) I get the response that I only have 8.9GB memory available on my driver - why is that?fyi, spark.dri...
- 978 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @dbuserng , The free -h command in the web terminal shows only 8.9GB of available memory on your driver, which is a Standard_DS3_v2 instance with 14GB of memory, because Databricks has services running on each node. This means the maximum allowabl...
- 0 kudos
- 635 Views
- 1 replies
- 0 kudos
JVM Heap Memory Graph - more memory used than available
I'm analyzing the memory usage of my Spark application and I see something strange when checking JVM Heap Memory Graph (see screenshot below). Each line on the graph is representing one executor.Why the memory usage sometimes reaches over 10GB, when ...
- 635 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @dbuserng , The memory usage in your Spark application can exceed the spark.executor.memory setting of 7GB for several reasons: • Off-Heap Memory Usage: Spark allows for off-heap memory allocation, which is not managed by the JVM garbage collector...
- 0 kudos
- 1157 Views
- 4 replies
- 0 kudos
photon is being used by a job or not
We have lots of customers using many job as well as interactive clusters with photon enabled which is drastically increasing the cost .We would like to know if there is any table in system or any details that we can get through API that lists if the ...
- 1157 Views
- 4 replies
- 0 kudos
- 0 kudos
Hi @sruthianki , If you want to check if the job is really using photon or not you can check the SQL query plan in spark UI for its stages and the metrics will highlighted in yellow colour.
- 0 kudos
- 21626 Views
- 2 replies
- 1 kudos
How to Know DBU consumption in azure databricks ?
In Azure portal - Billing we can get the COST but how to know How much DBU is consumed ?
- 21626 Views
- 2 replies
- 1 kudos
- 1 kudos
There was a promo on serverless early 2024 at the time which at some point got extended, and was bigger depending where you were.
- 1 kudos
- 1521 Views
- 2 replies
- 0 kudos
Configuration of NCC for Serverless to access SQL server running in a Azure VM
Hi Team, I am following this link to configure NCC for a Serverless compute to access a SQL Server running in a Azure VM. https://learn.microsoft.com/en-us/azure/databricks/security/network/serverless-network-security/This references to adding privat...
- 1521 Views
- 2 replies
- 0 kudos
- 0 kudos
also interested in doing this. Have federated queries for Classic Databricks cluster pointing to SQL server, but can't find documentation for Serverless plane connecting to SQL server on a VM
- 0 kudos
