- 11021 Views
- 8 replies
- 10 kudos
Library installation in cluster taking a long time
I am trying to install "pycaret" libraray in cluster using whl file.But it is creating conflict in the dependency sometimes (not always, sometimes it works too.) ​My questions are -1 - How to install libraries in cluster only single time (Maybe from ...
- 11021 Views
- 8 replies
- 10 kudos
- 10 kudos
@Retired_modWhat about question #1, which is what subsequent comments to this thread have been referring to? To recap the question: is it possible for "cluster-installed" libraries to be cached in such a way that they aren't completely reinstalled ev...
- 10 kudos
- 33844 Views
- 6 replies
- 10 kudos
Resolved! Table write command stuck "Filtering files for query."
Hello all,Background:I am having an issue today with databricks using pyspark-sql and writing a delta table. The dataframe is made by doing an inner join between two tables and that is the table which I am trying to write to a delta table. The table ...
- 33844 Views
- 6 replies
- 10 kudos
- 10 kudos
Even if I vacuum and optimize, it keeps getting stuck.cluster type is r6gd.xlarge min:4, max:6driver type is r6gd.2xlarge
- 10 kudos
- 23659 Views
- 9 replies
- 9 kudos
Cluster occasionally fails to launch
I have a daily running notebook that occasionally fails with the error:"Run result unavailable: job failed with error message Unexpected failure while waiting for the cluster Some((xxxxxxxxxxxxxxx) )to be readySome(: Cluster xxxxxxxxxxxxxxxx is in un...
- 23659 Views
- 9 replies
- 9 kudos
- 9 kudos
Cluster 'xxxxxxx' was terminated. Reason: WORKER_SETUP_FAILURE (SERVICE_FAULT). Parameters: databricks_error_message:DBFS Daemomn is not reachable., gcp_error_message:Unable to reach the colocated DBFS Daemon.Can Anyone help me how can we resolve thi...
- 9 kudos
- 9690 Views
- 9 replies
- 4 kudos
Resolved! Parsing 5 GB json file is running long on cluster
I was creating delta table from ADLS json input file. but the job was running long while creating delta table from json. Below is my cluster configuration. Is the issue related to cluster config ? Do I need to upgrade the cluster config ?The cluster ...
- 9690 Views
- 9 replies
- 4 kudos
- 4 kudos
with multiline = true, the json is read as a whole and processed as such.I'd try with a beefier cluster.
- 4 kudos
- 10134 Views
- 4 replies
- 3 kudos
Resolved! cluster creation - access mode option
I am a bit lazy and trying to manually recreate a cluster I have in one workspace into another one. The cluster was created some time ago. Looking at the configuration, the access mode field is "custom": When trying to create a new cluster, I do not...
- 10134 Views
- 4 replies
- 3 kudos
- 3 kudos
Hi All - I am new to databricks and trying to create my first workflow. For some reason, the cluster created is of type -"custom". I have not mentioned it anywhere in my asset bundle.Due to this - I cannot create get the Unity Catalog feature. Could ...
- 3 kudos
- 22132 Views
- 6 replies
- 3 kudos
Resolved! How to prevent my cluster to shut down after inactivity
Currently, I am running a cluster that is set to terminate after 60 minutes of inactivity. However, in one of my notebooks, one of the cells is still running. How can I prevent this from happening, if want my notebook to run overnight without monito...
- 22132 Views
- 6 replies
- 3 kudos
- 3 kudos
If a cell is already running ( I assume it's a streaming operation), then I think it doesn't mean that the cluster is inactive. The cluster should be running if a cell is running on it.On the other hand, if you want to keep running your clusters for ...
- 3 kudos
- 4542 Views
- 2 replies
- 0 kudos
- 4542 Views
- 2 replies
- 0 kudos
- 0 kudos
Yes! Cluster's owner/creator can be changed with the REST API - POST /api/2.1/clusters/change-ownerRequest Body sample:{ "cluster_id": "string", "owner_username": "string" }Ref: Clusters API | Change cluster ownerHope this helps!
- 0 kudos
- 5902 Views
- 6 replies
- 3 kudos
Resolved! Is Concurrent Writes from multiple databricks clusters to same delta table on S3 Supported?
Does databricks have support for writing to same Delta Table from multiple clusters concurrently. I am specifically interested to know if there is any solution for https://github.com/delta-io/delta/issues/41 implemented in databricks OR if you have a...
- 5902 Views
- 6 replies
- 3 kudos
- 3 kudos
Please note, the issue noted above [Storage System] Support for AWS S3 (multiple clusters/drivers/JVMs) is for Delta Lake OSS. As noted in this issue as well as Issue 324, as of this writing, S3 lacks putIfAbsent transactional consistency. For Del...
- 3 kudos
- 15318 Views
- 5 replies
- 12 kudos
Cluster start is currently disabled ?
Hi, i'm trying to run the notebooks but it doesn't do any activity.I had to create a cluster in order to start my code.pressing the play button inside of notebook does nothing at all.and the 'compute' , pressing play there on the clusters gives the e...
- 15318 Views
- 5 replies
- 12 kudos
- 12 kudos
This is very common issue I see with community edition. I suppose the only work around is to create new cluster each time. More info on stackoverflow:https://stackoverflow.com/questions/69072694/databricks-community-edition-cluster-wont-start
- 12 kudos
- 7654 Views
- 4 replies
- 3 kudos
- 7654 Views
- 4 replies
- 3 kudos
- 3 kudos
I am having the same issue (Azure Databricks).I have a running compute cluster analytics-compute-cluster running in Single User access mode. The Event Log for the cluster says the cluster is running and the "Driver is healthy".I have Manage permissi...
- 3 kudos
- 18914 Views
- 10 replies
- 6 kudos
Shared cluster configuration that permits `dbutils.fs` commands
My workspace has a couple different types of clusters, and I'm having issues using the `dbutils` filesystem utilities when connected to a shared cluster. I'm hoping you can help me fix the configuration of the shared cluster so that I can actually us...
- 18914 Views
- 10 replies
- 6 kudos
- 6 kudos
Can you not use a No Isolation Shared cluster with Table access controls enabled on workspace level?
- 6 kudos
- 7189 Views
- 8 replies
- 3 kudos
Resolved! How to preserve my database when the cluster is terminated?
Whenever my cluster is terminated, I lose my whole database(I'm not sure if it's related, I made those database with delta format. ) And since the cluster is terminated in 2 hours from not using it, I wake up with no database every morning.I don't wa...
- 7189 Views
- 8 replies
- 3 kudos
- 3 kudos
As the file still in the dbfs you can just recreate the reference of your tables and continue the work, with something like this:db_name = "mydb" from pathlib import Path path_db = f"dbfs:/user/hive/warehouse/{db_name}.db/" tables_dirs = dbutils.fs....
- 3 kudos
- 5706 Views
- 3 replies
- 3 kudos
Resolved! Single-Node cluster works but Multi-Node clusters do not read data.
I am currently working with a VNET injected databricks workspace. At the moment I have mounted a the databricks cluster on an ADLS G2 resource. When running notebooks on a single node that read, transform, and write data we do not encounter any probl...
- 5706 Views
- 3 replies
- 3 kudos
- 3 kudos
@TheDataDexter Did you find a solution to your problem? I am facing the same issue
- 3 kudos
- 8729 Views
- 3 replies
- 2 kudos
My cluster can't access any tables in data catalogs
My goal is to have table access control in the data science and engineering workspace. So I enabled access control to my cluster using this config "spark.databricks.acl.dfAclsEnabled": "true" and my cluster is shown as Table ACLs enabled now (shield ...
- 8729 Views
- 3 replies
- 2 kudos
- 2 kudos
Here is my use case: https://community.databricks.com/t5/data-engineering/structured-streaming-using-delta-as-source-and-delta-as-sink-and/td-p/67825And I get this error: "py4j.security.Py4JSecurityException: Method public org.apache.spark.sql.Datase...
- 2 kudos
- 4452 Views
- 4 replies
- 5 kudos
IP address fix
How can I fix the IP address of my Azure Cluster so that I can whitelist the IP address to run my job daily on my python notebook? Or can I find out the IP address to perform whitelisting? Thanks
- 4452 Views
- 4 replies
- 5 kudos
- 5 kudos
Depends on the scenario. You could expose a single ip address to the external internet, but databricks itself will always use many addresses.
- 5 kudos