cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

TheDataDexter
by New Contributor III
  • 2260 Views
  • 4 replies
  • 3 kudos

Resolved! Single-Node cluster works but Multi-Node clusters do not read data.

I am currently working with a VNET injected databricks workspace. At the moment I have mounted a the databricks cluster on an ADLS G2 resource. When running notebooks on a single node that read, transform, and write data we do not encounter any probl...

  • 2260 Views
  • 4 replies
  • 3 kudos
Latest Reply
ellafj
New Contributor II
  • 3 kudos

@TheDataDexter Did you find a solution to your problem? I am facing the same issue

  • 3 kudos
3 More Replies
bamhn
by New Contributor II
  • 2619 Views
  • 3 replies
  • 2 kudos

My cluster can't access any tables in data catalogs

My goal is to have table access control in the data science and engineering workspace. So I enabled access control to my cluster using this config "spark.databricks.acl.dfAclsEnabled": "true" and my cluster is shown as Table ACLs enabled now (shield ...

image.png image
  • 2619 Views
  • 3 replies
  • 2 kudos
Latest Reply
Karthik_Venu
New Contributor II
  • 2 kudos

Here is my use case: https://community.databricks.com/t5/data-engineering/structured-streaming-using-delta-as-source-and-delta-as-sink-and/td-p/67825And I get this error: "py4j.security.Py4JSecurityException: Method public org.apache.spark.sql.Datase...

  • 2 kudos
2 More Replies
Jon
by New Contributor II
  • 2189 Views
  • 4 replies
  • 5 kudos

IP address fix

How can I fix the IP address of my Azure Cluster so that I can whitelist the IP address to run my job daily on my python notebook? Or can I find out the IP address to perform whitelisting? Thanks

  • 2189 Views
  • 4 replies
  • 5 kudos
Latest Reply
-werners-
Esteemed Contributor III
  • 5 kudos

Depends on the scenario.  You could expose a single ip address to the external internet, but databricks itself will always use many addresses.

  • 5 kudos
3 More Replies
Braxx
by Contributor II
  • 2804 Views
  • 3 replies
  • 3 kudos

Resolved! cluster creation - access mode option

I am a bit lazy and trying to manually recreate a cluster I have in one workspace into another one. The cluster was created some time ago. Looking at the configuration, the access mode field is "custom": When trying to create a new cluster, I do not...

Captureaa Capturebb
  • 2804 Views
  • 3 replies
  • 3 kudos
Latest Reply
yatharth
New Contributor III
  • 3 kudos

I am facing the same type of issue and now changing back to old UI option is not available, cause of this issue i am not able to view cluster metrics

  • 3 kudos
2 More Replies
pokus
by New Contributor III
  • 3415 Views
  • 3 replies
  • 2 kudos

Resolved! use DeltaLog class in databricks cluster

I need to use DeltaLog class in the code to get the AddFiles dataset. I have to keep the implemented code in a repo and run it in databricks cluster. Some docs say to use org.apache.spark.sql.delta.DeltaLog class, but it seems databricks gets rid of ...

  • 3415 Views
  • 3 replies
  • 2 kudos
Latest Reply
dbal
New Contributor III
  • 2 kudos

Thanks for providing a solution @pokus .What I dont understand is why Databricks cannot provide the DeltaLog at runtime. How can this be the official solution? We need a better solution for this instead of depending on reflections.

  • 2 kudos
2 More Replies
Spencer_Kent
by New Contributor III
  • 5501 Views
  • 7 replies
  • 3 kudos

Shared cluster configuration that permits `dbutils.fs` commands

My workspace has a couple different types of clusters, and I'm having issues using the `dbutils` filesystem utilities when connected to a shared cluster. I'm hoping you can help me fix the configuration of the shared cluster so that I can actually us...

insufficient_permissions_on_shared_cluster shared_cluster_config individual_use_cluster
  • 5501 Views
  • 7 replies
  • 3 kudos
Latest Reply
Nikhil_G
New Contributor II
  • 3 kudos

There are two ways to grant access to DBFS using ANY FILE:To User: GRANT SELECT ON ANY FILE TO '<user_mail_id>'To Group: GRANT SELECT ON ANY FILE TO '<group_name>'"

  • 3 kudos
6 More Replies
Yoni
by New Contributor
  • 10827 Views
  • 5 replies
  • 3 kudos

Resolved! MLFlow failed: You haven't configured the CLI yet

I'm getting an errorYou haven’t configured the CLI yet! Please configure by entering `/databricks/python_shell/scripts/db_ipykernel_launcher.py configure`My cluster is running Databricks Runtime Version 10.1I've also installed mlflow to the cluster l...

  • 10827 Views
  • 5 replies
  • 3 kudos
Latest Reply
HemantKumar
New Contributor II
  • 3 kudos

dbutils.library.restartPython()Add that after you run the pip install mlflow, it worked for me in a non-ML cluster

  • 3 kudos
4 More Replies
Murthy1
by Contributor II
  • 3344 Views
  • 2 replies
  • 0 kudos

How can we use existing all purpose cluster for a DLT pipeline?

I understand that DLT is a separate job compute but I would like to use an existing all purpose cluster for the DLT pipeline. Is there a way I can achieve this?

  • 3344 Views
  • 2 replies
  • 0 kudos
Latest Reply
Kaniz
Community Manager
  • 0 kudos

Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers your question?This...

  • 0 kudos
1 More Replies
esauesp_co
by New Contributor III
  • 2884 Views
  • 5 replies
  • 1 kudos

Resolved! My jobs and cluster were deleted in a suspicious way

I want to know what happen with my cluster and if I can recover it.I entered to my Databricks account and I didn't found my jobs and my cluster. I couldn't find any log of the deleted cluster because the log is into the cluster interface. I entered t...

  • 2884 Views
  • 5 replies
  • 1 kudos
Latest Reply
Sid_databricks
New Contributor II
  • 1 kudos

Dear folks,When the tables has been deleted, then why I am unable to create the table with same name.It continiously giving me error"DeltaAnalysisException: Cannot create table ('`spark_catalog`.`default`.`Customer_Data`'). The associated location ('...

  • 1 kudos
4 More Replies
BWong
by New Contributor III
  • 4161 Views
  • 8 replies
  • 6 kudos

Resolved! Cannot spin up a cluster

HiWhen I try to spin up a cluster, it gives me a bootstrap timeout error{ "reason": { "code": "BOOTSTRAP_TIMEOUT", "parameters": { "databricks_error_message": "[id: InstanceId(i-00b2b7acdd82e5fde), status: INSTANCE_INITIALIZING, workerEnv...

  • 4161 Views
  • 8 replies
  • 6 kudos
Latest Reply
BWong
New Contributor III
  • 6 kudos

Thanks guys. It's indeed a network issue on the AWS side. It's resolved now

  • 6 kudos
7 More Replies
AyushModi038
by New Contributor III
  • 3042 Views
  • 4 replies
  • 3 kudos

Library installation in cluster taking a long time

I am trying to install "pycaret" libraray in cluster using whl file.But it is creating conflict in the dependency sometimes (not always, sometimes it works too.) ​My questions are -1 - How to install libraries in cluster only single time (Maybe from ...

  • 3042 Views
  • 4 replies
  • 3 kudos
Latest Reply
Anonymous
Not applicable
  • 3 kudos

Hi @Ayush Modi​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers yo...

  • 3 kudos
3 More Replies
mbvb_py
by New Contributor II
  • 1699 Views
  • 4 replies
  • 0 kudos

Create cluster error: Backend service unavailable

hello,i'm new to Databricks (community edition account) and encountered a problem just now.When creating a new cluster (default 10.4 LTS) it fails with the following error: Backend service unavailable.I've tried a different runtime > same issue.I've ...

  • 1699 Views
  • 4 replies
  • 0 kudos
Latest Reply
stefnhuy
New Contributor III
  • 0 kudos

Hey mbvb_py,I'm sorry to hear you're facing this "Backend service unavailable" issue with Databricks. I've encountered similar problems in the past, and it can be frustrating. Don't worry; you're not alone in this!From my experience, this error can o...

  • 0 kudos
3 More Replies
MadrasSenpai
by New Contributor II
  • 994 Views
  • 3 replies
  • 2 kudos

How to install cmdstanpy in dbx cluster

I have built an HMC model using cmdstand. In my local machine, I have install cmdstan for the following approach. import cmdstanpy cmdstanpy.install_cmdstan()But in Databricks I need to reinstall it every time when I train a new model, from the noteb...

  • 994 Views
  • 3 replies
  • 2 kudos
Latest Reply
Anonymous
Not applicable
  • 2 kudos

Hi @Rajamannar Aanjaram Krishnamoorthy​ Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question. Thanks.

  • 2 kudos
2 More Replies
Serhii
by Contributor
  • 1054 Views
  • 3 replies
  • 1 kudos

Could not launch jobs due to node_type_id (instance) unavailability

I am running hourly job on a cluster using p3.2xlarge GPU instance, but sometimes cluster couldn't start due to instance unavailability. I wander is there is any fallback mechanism to, for example, try a different instance type if one is not availabl...

  • 1054 Views
  • 3 replies
  • 1 kudos
Latest Reply
abagshaw
New Contributor III
  • 1 kudos

 (AWS only) For anyone experiencing capacity related cluster launch failures on non-GPU instance types, AWS Fleet instance types are now GA and available for clusters and instance pools. They help improve chance of successful cluster launch by allowi...

  • 1 kudos
2 More Replies
gilo12
by New Contributor III
  • 421 Views
  • 0 replies
  • 1 kudos

Connect to warehouse vs connect to compute/cluster

When I create a warehouse, there are connection details which I was able to successfully use, but there are also connection details when I navigate in the console to Compute->configuration->Advanced options-> JDBC/ODBC What is the difference between ...

  • 421 Views
  • 0 replies
  • 1 kudos
Labels