cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

pokus
by New Contributor III
  • 7997 Views
  • 2 replies
  • 2 kudos

Resolved! use DeltaLog class in databricks cluster

I need to use DeltaLog class in the code to get the AddFiles dataset. I have to keep the implemented code in a repo and run it in databricks cluster. Some docs say to use org.apache.spark.sql.delta.DeltaLog class, but it seems databricks gets rid of ...

  • 7997 Views
  • 2 replies
  • 2 kudos
Latest Reply
dbal
New Contributor III
  • 2 kudos

Thanks for providing a solution @pokus .What I dont understand is why Databricks cannot provide the DeltaLog at runtime. How can this be the official solution? We need a better solution for this instead of depending on reflections.

  • 2 kudos
1 More Replies
Yoni
by New Contributor
  • 15102 Views
  • 5 replies
  • 3 kudos

Resolved! MLFlow failed: You haven't configured the CLI yet

I'm getting an errorYou haven’t configured the CLI yet! Please configure by entering `/databricks/python_shell/scripts/db_ipykernel_launcher.py configure`My cluster is running Databricks Runtime Version 10.1I've also installed mlflow to the cluster l...

  • 15102 Views
  • 5 replies
  • 3 kudos
Latest Reply
HemantKumar
New Contributor II
  • 3 kudos

dbutils.library.restartPython()Add that after you run the pip install mlflow, it worked for me in a non-ML cluster

  • 3 kudos
4 More Replies
esauesp_co
by New Contributor III
  • 6080 Views
  • 5 replies
  • 1 kudos

Resolved! My jobs and cluster were deleted in a suspicious way

I want to know what happen with my cluster and if I can recover it.I entered to my Databricks account and I didn't found my jobs and my cluster. I couldn't find any log of the deleted cluster because the log is into the cluster interface. I entered t...

  • 6080 Views
  • 5 replies
  • 1 kudos
Latest Reply
Sid_databricks
New Contributor II
  • 1 kudos

Dear folks,When the tables has been deleted, then why I am unable to create the table with same name.It continiously giving me error"DeltaAnalysisException: Cannot create table ('`spark_catalog`.`default`.`Customer_Data`'). The associated location ('...

  • 1 kudos
4 More Replies
BWong
by New Contributor III
  • 7677 Views
  • 8 replies
  • 6 kudos

Resolved! Cannot spin up a cluster

HiWhen I try to spin up a cluster, it gives me a bootstrap timeout error{ "reason": { "code": "BOOTSTRAP_TIMEOUT", "parameters": { "databricks_error_message": "[id: InstanceId(i-00b2b7acdd82e5fde), status: INSTANCE_INITIALIZING, workerEnv...

  • 7677 Views
  • 8 replies
  • 6 kudos
Latest Reply
BWong
New Contributor III
  • 6 kudos

Thanks guys. It's indeed a network issue on the AWS side. It's resolved now

  • 6 kudos
7 More Replies
mbvb_py
by New Contributor II
  • 3237 Views
  • 4 replies
  • 0 kudos

Create cluster error: Backend service unavailable

hello,i'm new to Databricks (community edition account) and encountered a problem just now.When creating a new cluster (default 10.4 LTS) it fails with the following error: Backend service unavailable.I've tried a different runtime > same issue.I've ...

  • 3237 Views
  • 4 replies
  • 0 kudos
Latest Reply
stefnhuy
New Contributor III
  • 0 kudos

Hey mbvb_py,I'm sorry to hear you're facing this "Backend service unavailable" issue with Databricks. I've encountered similar problems in the past, and it can be frustrating. Don't worry; you're not alone in this!From my experience, this error can o...

  • 0 kudos
3 More Replies
MadrasSenpai
by New Contributor II
  • 1814 Views
  • 2 replies
  • 2 kudos

How to install cmdstanpy in dbx cluster

I have built an HMC model using cmdstand. In my local machine, I have install cmdstan for the following approach. import cmdstanpy cmdstanpy.install_cmdstan()But in Databricks I need to reinstall it every time when I train a new model, from the noteb...

  • 1814 Views
  • 2 replies
  • 2 kudos
Latest Reply
Anonymous
Not applicable
  • 2 kudos

Hi @Rajamannar Aanjaram Krishnamoorthy​ Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question. Thanks.

  • 2 kudos
1 More Replies
Serhii
by Contributor
  • 1902 Views
  • 3 replies
  • 1 kudos

Could not launch jobs due to node_type_id (instance) unavailability

I am running hourly job on a cluster using p3.2xlarge GPU instance, but sometimes cluster couldn't start due to instance unavailability. I wander is there is any fallback mechanism to, for example, try a different instance type if one is not availabl...

  • 1902 Views
  • 3 replies
  • 1 kudos
Latest Reply
abagshaw
New Contributor III
  • 1 kudos

 (AWS only) For anyone experiencing capacity related cluster launch failures on non-GPU instance types, AWS Fleet instance types are now GA and available for clusters and instance pools. They help improve chance of successful cluster launch by allowi...

  • 1 kudos
2 More Replies
gilo12
by New Contributor III
  • 985 Views
  • 0 replies
  • 1 kudos

Connect to warehouse vs connect to compute/cluster

When I create a warehouse, there are connection details which I was able to successfully use, but there are also connection details when I navigate in the console to Compute->configuration->Advanced options-> JDBC/ODBC What is the difference between ...

  • 985 Views
  • 0 replies
  • 1 kudos
bshirdi
by New Contributor II
  • 8295 Views
  • 1 replies
  • 2 kudos

Getting HTTP 502 bad gateway error!

Hello all,I am suddenly getting an HTTP 502 and DRIVER_LIBRARY_INSTALLATION_FAILURE error during the Python library installation when the cluster gets initialized. I have around 10 Python packages out of which 2-3, packages always failed to install a...

image.png
  • 8295 Views
  • 1 replies
  • 2 kudos
Latest Reply
Anonymous
Not applicable
  • 2 kudos

Hi @Bhargav Shir​ Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question. Thanks.

  • 2 kudos
Aanchal
by New Contributor III
  • 2699 Views
  • 4 replies
  • 2 kudos

Resolved! Unable to lunch cluster in Databricks as my azure subscription has been disabled

Azure subscription- disabledDatabricks subscription- free trial 13 day leftDatabricks host- AzureThe cluster is not getting created as my Azure subscription has been disabled after a month of free trial. However, Databricks subscription has still got...

  • 2699 Views
  • 4 replies
  • 2 kudos
Latest Reply
Anonymous
Not applicable
  • 2 kudos

Hi @Aanchal Soni​ We haven't heard from you since the last response from @Tyler Retzlaff​ â€‹, and I was checking back to see if her suggestions helped you.Or else, If you have any solution, please share it with the community, as it can be helpful to o...

  • 2 kudos
3 More Replies
ossinova
by Contributor II
  • 1777 Views
  • 1 replies
  • 0 kudos

Creating cluster from ADF linked service with Workspace init script

Similar issue: https://stackoverflow.com/questions/76220211/create-new-databricks-cluster-from-adf-linked-service-with-initscripts-from-abfsI am trying to create clusters using ADF linked service where the cluster is configured with a init script. As...

  • 1777 Views
  • 1 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Oscar Dyremyhr​ Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question. Thanks.

  • 0 kudos
Phani1
by Valued Contributor II
  • 2946 Views
  • 2 replies
  • 1 kudos

Integration Dolly with Databricks

Hi Databricks Team,Could you please share any links /docs/Sample notebooks to integrate Dolly with Databricks, our aim is to generate SQL queries based on the free text and execute it via databricks cluster/SQL warehouse.

  • 2946 Views
  • 2 replies
  • 1 kudos
Latest Reply
sean_owen
Databricks Employee
  • 1 kudos

https://www.dbdemos.ai/demo.html?demoName=llm-dolly-chatbot is a good demonstration of Dolly (or really any LLM) for question answering. LLMs like this are not for SQL generation, but other LLMs are, like starcoderbase

  • 1 kudos
1 More Replies
nav
by New Contributor II
  • 4828 Views
  • 8 replies
  • 0 kudos

R packages not getting installed on cluster when creating cluster from dockerfile

I'm trying to use dockerfile to create a cluster which has Robyn (https://facebookexperimental.github.io/Robyn/) and other R libraries installed. But it is failing to install the R libraries to the cluster. When I run the container in interactive mod...

  • 4828 Views
  • 8 replies
  • 0 kudos
Latest Reply
workingtogetdbw
New Contributor II
  • 0 kudos

What there has been no answer here!  @Debayan Mukherjee​ @Vartika Nain​ So I am running into this same problem as the idea of having to wait 45 minutes for libraries to install is absolutely wild as well as I have done everything outside of working w...

  • 0 kudos
7 More Replies
Retko
by Contributor
  • 10004 Views
  • 3 replies
  • 3 kudos

Resolved! Data Tab is not showing any databases and tables even though cluster is running (Community edition)

Hi, I have a cluster running: But I dont see anything in Data Tab: As you can see it tells about some error, but error appeared after I deleted clusters which were terminated. Before it said something that cluster is not running, dont remember exactl...

image image image
  • 10004 Views
  • 3 replies
  • 3 kudos
Latest Reply
Rajani
Contributor II
  • 3 kudos

@Retko Okter​ You need to enable DBFS File Browser from the Admin Settings hope this helps.

  • 3 kudos
2 More Replies
de-hru
by New Contributor III
  • 26188 Views
  • 4 replies
  • 1 kudos

Resolved! How to add pre-commit hook to the Git Client on Databricks Cluster?

I'd like to add a Git pre-commit hook to the Databricks Cluster.This pre-commit hook should be executed when pushing to GitHub.Why would I need a pre-commit hook on a Databricks Cluster?My goal is to run blackbricks and format all notebooks automatic...

  • 26188 Views
  • 4 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @Dejan Hrubenja​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Tha...

  • 1 kudos
3 More Replies
Labels