cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

niels
by New Contributor III
  • 1774 Views
  • 2 replies
  • 0 kudos

Why is my Plotly graph squished?

fig = make_subplots(1,4) cols = ['OrderValue', 'TransactionPrice', 'ProductPrice', 'ProductUnits']     for i, col in enumerate(cols): fig.add_trace( go.Histogram(x=silver_df.select(col).toPandas()[col]), row=1, col=i+1 )   p =...

  • 1774 Views
  • 2 replies
  • 0 kudos
Latest Reply
Vidula
Honored Contributor
  • 0 kudos

Hey there @Niels Ota​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.T...

  • 0 kudos
1 More Replies
askme
by New Contributor II
  • 2348 Views
  • 2 replies
  • 2 kudos

Databricks jobs UPdate/Reset API throws an unexpected error

{ "error_code": "INVALID_PARAMETER_VALUE", "message": "Missing required field: job_id"}I have a test job cluster and I need to update the docker image filed with the other version using reset/update job API. I went through the documentation of data b...

  • 2348 Views
  • 2 replies
  • 2 kudos
Latest Reply
Vidula
Honored Contributor
  • 2 kudos

Hey there @radha kilaru​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from yo...

  • 2 kudos
1 More Replies
Venkat1
by New Contributor II
  • 4845 Views
  • 3 replies
  • 1 kudos

Resolved! Error: "Backend service unavailable" when trying to start a cluster in community edition.

I am using community edition of databricks for learning and hands-on projects. However, when I try to create a cluster today, I am getting an error popup- "Backend service unavailable". I would like to know if it is a problem with my account or a bac...

  • 4845 Views
  • 3 replies
  • 1 kudos
Latest Reply
Vidula
Honored Contributor
  • 1 kudos

Hey there @Venkat K​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Th...

  • 1 kudos
2 More Replies
Tahseen0354
by Valued Contributor
  • 4277 Views
  • 3 replies
  • 1 kudos

Resolved! Can I add custom cluster tag from init script ?

Hi, is it possible to add custom tags from init script during cluster initialization ? We would like to automatically add custom tags whenever someone creates a new cluster in databricks.

  • 4277 Views
  • 3 replies
  • 1 kudos
Latest Reply
Prabakar
Databricks Employee
  • 1 kudos

Hi @Md Tahseen Anam​ I don't think there is a possibility to use an init script for cust tags. But the easiest way is to use cluster policies. You can mention a list of custom tags in the policy so that you can simply add the policy to the cluster wh...

  • 1 kudos
2 More Replies
Paramesh
by New Contributor II
  • 4514 Views
  • 3 replies
  • 2 kudos

Resolved! How to read multiple tiny XML files in parallel

Hi team, we are trying to read multiple tiny XML files, able to parse them using the data bricks XML jar, but is there any way to read these files in parallel and distribute the load across the cluster? right now our job is taking 90% of the time rea...

  • 4514 Views
  • 3 replies
  • 2 kudos
Latest Reply
Paramesh
New Contributor II
  • 2 kudos

Thank you @Hubert Dudek​ for the suggestion. Similar to your recommendation, we added a step in our pipeline to merge the small files to large files and make them available for the spark job.

  • 2 kudos
2 More Replies
ronaldolopes
by New Contributor
  • 29107 Views
  • 1 replies
  • 0 kudos

Resolved! Exporting data from databricks to external csv

I need to export some data from the database to csv which will be downloaded to another application. What would be the procedure for that? I don't have a lot of knowledge in DataBricks and I didn't find much information in the documentation.Thanks.

  • 29107 Views
  • 1 replies
  • 0 kudos
Latest Reply
AmanSehgal
Honored Contributor III
  • 0 kudos

You can manually download data to your local in CSV from databricks notebook cell and pass it to your another application.Your application can run Databricks notebook inside a workflow via an API that writes data to S3 bucket in CSV and in response y...

  • 0 kudos
raj_123469
by New Contributor II
  • 1720 Views
  • 2 replies
  • 2 kudos
  • 1720 Views
  • 2 replies
  • 2 kudos
Latest Reply
Vidula
Honored Contributor
  • 2 kudos

Hi @rajat kumar​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks...

  • 2 kudos
1 More Replies
niels
by New Contributor III
  • 2043 Views
  • 2 replies
  • 0 kudos

Azure SA mounted but can't load files

I am attempting to load an excel file that's located in a blob storage that I've mounted. In the first cell, when I use the dbutils.fs.ls command, I can see the file I want to load. However, when I try to actually load it, it can't find the file. It ...

  • 2043 Views
  • 2 replies
  • 0 kudos
Latest Reply
Vidula
Honored Contributor
  • 0 kudos

Hi @Niels Ota​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks!

  • 0 kudos
1 More Replies
Herkimer
by New Contributor II
  • 2406 Views
  • 3 replies
  • 0 kudos

Is it possible to install databricks-cli on a shared laptop.

I have a government furnished laptop (GFE). My normal user is not an admin on the laptop. I have a separate admin login on the laptop. I was able to install databricks-cli as the admin user but it installed under that users appdata python path which ...

  • 2406 Views
  • 3 replies
  • 0 kudos
Latest Reply
Vidula
Honored Contributor
  • 0 kudos

Hey there @John Zajic​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you....

  • 0 kudos
2 More Replies
RantoB
by Valued Contributor
  • 5952 Views
  • 3 replies
  • 4 kudos

Resolved! DeltaTable' object has no attribute 'clone'

Hello, I use delta on my local machine and I would like to clone a table, however the cloning is not working.I have the last version of delta installed (delta-spark==2.0.0) but the clone method does not exist in the python module.With this code :delt...

  • 5952 Views
  • 3 replies
  • 4 kudos
Latest Reply
Vidula
Honored Contributor
  • 4 kudos

Hello @Bertrand BURCKER​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from yo...

  • 4 kudos
2 More Replies
Tahseen0354
by Valued Contributor
  • 1829 Views
  • 2 replies
  • 0 kudos

User's name is empty sometimes when a new user is added from admin console

Hi, when I add a new user from admin console, the name of the user is empty. It does not happen all the time. For some users, the username and name both are available. But for some new users, the value in the name column in the users list is empty. W...

  • 1829 Views
  • 2 replies
  • 0 kudos
Chris_Shehu
by Valued Contributor III
  • 7682 Views
  • 8 replies
  • 2 kudos

Resolved! Do compute resources get removed after not being used for x number of days?

Currently we're getting reports of compute resources disappearing from one of our lesser used databricks platforms. I just turned on logging to see if we can find something but I'm wondering if a compute gets removed if it hasn't been used after so l...

  • 7682 Views
  • 8 replies
  • 2 kudos
Latest Reply
Hanna0805050
New Contributor II
  • 2 kudos

Pest Control Software to Grow Your Business Choosing the best pest control software for your business can have a powerful impact on your productivity. Fieldwork can help your workforce repel downtime, attract clients, get organized and get everything...

  • 2 kudos
7 More Replies
Data_Engineer3
by Contributor III
  • 6847 Views
  • 4 replies
  • 1 kudos

Unable to read data from Elasticsearch with spark in Databricks.

When I am trying to read data from elasticsearch by spark sql, it throw an error like RuntimeException: Error while encoding: java.lang.RuntimeException: scala.collection.convert.Wrappers$JListWrapper is not a valid external type for schema of string...

  • 6847 Views
  • 4 replies
  • 1 kudos
Latest Reply
Vidula
Honored Contributor
  • 1 kudos

Hi there @KARTHICK N​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.T...

  • 1 kudos
3 More Replies
rodrigocms
by New Contributor
  • 2449 Views
  • 2 replies
  • 0 kudos

Connect to SSAS

Hello everyone,I need to connect Databricks Pyspark to get information from Power BI XLMA EndPoint - the end point work as an SSAS host.So, I'm trying to find what I need to do to connect to SSAS tabular. Can anyone help?Many thanks.Rodrigo Souza

  • 2449 Views
  • 2 replies
  • 0 kudos
Latest Reply
Vidula
Honored Contributor
  • 0 kudos

Hey there @Rodrigo Camara de Souza​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to h...

  • 0 kudos
1 More Replies

Join Us as a Local Community Builder!

Passionate about hosting events and connecting people? Help us grow a vibrant local community—sign up today to get started!

Sign Up Now
Labels