cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Engineering & Streaming

Forum Posts

ferbystudy
by New Contributor III
  • 2959 Views
  • 4 replies
  • 3 kudos

Resolved! Can´t read a simple .CSV from a blob

Guys, I am using "Databricks Community" to study. I put some files in a Blob, granted all access but I have no ideia why DB is not reading. Please see the code below and thanks for helping! thanks!

csf
  • 2959 Views
  • 4 replies
  • 3 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 3 kudos

Hi @Fernando Rezende​, Thank you for sharing the solution with us.It would mean a lot if you could select the "Best Answer" to help others find the correct answer faster.This makes that answer appear right after the question, so it's easier to find w...

  • 3 kudos
3 More Replies
gideont
by New Contributor III
  • 3178 Views
  • 3 replies
  • 2 kudos

Resolved! spark sql update really slow

I tried to use Spark as much as possible but experience some regression. Hopefully to get some direction how to use it correctly.I've created a Databricks table using spark.sqlspark.sql('select * from example_view ') \ .write \ .mode('overwr...

image.png
  • 3178 Views
  • 3 replies
  • 2 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 2 kudos

Hi @Vincent Doe​ â€‹, It would mean a lot if you could select the "Best Answer" to help others find the correct answer faster.This makes that answer appear right after the question, so it's easier to find within a thread.It also helps us mark the quest...

  • 2 kudos
2 More Replies
bluetail
by Contributor
  • 4614 Views
  • 2 replies
  • 8 kudos

Resolved! Unable to create a cluster in Databricks with a new subscription

I have upgraded my expired Student subscription to 'Azure subscription 1' in Azure portal today. I want to use Databricks for personal projects as pay-as-you-go.When I go to my Databricks workspace and to my notebook and try to create a cluster,Comp...

  • 4614 Views
  • 2 replies
  • 8 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 8 kudos

Hi @Maria Bruevich​, We haven’t heard from you since the last response from me​ ​, and I was checking back to see if my suggestions helped you.Or else, If you have any solution, please share it with the community, as it can be helpful to others.Also,...

  • 8 kudos
1 More Replies
deficiant_codge
by Contributor II
  • 2039 Views
  • 3 replies
  • 8 kudos

Resolved! Delta Live tables support for UNITY CATALOG

Is there any upcoming update in which UC will support DLT? if yes any expected ETA?

  • 2039 Views
  • 3 replies
  • 8 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 8 kudos

Hi @Rahul Mishra​, We haven’t heard from you since the last response from @Pat Sienkiewicz​ and me ​, and I was checking back to see if our suggestions helped you.Or else, If you have any solution, please share it with the community, as it can be hel...

  • 8 kudos
2 More Replies
Viren123
by Contributor
  • 8327 Views
  • 3 replies
  • 3 kudos

Resolved! Error : MALFORMED_REQUEST

Hello,I get error for below json. Can you please advice what am I missing here?{    "error_code": "MALFORMED_REQUEST",    "message": "Invalid JSON given in the body of the request - failed to parse given JSON"_________________________________________...

  • 8327 Views
  • 3 replies
  • 3 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 3 kudos

Hi @Viren Devi​  ​, It would mean a lot if you could select the "Best Answer" to help others find the correct answer faster.This makes that answer appear right after the question, so it's easier to find within a thread.It also helps us mark the quest...

  • 3 kudos
2 More Replies
Tripalink
by New Contributor III
  • 6222 Views
  • 7 replies
  • 2 kudos

Resolved! Failed to fetch archive.ubuntu

I am trying to use selenium webdriver to do a scraping project in Databricks. The notebook used to run properly but now has an issue with the Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 fonts-liberation all 1:1.07.4-11 [822 kB]command .In...

  • 6222 Views
  • 7 replies
  • 2 kudos
Latest Reply
Hubert-Dudek
Esteemed Contributor III
  • 2 kudos

Hi, @Dagart Allison​ . I've created a new version of the selenium with the databricks manual. Please look here https://community.databricks.com/s/feed/0D58Y00009SWgVuSAL

  • 2 kudos
6 More Replies
Tripalink
by New Contributor III
  • 3577 Views
  • 2 replies
  • 0 kudos

Using Selenium Chrome Driver in Databricks, runs the first time but fails after that

I have a notebook that uses a Selenium Web Driver for Chrome and it works the first time I run the notebook. If I run the notebook again, it will not work and gives the error message: WebDriverException: Message: unknown error: unable to discover op...

  • 3577 Views
  • 2 replies
  • 0 kudos
Latest Reply
Hubert-Dudek
Esteemed Contributor III
  • 0 kudos

Hi, @Dagart Allison​ . I've created a new version of the selenium with the databricks manual. Please look here https://community.databricks.com/s/feed/0D58Y00009SWgVuSAL

  • 0 kudos
1 More Replies
Arun_tsr
by New Contributor III
  • 5456 Views
  • 6 replies
  • 2 kudos

How to do bucketing in Databricks?

We are migrating a job from onprem to databricks. We are trying to optimize the jobs but couldn't use bucketing because by default databricks stores all tables as delta table and it shows error that bucketing is not supported for delta. Is there anyw...

  • 5456 Views
  • 6 replies
  • 2 kudos
Latest Reply
Pat
Honored Contributor III
  • 2 kudos

Hi @Arun Balaji​ ,bucketing is not supported for the delta tables as you have noticed.For the optimization and best practices with delta tables check this:https://docs.databricks.com/optimizations/index.htmlhttps://docs.databricks.com/delta/best-prac...

  • 2 kudos
5 More Replies
164079
by Contributor II
  • 8814 Views
  • 15 replies
  • 23 kudos

Resolved! Users are failing query data from S3 bucket

Hi team, Users are unable run select on data located on S3 buckets, S3 permission are ok.The only way they manage do it by granted the databricks workspace admin permission.Attached the error.Thanks!

  • 8814 Views
  • 15 replies
  • 23 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 23 kudos

Hi @Avi Edri​ , We haven’t heard from you since the last response from @Pat Sienkiewicz​ and @karthik p​  , and I was checking back to see if their suggestions helped you.If you have any solution, please share it with the community, as it can be help...

  • 23 kudos
14 More Replies
THIAM_HUATTAN
by Valued Contributor
  • 3310 Views
  • 5 replies
  • 4 kudos

Resolved! Using R, how do we write csv file to say dbfs:/tmp?

let us say I already have the data 'TotalData'write.csv(TotalData,file='/tmp/TotalData.csv',row.names = FALSE)I do not see any error from abovewhen I list files below:%fs ls /tmpI do not see any files written there. Why?

  • 3310 Views
  • 5 replies
  • 4 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 4 kudos

Hi @THIAM HUAT TAN​  We haven’t heard from you since the last response from @Cedric Law Hing Ping​ , and I was checking back to see if you have a resolution yet. If you have any solution, please share it with the community as it can be helpful to oth...

  • 4 kudos
4 More Replies
Arun_tsr
by New Contributor III
  • 1707 Views
  • 2 replies
  • 0 kudos

Spark SQL output multiple small files

We are having multiple joins involving a large table (about 500gb in size). The output of the joins is stored into multiple small files each of size 800kb-1.5mb. Because of this the job is split into multiple tasks and taking a long time to complete....

Spark UI metrics
  • 1707 Views
  • 2 replies
  • 0 kudos
Latest Reply
Debayan
Esteemed Contributor III
  • 0 kudos

Hi @Arun Balaji​ , Could you please provide the error message you are receiving?

  • 0 kudos
1 More Replies
Ranjeeth
by New Contributor
  • 1476 Views
  • 2 replies
  • 2 kudos
  • 1476 Views
  • 2 replies
  • 2 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 2 kudos

Hi @Ranjeeth Rikkala​ , We haven’t heard from you since the last response from @Pat Sienkiewicz​ , and I was checking back to see if his suggestions helped you. Or else, If you have any solution, please share it with the community, as it can be helpf...

  • 2 kudos
1 More Replies
LJ
by New Contributor II
  • 1650 Views
  • 2 replies
  • 5 kudos

Resolved! Accept widget value during runtime from user

list1 = ('alpha', 'beta', 'gamma', 'eta', 'Theta')list2 = ('alpha', 'beta')df1 = spark.createDataFrame(list1, 'String').withColumnRenamed('value', 'dataset')df2 = spark.createDataFrame(list2, 'String').withColumnRenamed('value', 'dataset')df = df1.ex...

  • 1650 Views
  • 2 replies
  • 5 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 5 kudos

Hi @Liju Thomas​, We haven’t heard from you since the last response from @Hubert Dudek​ , and I was checking back to see if you have a resolution yet. If you have any solution, please share it with the community, as it can be helpful to others. Other...

  • 5 kudos
1 More Replies

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels