cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Engineering & Streaming

Forum Posts

Manjusha
by New Contributor II
  • 1896 Views
  • 1 replies
  • 1 kudos

SocketTimeout exception when running a display command on spark dataframe

I am using runtime 9.1LTSI have a R notebook that reads a csv into a R dataframe and does some transformations and finally is converted to spark dataframe using the createDataFrame function.after that when I call the display function on this spark da...

  • 1896 Views
  • 1 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @Manjusha Unnikrishnan​ Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question first. Or else bricksters will get back to you soon. Thanks.

  • 1 kudos
MBV3
by New Contributor III
  • 1568 Views
  • 1 replies
  • 2 kudos

Delete a file from GCS folder

What is the best way to delete files from the gcp bucket inside spark job?

  • 1568 Views
  • 1 replies
  • 2 kudos
Latest Reply
Unforgiven
Valued Contributor III
  • 2 kudos

@M Baig​ yes you need just to create service account for databricks and than assign storage admin role to bucket. After that you can mount GCS standard way:bucket_name = "<bucket-name>"mount_name = "<mount-name>"dbutils.fs.mount("gs://%s" % bucket_na...

  • 2 kudos
User16844487905
by New Contributor III
  • 3667 Views
  • 4 replies
  • 5 kudos

AWS quickstart - Cloudformation failure When deploying your workspace with the recommended AWS quickstart method, a Cloudformation template will be la...

AWS quickstart - Cloudformation failureWhen deploying your workspace with the recommended AWS quickstart method, a Cloudformation template will be launched in your AWS account. If you experience a failure with the error message along the lines of ROL...

Screen Shot 2021-10-12 at 11.46.28 AM Screen Shot 2021-10-13 at 3.09.01 PM
  • 3667 Views
  • 4 replies
  • 5 kudos
Latest Reply
yalun
New Contributor III
  • 5 kudos

How do I launch the "Quickstart" again? Where is it in the console?

  • 5 kudos
3 More Replies
SM
by New Contributor III
  • 4106 Views
  • 4 replies
  • 10 kudos

How to use Azure Data lake as a storage location to store the Delta Live Tables?

I am trying write data into Azure Datalake. I am reading files from Azure Blob Storage however when I try to create the Delta Live Table to Azure Datalake I get error the following errorshaded.databricks.azurebfs.org.apache.hadoop.fs.azurebfs.contrac...

image
  • 4106 Views
  • 4 replies
  • 10 kudos
Latest Reply
RThornton
New Contributor III
  • 10 kudos

@Kaniz Fatma​ I don't think you quite understand the question. I'm running into the same problem. When creating a Delta Live Table pipeline to write to Azure Data Lake Storage (abfss://etc...) as the Storage Location, the pipeline fails with the erro...

  • 10 kudos
3 More Replies
AmarJT
by New Contributor II
  • 2301 Views
  • 2 replies
  • 6 kudos

Lakehouse Fundamentals Accreditation badge not received

Hi Team,I have successfully passed the test after completion of the course. But i have not received any badge from your side. I have just been provided a certificate. Certificate ID:ID: E-E04YDVAs mentioned in the web portals i tried accessing "http...

image image
  • 2301 Views
  • 2 replies
  • 6 kudos
Latest Reply
Geeta1
Valued Contributor
  • 6 kudos

Hi @Amarjeet Kumar​ , you will receive the badge in a day after completion. Even I received it a day after I cleared the exam. If you don't receive it the next day also, then you can raise a ticket at  https://help.databricks.com/s/contact-us?ReqType...

  • 6 kudos
1 More Replies
yogu
by Honored Contributor III
  • 1536 Views
  • 2 replies
  • 18 kudos

Resolved! Explain about "Activate Gift Certificate" section in Databricks Community Rewards.

Hello everyone,Can any one explain about the Active Gift Certificate in in Databricks Community Rewards. And how to use it?

image
  • 1536 Views
  • 2 replies
  • 18 kudos
Latest Reply
-werners-
Esteemed Contributor III
  • 18 kudos

it boils down to this:you earn points with forum interaction. Those points can be exchanged for 'credits'.With those credits you can buy Databricks swag.Your lifetime points (so the cumulated amount of points) are not affected by this.

  • 18 kudos
1 More Replies
-werners-
by Esteemed Contributor III
  • 1683 Views
  • 3 replies
  • 22 kudos

Resolved! Package cells (scala), who uses them?

So I was wondering who uses package cells in scala?We have this library (jar) which has some useful functions we use all over the place. But that's about it. So I think we can do the same thing without a jar but with package cells.But I never hear ...

  • 1683 Views
  • 3 replies
  • 22 kudos
Latest Reply
Anonymous
Not applicable
  • 22 kudos

Hi @Werner Stinckens​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.T...

  • 22 kudos
2 More Replies
Constantino
by New Contributor III
  • 1166 Views
  • 1 replies
  • 2 kudos

Is there any way to prevent non-admin users from creating new jobs?

This is specific to creating new jobs, I understand that various permissions can be set on existing jobs using job access control. This seems to suggest no, I can't find anything in the Databricks docs either.

  • 1166 Views
  • 1 replies
  • 2 kudos
Latest Reply
-werners-
Esteemed Contributor III
  • 2 kudos

nope.Looked for that too, but it does not seem to be possible. Perhaps with Unity catalog, as there you have more permission controls.But using Unity is not an overnight decision.

  • 2 kudos
Anonymous
by Not applicable
  • 2059 Views
  • 3 replies
  • 28 kudos

Resolved! Refresh Dashboard also make all related queried refresh?

Hi all,I have a quick currious. I know both query and dashboard page in Databricks SQL have refresh button to can them refresh. But one question it, when I'm in Dashboard page and click the refesh button. Does this thing also force every related quer...

  • 2059 Views
  • 3 replies
  • 28 kudos
Latest Reply
Anonymous
Not applicable
  • 28 kudos

Thanks all your support. It's totally clear for me now!!!

  • 28 kudos
2 More Replies
Pradeep_Namani
by New Contributor III
  • 3849 Views
  • 5 replies
  • 2 kudos

Date field getting changed when reading from excel file to dataframe in pyspark

The date field is getting changed while reading data from source .xls file to the dataframe. In the source xl file all columns are strings but i am not sure why date column alone behaves differentlyIn Source file date is 1/24/1947.In pyspark datafram...

  • 3849 Views
  • 5 replies
  • 2 kudos
Latest Reply
-werners-
Esteemed Contributor III
  • 2 kudos

how about using inferschema one single time to create a correct DF, then create a schema from the df-schema.something like this f.e.from pyspark.sql.types import StructType   # Save schema from the original DataFrame into json: schema_json = df.s...

  • 2 kudos
4 More Replies
JordanYaker
by Contributor
  • 4717 Views
  • 7 replies
  • 8 kudos

Resolved! Is anyone else experiencing intermittent "Failure starting REPL" errors with PySpark Jobs?

I have a Multi-Task Job that is running a bunch of PySpark notebooks and about 30-60% of the time, my jobs fail with the following error:I haven't seen any consistency with this error. I've had as many as all of the tasks in the job giving this error...

image.png
  • 4717 Views
  • 7 replies
  • 8 kudos
Latest Reply
James_Cole
New Contributor III
  • 8 kudos

Hi. Did you ever got a resolution to this problem outside of rolling back to 10.4? I have recently moved some workloads over to runtime 11.3 and am experiencing intermittent "repl did not start in 30 seconds." errors.I have increased the repl timeout...

  • 8 kudos
6 More Replies
Ajay-Pandey
by Esteemed Contributor III
  • 2390 Views
  • 3 replies
  • 15 kudos

Resolved! MongoDB Connection

How can we connect databricks to MongoDB?Any code reference will help

  • 2390 Views
  • 3 replies
  • 15 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 15 kudos

Hi @Ajay Pandey​  (Customer)​ ​, It would mean a lot if you could select the "Best Answer" to help others find the correct answer faster.This makes that answer appear right after the question, so it's easier to find within a thread.It also helps us m...

  • 15 kudos
2 More Replies

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels