cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Mayank
by New Contributor III
  • 14476 Views
  • 8 replies
  • 4 kudos

Resolved! Unable to load Parquet file using Autoloader. Can someone help?

I am trying to load parquet files using Autoloader. Below is the code def autoload_to_table (data_source, source_format, table_name, checkpoint_path): query = (spark.readStream .format('cloudFiles') .option('cl...

  • 14476 Views
  • 8 replies
  • 4 kudos
Latest Reply
Anonymous
Not applicable
  • 4 kudos

Hi again @Mayank Srivastava​ Thank you so much for getting back to us and marking the answer as best.We really appreciate your time.Wish you a great Databricks journey ahead!

  • 4 kudos
7 More Replies
ekhool
by New Contributor
  • 1028 Views
  • 0 replies
  • 0 kudos

 E-khool is an Online Learning Management System software for Industries with low cost and no extra charges. One stop solution for live classes, trai...

E-khool is an Online Learning Management System software for Industries with low cost and no extra charges. One stop solution for live classes, trainings, videos & materials, unlimited exams, corporate training & professional development, online int...

  • 1028 Views
  • 0 replies
  • 0 kudos
Phani1
by Databricks MVP
  • 2074 Views
  • 1 replies
  • 1 kudos

DeltaSharing

Hi, Databricks team,I created the share and provided access to the recipient, and the recipient is consuming or accessing the share. My question is who is going to bear the cost and how can we track the computation charges for it?As per the attached ...

  • 2074 Views
  • 1 replies
  • 1 kudos
Latest Reply
Phani1
Databricks MVP
  • 1 kudos

Hi @Kaniz Fatma​ , Thanks for your response on Delta sharing information, Could you please provide details about who is going to bear the cost[either provider/Consumer] and how can we track the computation charges for it?

  • 1 kudos
EveryDayData
by Contributor
  • 21860 Views
  • 23 replies
  • 7 kudos

Resolved! Data AI Summit Training

Hi Team,I am asking a little off topic question here. I have been attending paid session for Data AI summit in last 2-3 years. Is it possible to get videos for sessions which we have not attended due to session conflicts at the same time. I mean to s...

  • 21860 Views
  • 23 replies
  • 7 kudos
Latest Reply
RajeshRK
Contributor II
  • 7 kudos

@Juliet Wu​ Hi Juliet,Thank you so much for sharing the recording. Regards,Rajesh.

  • 7 kudos
22 More Replies
Pete_M
by New Contributor II
  • 3785 Views
  • 1 replies
  • 2 kudos

Is it possible to alter the text in a Databricks Job Alert Email?

I have configured a Databricks job to send email alters to me whenever my job fails. However, I would very much like to alter the text in the Alert Email to something a little more bespoke. Is there any way to alter the text in email or even just t...

  • 3785 Views
  • 1 replies
  • 2 kudos
Latest Reply
Anonymous
Not applicable
  • 2 kudos

Hey there @Peter Mayers​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from yo...

  • 2 kudos
Mohit_m
by Databricks Employee
  • 15243 Views
  • 1 replies
  • 3 kudos

Resolved! How to Install Python packages from the own artifactory

We have created our own artifactory and we use this to install python dependencies or libraries.We would like to know how we can make use of our own artifactory to install dependencies or libraries on Databricks clusters..

  • 15243 Views
  • 1 replies
  • 3 kudos
Latest Reply
Mohit_m
Databricks Employee
  • 3 kudos

For private repos, you can find some good examples herehttps://kb.databricks.com/clusters/install-private-pypi-repo.htmlhttps://towardsdatascience.com/install-custom-python-libraries-from-private-pypi-on-databricks-6a7669f6e6fd

  • 3 kudos
Anonymous
by Not applicable
  • 3098 Views
  • 4 replies
  • 5 kudos

Hello Databricks Community!  We are getting so excited about the upcoming event of the year Data & AI Summit!  If you still need to sign up, visit...

Hello Databricks Community! We are getting so excited about the upcoming event of the year Data & AI Summit! If you still need to sign up, visit our registration page [link]! We have an in-person and virtual option for attending this year. In prepara...

  • 3098 Views
  • 4 replies
  • 5 kudos
Latest Reply
lakshmidvm
New Contributor II
  • 5 kudos

me too, didn't receive giftcard.

  • 5 kudos
3 More Replies
MxSasch
by New Contributor II
  • 12302 Views
  • 11 replies
  • 5 kudos

Cluster terminated.Reason:Unexpected launch failure

Is there any known issues affecting the creation of clusters? I've been unable to get any clusters to start today so far!Have received this error "Cluster terminated.Reason:Unexpected launch failure"Help!

  • 12302 Views
  • 11 replies
  • 5 kudos
Latest Reply
AWe
New Contributor II
  • 5 kudos

good

  • 5 kudos
10 More Replies
drewster
by New Contributor III
  • 18966 Views
  • 13 replies
  • 13 kudos

Resolved! Spark streaming autoloader slow second batch - checkpoint issues?

I am running a massive history of about 250gb ~6mil phone call transcriptions (json read in as raw text) from a raw -> bronze pipeline in Azure Databricks using pyspark. The source is mounted storage and is continuously having files added and we do n...

  • 18966 Views
  • 13 replies
  • 13 kudos
Latest Reply
Brooksjit
New Contributor III
  • 13 kudos

Thank you for the explanation.

  • 13 kudos
12 More Replies
sarveshpandey23
by New Contributor II
  • 3846 Views
  • 4 replies
  • 3 kudos

There is some issue in Column

I have loaded the csv file from my local PC to data bricks. Now, when i am trying to query the table over column its throwing error. Note - select * from table_name is working but select column_name from table_name is throwing error.

image
  • 3846 Views
  • 4 replies
  • 3 kudos
Latest Reply
Anonymous
Not applicable
  • 3 kudos

Thanks for the information, I will try to figure it out for more. Keep sharing such informative post keep suggesting such post.

  • 3 kudos
3 More Replies
Scorpius
by New Contributor II
  • 2727 Views
  • 1 replies
  • 0 kudos

Unregistering / Removing A SparkListener

Hey, So currently I'm using Pyspark and utilizing SparkListener to track metrics of my spark jobs. The problem I can't seem to solve is why in databricks the listener cannot be removed with removeSparkListener as it still is attached to the context.#...

  • 2727 Views
  • 1 replies
  • 0 kudos
Raymond_Garcia
by Contributor II
  • 3868 Views
  • 4 replies
  • 2 kudos

Migrating from Databricks Notebooks to IDE for Development

Hello, we are developers who have been creating a system in Databricks with Scala. We enabled the Git feature, so the project is in a repository. The project has a lot of notebooks and a lot of calls to other notebooks. Sometimes it is a little overw...

  • 3868 Views
  • 4 replies
  • 2 kudos
Latest Reply
Raymond_Garcia
Contributor II
  • 2 kudos

it is true that we can't work without data bricks but we can develop an IDE and send the jar to databricks, this will allow us to create unit tests, and use the IDE capabilities (i.e fast navigation among classes).

  • 2 kudos
3 More Replies
Phani1
by Databricks MVP
  • 6875 Views
  • 3 replies
  • 2 kudos

Resolved! is it possible to have multiple tabs in Dashboard? if not is there any workaround for this.

is it possible to have multiple tabs in Dashboard? if not is there any workaround for this.

  • 6875 Views
  • 3 replies
  • 2 kudos
Latest Reply
Anonymous
Not applicable
  • 2 kudos

Hey @Janga Reddy​ Hope you are well. Just wanted to see if you were able to find an answer to your question and would you like to mark an answer as best? It would be really helpful for the other members too.Cheers!

  • 2 kudos
2 More Replies
Labels