cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

PareDesa_10157
by New Contributor II
  • 970 Views
  • 1 replies
  • 1 kudos

My cluster shows time to time this message, what is the permanent solution

MessageCluster terminated. Reason: Spark Image Download FailureHelpFailed to set up spark container due to an image download failure: Exception when downloading spark image: Stdout: /usr/local/bin/fastar /usr/local/bin/fastarIhave tried enabling the ...

  • 970 Views
  • 1 replies
  • 1 kudos
Latest Reply
Debayan
Databricks Employee
  • 1 kudos

Hi, This error can be surfaced due to various reasons, such as networking , slowness etc.. Is there any other significant errors showing up in the driver logs? Is this for a single cluster? Is this a new workspace? You can start with checking on the ...

  • 1 kudos
duliu
by New Contributor II
  • 8978 Views
  • 6 replies
  • 0 kudos

databricks-connect fails with java.lang.IllegalStateException: No api token found in local properties

I configured databricks-connect locally and want to run spark code against a remote cluster.I verified `databricks-connect test` passes and it can connect to remote databricks cluster.However when I query a tables or read parquet from s3, it fails wi...

  • 8978 Views
  • 6 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

@Du Liu​ :The error message suggests that there is no API token found in the local properties. This could be the cause of the failure when trying to access the tables or read parquet files from S3.To fix this issue, you need to ensure that the API to...

  • 0 kudos
5 More Replies
Sid0610
by New Contributor II
  • 7553 Views
  • 3 replies
  • 3 kudos

Resolved! Databricks SQL CREATE TABLE ParseException

I am trying to use the following code to create a deltatable%sqlCREATE TABLE rectangles(a INT, b INT, area INT GENERATED ALWAYS AS IDENTITY (START WITH 1, STEP BY 1))I don't know why but I am always getting the ParseException error.I tried all other ...

  • 7553 Views
  • 3 replies
  • 3 kudos
Latest Reply
emiratesevisaon
New Contributor II
  • 3 kudos

How can we use SQL for my website emiratesevisaonline.com backend date?

  • 3 kudos
2 More Replies
Johny
by New Contributor III
  • 6739 Views
  • 2 replies
  • 4 kudos

Insert data to a CDF-enabled Delta table throwing java.lang.StackOverflowError

I am building a bronze table with CDF-enables in these steps:Initially, Reading json file from landing zone and write to table locationdf = spark.readStream.format("cloudFiles") \ .option("cloudFiles.schemaLocation", <schema_loc>) \ .option("clou...

  • 6739 Views
  • 2 replies
  • 4 kudos
Latest Reply
Johny
New Contributor III
  • 4 kudos

I tried with a simple csv file that only has one column. I got the same error.

  • 4 kudos
1 More Replies
varunsaagar
by New Contributor III
  • 9984 Views
  • 17 replies
  • 28 kudos

Request for reattempt voucher. Databricks Certified Machine Learning Professional exam

HiOn December 28th ,I attempted the Databricks Certified Machine Learning Professional exam for 1st time , unfortunately I ended up by failing grade. My passing grade was 70%, and I received 68.33%.I am planning to reattempt the exam, Could you kindl...

  • 9984 Views
  • 17 replies
  • 28 kudos
Latest Reply
girl_chan
New Contributor II
  • 28 kudos

What is the next event where they will give a voucher?

  • 28 kudos
16 More Replies
JKR
by Contributor
  • 2688 Views
  • 2 replies
  • 0 kudos

The spark driver has stopped unexpectedly and is restarting. Your notebook will be automatically reattached.

Getting below error Context: Using Databricks shared interactive cluster for scheduled run multiple parallel jobs at the same time after every 5 mins. When I check Ganglia, driver node's memory reaches almost max and then restart of driver happens an...

  • 2688 Views
  • 2 replies
  • 0 kudos
Latest Reply
jose_gonzalez
Databricks Employee
  • 0 kudos

please check the driver's logs, for example the log4j and the GC logs

  • 0 kudos
1 More Replies
jonasmin
by New Contributor III
  • 9057 Views
  • 7 replies
  • 2 kudos

Error while establishing JDBC connection to Azure databricks via HTTP proxy

I am using the databricks JDBC driver (https://databricks.com/spark/jdbc-drivers-download) to connect to Azure databricks.The connection needs to be routed through a HTTP proxy. I found parameters that can be configured for using the HTTP proxy:By pa...

databricks jdbc
  • 9057 Views
  • 7 replies
  • 2 kudos
Latest Reply
MS_Varma
New Contributor II
  • 2 kudos

Hi @Jonas Minning​ , actually I am also having the same issue and when i looked into the driver related documentation I found that the driver currently only supports SOCKS proxies and I believe this is the reason why we are getting this error. So, I ...

  • 2 kudos
6 More Replies
knowAsha
by New Contributor II
  • 3523 Views
  • 3 replies
  • 3 kudos

Error while running the data engineering course notebook : "DE 2.2 - Providing Options for External Sources"

 Can somebody help me fixing this problem? I am running this notebook on databricks community edition

I am running this notebook in Databricks community edition.
  • 3523 Views
  • 3 replies
  • 3 kudos
Latest Reply
lemfo
New Contributor II
  • 3 kudos

df = spark.read.format('parquet').load(path = datasource_path) df = df.select("*").toPandas() df.to_sql('users', conn, if_exists='replace', index = False)

  • 3 kudos
2 More Replies
Hubert-Dudek
by Esteemed Contributor III
  • 3000 Views
  • 2 replies
  • 8 kudos

Implementing a data vault model in databricks can be challenging, but it can significantly improve the manageability of your data, particularly in hea...

Implementing a data vault model in databricks can be challenging, but it can significantly improve the manageability of your data, particularly in heavily regulated industries such as banking. While it may involve significant data duplication, duplic...

ezgif-4-58a39917b2
  • 3000 Views
  • 2 replies
  • 8 kudos
Latest Reply
Priyag1
Honored Contributor II
  • 8 kudos

helpful

  • 8 kudos
1 More Replies
Priyag1
by Honored Contributor II
  • 1605 Views
  • 1 replies
  • 11 kudos

How to get reward points ?

How to get reward points ?

  • 1605 Views
  • 1 replies
  • 11 kudos
Latest Reply
samhita
New Contributor III
  • 11 kudos

You must share informative posts and be active in the community.Thank you

  • 11 kudos
pinaki1
by New Contributor III
  • 4444 Views
  • 5 replies
  • 0 kudos

connect rds from databricks sql editor

Is it possible to connect and execute query directly from rds in sql editor without using unity catelog

  • 4444 Views
  • 5 replies
  • 0 kudos
Latest Reply
luis_herrera
Databricks Employee
  • 0 kudos

Hi there, Yes, you could do federated queries from DB SQL Editor. This is an experimental feature, though. UC is actually not supported. You can read more here:https://docs.databricks.com/query-federation/index.htmlPS: check out #DAIS2023 talks

  • 0 kudos
4 More Replies
81528
by New Contributor II
  • 2077 Views
  • 2 replies
  • 3 kudos

The workers in the cluster uses old end-of-life Ubuntu 18.04.

I create a cluster or a pool with the runtime version 12.2 LTS or even with the latest 13.0According to the documentation the worker should use an image. with Ubuntu 20.04 https://docs.databricks.com/release-notes/runtime/12.2.html#system-environment...

ubuntu_ip-10-20-25-228___
  • 2077 Views
  • 2 replies
  • 3 kudos
Latest Reply
Priyag1
Honored Contributor II
  • 3 kudos

Contact support team

  • 3 kudos
1 More Replies
DavyN
by New Contributor II
  • 3593 Views
  • 3 replies
  • 3 kudos

Resolved! Unable to take Lakehouse Fundamentals Quiz

Hi,I watched the videos for Lakehouse Fundamentals. However, when I click on "Take the quiz" it opens another tab that says I don't have permission to access the page.I've done all the necessary signing up.Could someone please help. Thanks!

  • 3593 Views
  • 3 replies
  • 3 kudos
Latest Reply
MandatoryNickna
New Contributor II
  • 3 kudos

This still seems to be unavailable. Very annoying.

  • 3 kudos
2 More Replies
Oliver_Angelil
by Valued Contributor II
  • 1877 Views
  • 2 replies
  • 2 kudos

Automated CI code checks using workflows when PR is raised

I'm familiar with Github Actions workflows to automate code checks whenever a PR is raised to a specified branch. For example for Python code, very useful is if unit tests (e.g. pytest), syntax (flake8), and code formatting (black formatter), type h...

  • 1877 Views
  • 2 replies
  • 2 kudos
Latest Reply
Priyag1
Honored Contributor II
  • 2 kudos

In a typical software development workflow (e.g. Github flow), a feature branch is created based on the master branch for feature development. A notebook can be synced to the feature branch via Github integration. Or a notebook can be exported from D...

  • 2 kudos
1 More Replies

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels