cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Engineering & Streaming

Forum Posts

Priyag1
by Honored Contributor II
  • 532 Views
  • 0 replies
  • 9 kudos

New unified Databricks navigation - New Release to Public preview check it out : Databricks release a new navigation experience to public preview.  Th...

New unified Databricks navigation - New Release to Public preview check it out :Databricks release a new navigation experience to public preview. The goal is to reduce clicks and context switches required to complete tasks. The new experience include...

  • 532 Views
  • 0 replies
  • 9 kudos
Priyag1
by Honored Contributor II
  • 516 Views
  • 0 replies
  • 9 kudos

Databricks MarketplaceDatabricks Marketplace, an open forum for exchanging data products. Databricks Marketplace takes advantage of Delta Sharing to g...

Databricks MarketplaceDatabricks Marketplace, an open forum for exchanging data products. Databricks Marketplace takes advantage of Delta Sharing to give data providers the tools to share data products securely and data consumers the power to explore...

  • 516 Views
  • 0 replies
  • 9 kudos
PareDesa_10157
by New Contributor II
  • 671 Views
  • 1 replies
  • 1 kudos

My cluster shows time to time this message, what is the permanent solution

MessageCluster terminated. Reason: Spark Image Download FailureHelpFailed to set up spark container due to an image download failure: Exception when downloading spark image: Stdout: /usr/local/bin/fastar /usr/local/bin/fastarIhave tried enabling the ...

  • 671 Views
  • 1 replies
  • 1 kudos
Latest Reply
Debayan
Esteemed Contributor III
  • 1 kudos

Hi, This error can be surfaced due to various reasons, such as networking , slowness etc.. Is there any other significant errors showing up in the driver logs? Is this for a single cluster? Is this a new workspace? You can start with checking on the ...

  • 1 kudos
duliu
by New Contributor II
  • 3946 Views
  • 6 replies
  • 0 kudos

databricks-connect fails with java.lang.IllegalStateException: No api token found in local properties

I configured databricks-connect locally and want to run spark code against a remote cluster.I verified `databricks-connect test` passes and it can connect to remote databricks cluster.However when I query a tables or read parquet from s3, it fails wi...

  • 3946 Views
  • 6 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

@Du Liu​ :The error message suggests that there is no API token found in the local properties. This could be the cause of the failure when trying to access the tables or read parquet files from S3.To fix this issue, you need to ensure that the API to...

  • 0 kudos
5 More Replies
Sid0610
by New Contributor II
  • 1957 Views
  • 3 replies
  • 3 kudos

Resolved! Databricks SQL CREATE TABLE ParseException

I am trying to use the following code to create a deltatable%sqlCREATE TABLE rectangles(a INT, b INT, area INT GENERATED ALWAYS AS IDENTITY (START WITH 1, STEP BY 1))I don't know why but I am always getting the ParseException error.I tried all other ...

  • 1957 Views
  • 3 replies
  • 3 kudos
Latest Reply
emiratesevisaon
New Contributor II
  • 3 kudos

How can we use SQL for my website emiratesevisaonline.com backend date?

  • 3 kudos
2 More Replies
Johny
by New Contributor III
  • 3146 Views
  • 2 replies
  • 4 kudos

Insert data to a CDF-enabled Delta table throwing java.lang.StackOverflowError

I am building a bronze table with CDF-enables in these steps:Initially, Reading json file from landing zone and write to table locationdf = spark.readStream.format("cloudFiles") \ .option("cloudFiles.schemaLocation", <schema_loc>) \ .option("clou...

  • 3146 Views
  • 2 replies
  • 4 kudos
Latest Reply
Johny
New Contributor III
  • 4 kudos

I tried with a simple csv file that only has one column. I got the same error.

  • 4 kudos
1 More Replies
varunsaagar
by New Contributor III
  • 6661 Views
  • 18 replies
  • 31 kudos

Request for reattempt voucher. Databricks Certified Machine Learning Professional exam

HiOn December 28th ,I attempted the Databricks Certified Machine Learning Professional exam for 1st time , unfortunately I ended up by failing grade. My passing grade was 70%, and I received 68.33%.I am planning to reattempt the exam, Could you kindl...

  • 6661 Views
  • 18 replies
  • 31 kudos
Latest Reply
girl_chan
New Contributor II
  • 31 kudos

What is the next event where they will give a voucher?

  • 31 kudos
17 More Replies
JKR
by Contributor
  • 1934 Views
  • 2 replies
  • 0 kudos

The spark driver has stopped unexpectedly and is restarting. Your notebook will be automatically reattached.

Getting below error Context: Using Databricks shared interactive cluster for scheduled run multiple parallel jobs at the same time after every 5 mins. When I check Ganglia, driver node's memory reaches almost max and then restart of driver happens an...

  • 1934 Views
  • 2 replies
  • 0 kudos
Latest Reply
jose_gonzalez
Moderator
  • 0 kudos

please check the driver's logs, for example the log4j and the GC logs

  • 0 kudos
1 More Replies
jonasmin
by New Contributor III
  • 6526 Views
  • 7 replies
  • 2 kudos

Error while establishing JDBC connection to Azure databricks via HTTP proxy

I am using the databricks JDBC driver (https://databricks.com/spark/jdbc-drivers-download) to connect to Azure databricks.The connection needs to be routed through a HTTP proxy. I found parameters that can be configured for using the HTTP proxy:By pa...

databricks jdbc
  • 6526 Views
  • 7 replies
  • 2 kudos
Latest Reply
MS_Varma
New Contributor II
  • 2 kudos

Hi @Jonas Minning​ , actually I am also having the same issue and when i looked into the driver related documentation I found that the driver currently only supports SOCKS proxies and I believe this is the reason why we are getting this error. So, I ...

  • 2 kudos
6 More Replies
knowAsha
by New Contributor II
  • 2220 Views
  • 3 replies
  • 3 kudos

Error while running the data engineering course notebook : "DE 2.2 - Providing Options for External Sources"

 Can somebody help me fixing this problem? I am running this notebook on databricks community edition

I am running this notebook in Databricks community edition.
  • 2220 Views
  • 3 replies
  • 3 kudos
Latest Reply
lemfo
New Contributor II
  • 3 kudos

df = spark.read.format('parquet').load(path = datasource_path) df = df.select("*").toPandas() df.to_sql('users', conn, if_exists='replace', index = False)

  • 3 kudos
2 More Replies
Hubert-Dudek
by Esteemed Contributor III
  • 1777 Views
  • 2 replies
  • 8 kudos

Implementing a data vault model in databricks can be challenging, but it can significantly improve the manageability of your data, particularly in hea...

Implementing a data vault model in databricks can be challenging, but it can significantly improve the manageability of your data, particularly in heavily regulated industries such as banking. While it may involve significant data duplication, duplic...

ezgif-4-58a39917b2
  • 1777 Views
  • 2 replies
  • 8 kudos
Latest Reply
Priyag1
Honored Contributor II
  • 8 kudos

helpful

  • 8 kudos
1 More Replies
Priyag1
by Honored Contributor II
  • 863 Views
  • 1 replies
  • 11 kudos

How to get reward points ?

How to get reward points ?

  • 863 Views
  • 1 replies
  • 11 kudos
Latest Reply
samhita
New Contributor III
  • 11 kudos

You must share informative posts and be active in the community.Thank you

  • 11 kudos
kjoth
by Contributor II
  • 13567 Views
  • 9 replies
  • 7 kudos

How to make the job fail via code after handling exception

Hi , We are capturing the exception if an error occurs using try except. But we want the job status to be failed once we got the exception. Whats the best way to do that. We are using pyspark.

  • 13567 Views
  • 9 replies
  • 7 kudos
Latest Reply
AkA
New Contributor II
  • 7 kudos

Instead of exiting the notebook which make the task/job success, Exception objects needs to be raised again from Exception block to fail the job.try: <you code>except Exception as err: <your block of exception handling> raise err

  • 7 kudos
8 More Replies
pinaki1
by New Contributor III
  • 3015 Views
  • 5 replies
  • 0 kudos

connect rds from databricks sql editor

Is it possible to connect and execute query directly from rds in sql editor without using unity catelog

  • 3015 Views
  • 5 replies
  • 0 kudos
Latest Reply
luis_herrera
New Contributor III
  • 0 kudos

Hi there, Yes, you could do federated queries from DB SQL Editor. This is an experimental feature, though. UC is actually not supported. You can read more here:https://docs.databricks.com/query-federation/index.htmlPS: check out #DAIS2023 talks

  • 0 kudos
4 More Replies
Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!

Labels