cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
vijaykumarbotla
by • New Contributor III
  • 4556 Views
  • 5 replies
  • 1 kudos

Resolved! Getting error : Analysis Exception : olumn Is There a PO#17748 are ambiguous. It's probably because you joined several Datasets together, and some of these Datasets are the same. This column points to one of the Datasets but Spark.

AnalysisException: Column Is There a PO#17748 are ambiguous. It's probably because you joined several Datasets together, and some of these Datasets are the same. This column points to one of the Datasets but Spark is unable to figure out which one. ...

Data Engineering
Reply
  • 4556 Views
  • 5 replies
  • 1 kudos
View Replies
Latest Reply
vijaykumarbotla
New Contributor III
  • 1 kudos

Hi All,the solution for this problem is very strange.this has caused due to the version of the Databricks runtime.We are using Runtime version 7.0 with Apache Spark 3.0.0 version.In PRD we are using Runtime version 11.3LTS with Apache Spark 3.3.0 ver...

  • 1 kudos
4 More Replies
StephanieAlba
by • Databricks Employee
  • 10678 Views
  • 4 replies
  • 2 kudos

Resolved! How do I download and unzip datasets from Kaggle into DBFS?

Data Engineering
Reply
  • 10678 Views
  • 4 replies
  • 2 kudos
View Replies
Latest Reply
Debayan
Databricks Employee
  • 2 kudos

Hi, You can refer to https://docs.databricks.com/files/unzip-files.html. You can curl the file you want and then it can be unzipped as mentioned in the doc. Please let us know if this helps.Also, please tag @Debayan with your next update which will n...

  • 2 kudos
3 More Replies
quakenbush
by • Contributor
  • 4195 Views
  • 3 replies
  • 4 kudos

Resolved! Does Databricks offer something like Oracle's dblink?

I am aware, I can load anything into a DataFrame using JDBC, that works well from Oracle sources. Is there an equivalent in Spark SQL, so I can combine datasets as well?Basically something like so - you get the idea...select lt.field1, rt.fie...

Data Engineering
Reply
  • 4195 Views
  • 3 replies
  • 4 kudos
View Replies
Latest Reply
quakenbush
Contributor
  • 4 kudos

Thanks everyone for helping.

  • 4 kudos
2 More Replies
Kavin
by • New Contributor II
  • 1839 Views
  • 1 replies
  • 2 kudos

Issue converting the datasets into JSON

Im a newbie to Databricks, I need to convert the data sets into JSON. i tried bth FOR JSON AUTO AND FOR JSON PATH, However im getting an issue - [PARSE_SYNTAX_ERROR] Syntax error at or near 'json'line My Query works fine without FOR JSON AUTO AND FOR...

Data Engineering
Reply
  • 1839 Views
  • 1 replies
  • 2 kudos
View Replies
Latest Reply
Debayan
Databricks Employee
  • 2 kudos

Hi @Kavin Natarajan​ , Could you please go through https://www.tutorialkart.com/apache-spark/spark-write-dataset-to-json-file-example/ , looks like the steps are okay.

  • 2 kudos
Geeya
by • New Contributor II
  • 1680 Views
  • 1 replies
  • 0 kudos

After several iteration of filter and union, the data is bigger than spark.driver.maxResultSize

The process for me to build model is:filter dataset and split into two datasetsfit model based on two datasets union two datasetsrepeat 1-3 stepsThe problem is that after several iterations, the model fitting time becomes longer dramatically, and the...

Data Engineering
Reply
  • 1680 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Ryan_Chynoweth
Esteemed Contributor
  • 0 kudos

I assume that you are using PySpark to train a model? It sounds like you are collecting data on the driver and likely need to increase the size. Can you share any code?

  • 0 kudos
Labels