cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Engineering & Streaming

Forum Posts

shrikant_kulkar
by New Contributor II
  • 645 Views
  • 0 replies
  • 0 kudos

autoloader schema inference date column

I have hire_date and term_dates in the "MM/dd/YYYY" format in underneath csv files. Schema hint "cloudFiles.schemaHints" : "Hire_Date Date,Term_Date Date" - push data into _rescued_data column due to conversion failure. I am looking out solution to c...

  • 645 Views
  • 0 replies
  • 0 kudos
Rex
by New Contributor III
  • 5197 Views
  • 7 replies
  • 3 kudos

Resolved! Cannot connect to Databricks SQL Endpoint using PHP and ODBC

I am trying to connect to our Databricks SQL endpoint using PHP in a Docker container.I setup my Docker container to download and configure the ODBC driver as specified here: https://docs.databricks.com/integrations/bi/jdbc-odbc-bi.html#install-and-c...

  • 5197 Views
  • 7 replies
  • 3 kudos
Latest Reply
Rex
New Contributor III
  • 3 kudos

The problem was that the Databricks SQL driver does not yet support ARM, which my laptop and Docker container was building for. See ('01000', "[01000] [unixODBC][Driver Manager]Can't open lib '/opt/simba/spark/lib/64/libsparkodbc_sb64.so' : file not ...

  • 3 kudos
6 More Replies
style
by New Contributor
  • 365 Views
  • 0 replies
  • 0 kudos

newstyleinteriors.com

Welcome to <a href="https://newstyleinteriors.com/">New Style Interior</a>. We offer high quality interior design and fit-out solutions for homes, offices, retail, commercial and event spaces.Our team of experienced and passionate designers combine c...

  • 365 Views
  • 0 replies
  • 0 kudos
HQJaTu
by New Contributor III
  • 5124 Views
  • 12 replies
  • 1 kudos

Resolved! Azure Databricks container runtime broken in 9.1 LTS, how to fix?

For stability, I've stuck with LTS. Last Friday my containers stopped working with error message:Py4JException: An exception was raised by the Python Proxy. Return Message: Traceback (most recent call last): File "/databricks/spark/python/lib/py4j-...

  • 5124 Views
  • 12 replies
  • 1 kudos
Latest Reply
HQJaTu
New Contributor III
  • 1 kudos

This is getting worse. Now JDBC write to SQL is failing for same reason. I haven't yet found a solution for this.Am I not supposed to use containers? Python?This is not cool.

  • 1 kudos
11 More Replies
SailajaB
by Valued Contributor III
  • 5585 Views
  • 12 replies
  • 4 kudos

Resolved! JSON validation is getting failed after writing Pyspark dataframe to json format

Hi We have to convert transformed dataframe to json format. So we used write and json format on top of final dataframe to convert it to json. But when we validating the output json its not in proper json format.Could you please provide your suggestio...

  • 5585 Views
  • 12 replies
  • 4 kudos
Latest Reply
Anonymous
Not applicable
  • 4 kudos

@Sailaja B​ - Does @Aman Sehgal​'s most recent answer help solve the problem? If it does, would you be happy to mark their answer as best?

  • 4 kudos
11 More Replies
aladda
by Honored Contributor II
  • 2505 Views
  • 2 replies
  • 3 kudos
  • 2505 Views
  • 2 replies
  • 3 kudos
Latest Reply
User16255483290
Contributor
  • 3 kudos

@Anand Ladda​ @André Monteiro​ From comments in the code:Indicates whether the task should be run in a REPL. This value must be true to run on an existing cluster. Please ignore the 'run_as_repl' parameters it will be removed from public docs as it i...

  • 3 kudos
1 More Replies
al_joe
by Contributor
  • 2546 Views
  • 3 replies
  • 1 kudos

Resolved! Where / how does DBFS store files?

I tried to use %fs head to print the contents of a CSV file used in a training%fs head "/mnt/path/file.csv"but got an error saying cannot head a directory!?Then I did %fs ls on the same CSV file and got a list of 4 files under a directory named as a ...

screenshot image
  • 2546 Views
  • 3 replies
  • 1 kudos
Latest Reply
User16753725182
Contributor III
  • 1 kudos

Hi @Al Jo​ , are you still seeing the error while printing the contents of te CSV file?

  • 1 kudos
2 More Replies
digitalinstitut
by New Contributor
  • 428 Views
  • 0 replies
  • 0 kudos

www.amritsardigitalacademy.in

Amritsar Digital Academy is the best https://www.amritsardigitalacademy.in/ digital marketing institute In Punjab. if you want to do a digital marketing course. you can enroll now!

  • 428 Views
  • 0 replies
  • 0 kudos
Infosys_128139
by New Contributor III
  • 4618 Views
  • 8 replies
  • 5 kudos

Resolved! Unable to start SQL End point in DATABRICKS SQL

Hello All, I am trying to use Databricks SQL but somehow the SQL end point is not getting started. It is in starting state for long time and then session is getting expired. Please note , the default SQL End point also not getting started. I am using...

  • 4618 Views
  • 8 replies
  • 5 kudos
Latest Reply
BilalAslamDbrx
Honored Contributor III
  • 5 kudos

@AMZ DUD​ did you get this working? With a quota of 500, 43 mins is a long time for a cluster to launch. Perhaps a something in the account isn’t set up correctly. Can you please email me your workspace ID please at bilal dot aslam at databricks dot ...

  • 5 kudos
7 More Replies
BasavarajAngadi
by Contributor
  • 4065 Views
  • 6 replies
  • 6 kudos

Resolved! Hi Experts I want to know the difference between connecting any BI Tool to Spark SQL and Databricks SQL end point?

Its all about spinning the spark cluster and both spark Sql api and databricks does the same operation what difference does it make to BI tools ?

  • 4065 Views
  • 6 replies
  • 6 kudos
Latest Reply
Anonymous
Not applicable
  • 6 kudos

Thanks @Bilal Aslam​ and @Aman Sehgal​ for jumping in! @Basavaraj Angadi​ ​ I want to make sure you got your question(s) answered! Will you let us know? Don't forget, you can select any reply as the "best answer" !

  • 6 kudos
5 More Replies
hare
by New Contributor III
  • 2046 Views
  • 4 replies
  • 8 kudos

Azure DBR - Have to load list of json files but the column has special character.(ex: {"hydra:xxxx": {"hydra:value":"yyyy", "hydra:value1":"zzzzz"}

Azure DBR - Have to load list of json files into data frame and then from DF to data bricks table but the column has special character and getting below error.Both column(key) and value (as json record) has special characters in the json file. # Can...

  • 2046 Views
  • 4 replies
  • 8 kudos
Latest Reply
Hubert-Dudek
Esteemed Contributor III
  • 8 kudos

The best is just define schema manually. There is nice article from person who had exactly the same problem https://towardsdev.com/create-a-spark-hive-meta-store-table-using-nested-json-with-invalid-field-names-505f215eb5bf

  • 8 kudos
3 More Replies
alejandrofm
by Valued Contributor
  • 826 Views
  • 1 replies
  • 2 kudos

Resolved! Feature request for spark performance tuning

Hi, I don't think there's a place to see this, please correct me if I'm wrong.Now to see performance tuning tips I have to go to spark UI, then to SQL view and on top I could see performance alerts that help me know If I need apply a spark config, co...

  • 826 Views
  • 1 replies
  • 2 kudos
Latest Reply
-werners-
Esteemed Contributor III
  • 2 kudos

I think that can be requested at ideas.databricks.com

  • 2 kudos
LukaszJ
by Contributor III
  • 8206 Views
  • 5 replies
  • 0 kudos

Resolved! Send UPDATE from Databricks to Azure SQL DataBase

Hello.I want to know how to do an UPDATE on Azure SQL DataBase from Azure Databricks using PySpark.I know how to make query as SELECT and turn it into DataFrame, but how to send back some data (as UPDATE on rows)?I want to use build in pyspark istead...

  • 8206 Views
  • 5 replies
  • 0 kudos
Latest Reply
-werners-
Esteemed Contributor III
  • 0 kudos

This is discussed on Stack Overflow. As you see for Azure Synapse there is a way, but for a plain SQL database you will have to use some kind of driver like odbc/jdbc.

  • 0 kudos
4 More Replies
Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!

Labels