cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Engineering & Streaming

Forum Posts

Start a conversation
Jeff1
by • Contributor II
  • 1831 Views
  • 3 replies
  • 5 kudos

Resolved! How to convert Data Chr Strings to Date Strings

Databricks CommunityNew to Databricks and work in R code. I have a data from with a date field that is a chr string and need to convert to a date field. Tried the standard as.Date(x, format = "%Y-%m-%d") , then tried the dplyr::mutate function and th...

Data Engineering
Reply
  • 1831 Views
  • 3 replies
  • 5 kudos
View Replies
Latest Reply
Jeff1
Contributor II
  • 5 kudos

Based upon the initial response I went with:my_data.frame <- my_data.frame %>% mutate(date = to_date(data.frame_variable, "yyyy-mm-dd"))

  • 5 kudos
2 More Replies
shrikant_kulkar
by • New Contributor III
  • 879 Views
  • 0 replies
  • 0 kudos

autoloader schema inference date column

I have hire_date and term_dates in the "MM/dd/YYYY" format in underneath csv files. Schema hint "cloudFiles.schemaHints" : "Hire_Date Date,Term_Date Date" - push data into _rescued_data column due to conversion failure. I am looking out solution to c...

Data Engineering
Reply
  • 879 Views
  • 0 replies
  • 0 kudos
Rex
by • New Contributor III
  • 6764 Views
  • 7 replies
  • 3 kudos

Resolved! Cannot connect to Databricks SQL Endpoint using PHP and ODBC

I am trying to connect to our Databricks SQL endpoint using PHP in a Docker container.I setup my Docker container to download and configure the ODBC driver as specified here: https://docs.databricks.com/integrations/bi/jdbc-odbc-bi.html#install-and-c...

Data Engineering
Reply
  • 6764 Views
  • 7 replies
  • 3 kudos
View Replies
Latest Reply
Rex
New Contributor III
  • 3 kudos

The problem was that the Databricks SQL driver does not yet support ARM, which my laptop and Docker container was building for. See ('01000', "[01000] [unixODBC][Driver Manager]Can't open lib '/opt/simba/spark/lib/64/libsparkodbc_sb64.so' : file not ...

  • 3 kudos
6 More Replies
Suman
by • New Contributor III
  • 3091 Views
  • 4 replies
  • 2 kudos

Change Data Feed functionality from SQL Endpoint

I am trying to run command to retrieve change data from sql endpoint. It is throwing below error."The input query contains unsupported data source(s).Only csv, json, avro, delta, parquet, orc, text data sources are supported on Databricks SQL."But th...

Data Engineering
Reply
  • 3091 Views
  • 4 replies
  • 2 kudos
View Replies
Latest Reply
Hubert-Dudek
Esteemed Contributor III
  • 2 kudos

It is separate runtime https://docs.microsoft.com/en-us/azure/databricks/sql/release-notes/#channels

  • 2 kudos
3 More Replies
style
by • New Contributor
  • 505 Views
  • 0 replies
  • 0 kudos

newstyleinteriors.com

Welcome to <a href="https://newstyleinteriors.com/">New Style Interior</a>. We offer high quality interior design and fit-out solutions for homes, offices, retail, commercial and event spaces.Our team of experienced and passionate designers combine c...

Data Engineering
Reply
  • 505 Views
  • 0 replies
  • 0 kudos
HQJaTu
by • New Contributor III
  • 6960 Views
  • 10 replies
  • 1 kudos

Resolved! Azure Databricks container runtime broken in 9.1 LTS, how to fix?

For stability, I've stuck with LTS. Last Friday my containers stopped working with error message:Py4JException: An exception was raised by the Python Proxy. Return Message: Traceback (most recent call last): File "/databricks/spark/python/lib/py4j-...

Data Engineering
Reply
  • 6960 Views
  • 10 replies
  • 1 kudos
View Replies
Latest Reply
HQJaTu
New Contributor III
  • 1 kudos

This is getting worse. Now JDBC write to SQL is failing for same reason. I haven't yet found a solution for this.Am I not supposed to use containers? Python?This is not cool.

  • 1 kudos
9 More Replies
DavideCagnoni
by • Contributor
  • 4765 Views
  • 4 replies
  • 1 kudos

How to force pandas_on_spark plots to use all dataframe data?

When I load a table as a `pandas_on_spark` dataframe, and try to e.g. scatterplot two columns, what I obtain is a subset of the desired points. For example, if I try to plot two columns from a table with 1000000 rows, I only see some of the data - i...

Data Engineering
Reply
  • 4765 Views
  • 4 replies
  • 1 kudos
View Replies
Latest Reply
DavideCagnoni
Contributor
  • 1 kudos

@Kaniz Fatma​  The problem is not about performance or plotly. It is about the pandas_on_spark dataframe arbitrarily subsampling the input data when plotting, without notifying the user about it.While subsampling is comprehensible and maybe even nece...

  • 1 kudos
3 More Replies
SailajaB
by • Valued Contributor III
  • 7811 Views
  • 12 replies
  • 4 kudos

Resolved! JSON validation is getting failed after writing Pyspark dataframe to json format

Hi We have to convert transformed dataframe to json format. So we used write and json format on top of final dataframe to convert it to json. But when we validating the output json its not in proper json format.Could you please provide your suggestio...

Data Engineering
Reply
  • 7811 Views
  • 12 replies
  • 4 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 4 kudos

@Sailaja B​ - Does @Aman Sehgal​'s most recent answer help solve the problem? If it does, would you be happy to mark their answer as best?

  • 4 kudos
11 More Replies
aladda
by • Databricks Employee
  • 2988 Views
  • 2 replies
  • 3 kudos

Resolved! What does the run_as_repl parameter do in the databricks Jobs API - https://docs.databricks.com/dev-tools/api/latest/examples.html#jobs-api-examples

Data Engineering
Reply
  • 2988 Views
  • 2 replies
  • 3 kudos
View Replies
Latest Reply
User16255483290
Contributor
  • 3 kudos

@Anand Ladda​ @André Monteiro​ From comments in the code:Indicates whether the task should be run in a REPL. This value must be true to run on an existing cluster. Please ignore the 'run_as_repl' parameters it will be removed from public docs as it i...

  • 3 kudos
1 More Replies
al_joe
by • Contributor
  • 3767 Views
  • 2 replies
  • 0 kudos

Where / how does DBFS store files?

I tried to use %fs head to print the contents of a CSV file used in a training%fs head "/mnt/path/file.csv"but got an error saying cannot head a directory!?Then I did %fs ls on the same CSV file and got a list of 4 files under a directory named as a ...

screenshot image
Data Engineering
Reply
  • 3767 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
User16753725182
Databricks Employee
  • 0 kudos

Hi @Al Jo​ , are you still seeing the error while printing the contents of te CSV file?

  • 0 kudos
1 More Replies
digitalinstitut
by • New Contributor
  • 555 Views
  • 0 replies
  • 0 kudos

www.amritsardigitalacademy.in

Amritsar Digital Academy is the best https://www.amritsardigitalacademy.in/ digital marketing institute In Punjab. if you want to do a digital marketing course. you can enroll now!

Data Engineering
Reply
  • 555 Views
  • 0 replies
  • 0 kudos
Infosys_128139
by • New Contributor III
  • 6222 Views
  • 8 replies
  • 5 kudos

Resolved! Unable to start SQL End point in DATABRICKS SQL

Hello All, I am trying to use Databricks SQL but somehow the SQL end point is not getting started. It is in starting state for long time and then session is getting expired. Please note , the default SQL End point also not getting started. I am using...

Data Engineering
Reply
  • 6222 Views
  • 8 replies
  • 5 kudos
View Replies
Latest Reply
BilalAslamDbrx
Databricks Employee
  • 5 kudos

@AMZ DUD​ did you get this working? With a quota of 500, 43 mins is a long time for a cluster to launch. Perhaps a something in the account isn’t set up correctly. Can you please email me your workspace ID please at bilal dot aslam at databricks dot ...

  • 5 kudos
7 More Replies

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels