- 3513 Views
- 3 replies
- 5 kudos
Resolved! How to convert Data Chr Strings to Date Strings
Databricks CommunityNew to Databricks and work in R code. I have a data from with a date field that is a chr string and need to convert to a date field. Tried the standard as.Date(x, format = "%Y-%m-%d") , then tried the dplyr::mutate function and th...
- 3513 Views
- 3 replies
- 5 kudos
- 5 kudos
Based upon the initial response I went with:my_data.frame <- my_data.frame %>% mutate(date = to_date(data.frame_variable, "yyyy-mm-dd"))
- 5 kudos
- 1558 Views
- 0 replies
- 0 kudos
autoloader schema inference date column
I have hire_date and term_dates in the "MM/dd/YYYY" format in underneath csv files. Schema hint "cloudFiles.schemaHints" : "Hire_Date Date,Term_Date Date" - push data into _rescued_data column due to conversion failure. I am looking out solution to c...
- 1558 Views
- 0 replies
- 0 kudos
- 24729 Views
- 7 replies
- 3 kudos
Resolved! Cannot connect to Databricks SQL Endpoint using PHP and ODBC
I am trying to connect to our Databricks SQL endpoint using PHP in a Docker container.I setup my Docker container to download and configure the ODBC driver as specified here: https://docs.databricks.com/integrations/bi/jdbc-odbc-bi.html#install-and-c...
- 24729 Views
- 7 replies
- 3 kudos
- 3 kudos
The problem was that the Databricks SQL driver does not yet support ARM, which my laptop and Docker container was building for. See ('01000', "[01000] [unixODBC][Driver Manager]Can't open lib '/opt/simba/spark/lib/64/libsparkodbc_sb64.so' : file not ...
- 3 kudos
- 5068 Views
- 4 replies
- 2 kudos
Change Data Feed functionality from SQL Endpoint
I am trying to run command to retrieve change data from sql endpoint. It is throwing below error."The input query contains unsupported data source(s).Only csv, json, avro, delta, parquet, orc, text data sources are supported on Databricks SQL."But th...
- 5068 Views
- 4 replies
- 2 kudos
- 2 kudos
It is separate runtime https://docs.microsoft.com/en-us/azure/databricks/sql/release-notes/#channels
- 2 kudos
- 875 Views
- 0 replies
- 0 kudos
www.scent-sational-waxmelts.co.uk
Ignite your senses with distinctive and Fm World Shop delightful fragrances for your home, Discover scents to set the mood and inspire fragrant memories
- 875 Views
- 0 replies
- 0 kudos
- 10420 Views
- 10 replies
- 1 kudos
Resolved! Azure Databricks container runtime broken in 9.1 LTS, how to fix?
For stability, I've stuck with LTS. Last Friday my containers stopped working with error message:Py4JException: An exception was raised by the Python Proxy. Return Message: Traceback (most recent call last): File "/databricks/spark/python/lib/py4j-...
- 10420 Views
- 10 replies
- 1 kudos
- 1 kudos
This is getting worse. Now JDBC write to SQL is failing for same reason. I haven't yet found a solution for this.Am I not supposed to use containers? Python?This is not cool.
- 1 kudos
- 7127 Views
- 4 replies
- 1 kudos
How to force pandas_on_spark plots to use all dataframe data?
When I load a table as a `pandas_on_spark` dataframe, and try to e.g. scatterplot two columns, what I obtain is a subset of the desired points. For example, if I try to plot two columns from a table with 1000000 rows, I only see some of the data - i...
- 7127 Views
- 4 replies
- 1 kudos
- 1 kudos
@Kaniz Fatma The problem is not about performance or plotly. It is about the pandas_on_spark dataframe arbitrarily subsampling the input data when plotting, without notifying the user about it.While subsampling is comprehensible and maybe even nece...
- 1 kudos
- 12629 Views
- 12 replies
- 4 kudos
Resolved! JSON validation is getting failed after writing Pyspark dataframe to json format
Hi We have to convert transformed dataframe to json format. So we used write and json format on top of final dataframe to convert it to json. But when we validating the output json its not in proper json format.Could you please provide your suggestio...
- 12629 Views
- 12 replies
- 4 kudos
- 4 kudos
@Sailaja B - Does @Aman Sehgal's most recent answer help solve the problem? If it does, would you be happy to mark their answer as best?
- 4 kudos
- 4923 Views
- 2 replies
- 3 kudos
- 4923 Views
- 2 replies
- 3 kudos
- 3 kudos
@Anand Ladda @André Monteiro From comments in the code:Indicates whether the task should be run in a REPL. This value must be true to run on an existing cluster. Please ignore the 'run_as_repl' parameters it will be removed from public docs as it i...
- 3 kudos
- 5561 Views
- 2 replies
- 0 kudos
Where / how does DBFS store files?
I tried to use %fs head to print the contents of a CSV file used in a training%fs head "/mnt/path/file.csv"but got an error saying cannot head a directory!?Then I did %fs ls on the same CSV file and got a list of 4 files under a directory named as a ...
- 5561 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @Al Jo , are you still seeing the error while printing the contents of te CSV file?
- 0 kudos
- 908 Views
- 0 replies
- 0 kudos
www.amritsardigitalacademy.in
Amritsar Digital Academy is the best https://www.amritsardigitalacademy.in/ digital marketing institute In Punjab. if you want to do a digital marketing course. you can enroll now!
- 908 Views
- 0 replies
- 0 kudos
- 10097 Views
- 8 replies
- 5 kudos
Resolved! Unable to start SQL End point in DATABRICKS SQL
Hello All, I am trying to use Databricks SQL but somehow the SQL end point is not getting started. It is in starting state for long time and then session is getting expired. Please note , the default SQL End point also not getting started. I am using...
- 10097 Views
- 8 replies
- 5 kudos
- 5 kudos
@AMZ DUD did you get this working? With a quota of 500, 43 mins is a long time for a cluster to launch. Perhaps a something in the account isn’t set up correctly. Can you please email me your workspace ID please at bilal dot aslam at databricks dot ...
- 5 kudos
- 851 Views
- 0 replies
- 0 kudos
Workshopmanuals.co is proud to offer complete workshop manuals for most vehicle makes and models.
Workshopmanuals.co is proud to offer complete workshop manuals for most vehicle makes and models.
- 851 Views
- 0 replies
- 0 kudos
- 7064 Views
- 6 replies
- 6 kudos
Resolved! Hi Experts I want to know the difference between connecting any BI Tool to Spark SQL and Databricks SQL end point?
Its all about spinning the spark cluster and both spark Sql api and databricks does the same operation what difference does it make to BI tools ?
- 7064 Views
- 6 replies
- 6 kudos
- 6 kudos
Thanks @Bilal Aslam and @Aman Sehgal for jumping in! @Basavaraj Angadi I want to make sure you got your question(s) answered! Will you let us know? Don't forget, you can select any reply as the "best answer" !
- 6 kudos
- 4246 Views
- 4 replies
- 8 kudos
Azure DBR - Have to load list of json files but the column has special character.(ex: {"hydra:xxxx": {"hydra:value":"yyyy", "hydra:value1":"zzzzz"}
Azure DBR - Have to load list of json files into data frame and then from DF to data bricks table but the column has special character and getting below error.Both column(key) and value (as json record) has special characters in the json file. # Can...
- 4246 Views
- 4 replies
- 8 kudos
- 8 kudos
The best is just define schema manually. There is nice article from person who had exactly the same problem https://towardsdev.com/create-a-spark-hive-meta-store-table-using-nested-json-with-invalid-field-names-505f215eb5bf
- 8 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 555 | |
| 349 | |
| 287 |
