- 24003 Views
- 2 replies
- 7 kudos
Resolved! SQL CONCAT returning null
Has anyone else experienced this problem? I'm attempting to SQL concat two fields and if the second field is null, the entire string appears as null. The documentation is unclear on the expected outcome, and contrary to how concat_ws operates.SELECT ...
- 24003 Views
- 2 replies
- 7 kudos
- 7 kudos
CONCAT is a function defined in the SQL standard and available across a wide variety of DBMS. With the exception of Oracle which uses VARCHAR2 semantic across the board, the function returns NULL on NULL input.CONCAT_WS() is not standard and is mostl...
- 7 kudos
- 2305 Views
- 3 replies
- 4 kudos
SparkJDBC42.jar Issue ?
Hi there!I am using the SparkJDBC42.jar in my Java application to use my delta lake tables , The connection is made through databricks sql endpoint in where I created a database and store in it my delta tables. I have a simple code to open connection...
- 2305 Views
- 3 replies
- 4 kudos
- 4 kudos
Hi @Seifeddine SNOUSSI ,Are you still having issue or you were able to resolve this issue? please let us know
- 4 kudos
- 1963 Views
- 3 replies
- 5 kudos
Resolved! How to convert Data Chr Strings to Date Strings
Databricks CommunityNew to Databricks and work in R code. I have a data from with a date field that is a chr string and need to convert to a date field. Tried the standard as.Date(x, format = "%Y-%m-%d") , then tried the dplyr::mutate function and th...
- 1963 Views
- 3 replies
- 5 kudos
- 5 kudos
Based upon the initial response I went with:my_data.frame <- my_data.frame %>% mutate(date = to_date(data.frame_variable, "yyyy-mm-dd"))
- 5 kudos
- 7110 Views
- 7 replies
- 3 kudos
Resolved! Cannot connect to Databricks SQL Endpoint using PHP and ODBC
I am trying to connect to our Databricks SQL endpoint using PHP in a Docker container.I setup my Docker container to download and configure the ODBC driver as specified here: https://docs.databricks.com/integrations/bi/jdbc-odbc-bi.html#install-and-c...
- 7110 Views
- 7 replies
- 3 kudos
- 3 kudos
The problem was that the Databricks SQL driver does not yet support ARM, which my laptop and Docker container was building for. See ('01000', "[01000] [unixODBC][Driver Manager]Can't open lib '/opt/simba/spark/lib/64/libsparkodbc_sb64.so' : file not ...
- 3 kudos
- 6583 Views
- 8 replies
- 5 kudos
Resolved! Unable to start SQL End point in DATABRICKS SQL
Hello All, I am trying to use Databricks SQL but somehow the SQL end point is not getting started. It is in starting state for long time and then session is getting expired. Please note , the default SQL End point also not getting started. I am using...
- 6583 Views
- 8 replies
- 5 kudos
- 5 kudos
@AMZ DUD did you get this working? With a quota of 500, 43 mins is a long time for a cluster to launch. Perhaps a something in the account isn’t set up correctly. Can you please email me your workspace ID please at bilal dot aslam at databricks dot ...
- 5 kudos
- 5160 Views
- 6 replies
- 6 kudos
Resolved! Hi Experts I want to know the difference between connecting any BI Tool to Spark SQL and Databricks SQL end point?
Its all about spinning the spark cluster and both spark Sql api and databricks does the same operation what difference does it make to BI tools ?
- 5160 Views
- 6 replies
- 6 kudos
- 6 kudos
Thanks @Bilal Aslam and @Aman Sehgal for jumping in! @Basavaraj Angadi I want to make sure you got your question(s) answered! Will you let us know? Don't forget, you can select any reply as the "best answer" !
- 6 kudos
- 10994 Views
- 4 replies
- 0 kudos
Resolved! Send UPDATE from Databricks to Azure SQL DataBase
Hello.I want to know how to do an UPDATE on Azure SQL DataBase from Azure Databricks using PySpark.I know how to make query as SELECT and turn it into DataFrame, but how to send back some data (as UPDATE on rows)?I want to use build in pyspark istead...
- 10994 Views
- 4 replies
- 0 kudos
- 0 kudos
This is discussed on Stack Overflow. As you see for Azure Synapse there is a way, but for a plain SQL database you will have to use some kind of driver like odbc/jdbc.
- 0 kudos
- 5298 Views
- 3 replies
- 4 kudos
Resolved! Reading multiple parquet files from same _delta_log under a path
I have a path where there is _delta_log and 3 snappy.parquet files. I am trying to read all those .parquet using spark.read.format('delta').load(path) but I am getting data from only one same file all the time. Can't I read from all these files? If s...
- 5298 Views
- 3 replies
- 4 kudos
- 4 kudos
@Werner Stinckens Thanks for the reply and explanation, that was helpful to understand the delta feature.
- 4 kudos
- 2988 Views
- 4 replies
- 5 kudos
Resolved! Databricks: Report on SQL queries that are being executed
We have a SQL workspace with a cluster running that services a number of self service reports against a range of datasets. We want to be able to analyse and report on the queries our self service users are executing so we can get better visibility of...
- 2988 Views
- 4 replies
- 5 kudos
- 5 kudos
Looks like the people have spoken: API is your best option! (thanks @Werner Stinckens @Chris Grabiel and @Bilal Aslam !) @eni chante Let us know if you have questions about the API! If not, please mark one of the replies above as the "best answ...
- 5 kudos
- 2892 Views
- 1 replies
- 2 kudos
Resolved! Convert Week of Year to Month in SQL?
Hi all, Was wondering if there was any built in function or code that I could utilize to convert a singular week of year integer (i.e. 1 to 52), into a value representing month (i.e. 1-12)? The assumption is that a week start on a Monday and end on a...
- 2892 Views
- 1 replies
- 2 kudos
- 2 kudos
we need old parser as new doesn't support weeks. Than we can map what we need using w - year of year and u - first day of the week:spark.sql("set spark.sql.legacy.timeParserPolicy=LEGACY") spark.sql(""" SELECT extract( month from to_date...
- 2 kudos
- 4817 Views
- 9 replies
- 0 kudos
Resolved! Query AWS Redshift from Databricks SQL
Greetings,In the documentation for Databricks SQL it states that it supports JDBC connections, however when connecting to AWS Redshift via the built in PostgreSQL driver ("CREATE TABLE sample USING JDBC" and "jdbc://postgresql:/..." URI) I'm getting ...
- 4817 Views
- 9 replies
- 0 kudos
- 0 kudos
@Bilal Aslam anytime! Is there a place where customers could follow the timeline when such features are introduced?
- 0 kudos
- 4077 Views
- 6 replies
- 4 kudos
Resolved! SQL Analytics: Is it possible to configure the default number format
HiWhen querying an integer value, the default format is '0.0' which results in an integer value 202111, displayed as 202,111. I can resolve the issue by updating the visualisation or using formatnumber in my query, however I would like to set a defa...
- 4077 Views
- 6 replies
- 4 kudos
- 4 kudos
@Bilal Haniff - Would you be happy to mark whichever answer helped you the most as best? That helps others find solutions more quickly.
- 4 kudos
- 6203 Views
- 3 replies
- 0 kudos
Resolved! Help trying to calculate a percentage
The image below shows what my source data is (HAVE) and what I'm trying to get to (WANT).I want to be able to calculate the percentage of bad messages (where formattedMessage = false) by source and date.I'm not sure how to achieve this in DatabricksS...
- 6203 Views
- 3 replies
- 0 kudos
- 0 kudos
you could use a windows function over source and date with a sum of messagecount. This gives you the total per source/date repeated on every line.Then apply a filter on formattedmessage == false and divide messagecount by the sum above.
- 0 kudos
- 567 Views
- 0 replies
- 0 kudos
data=[['x', 20220118, 'FALSE', 3],['x', 20220118, 'TRUE', 97],['x', 20220119, 'FALSE', 1],['x'...
data=[['x', 20220118, 'FALSE', 3],['x', 20220118, 'TRUE', 97],['x', 20220119, 'FALSE', 1],['x', 20220119, 'TRUE', 49],['Y', 20220118, 'FALSE', 100],['Y', 20220118, 'TRUE', 900],['Y', 20220119, 'FALSE', 200],['Y', 20220119, 'TRUE', 800]]df=spark.creat...
- 567 Views
- 0 replies
- 0 kudos
- 2244 Views
- 3 replies
- 2 kudos
Resolved! Query RestAPI end point in Databricks Standard Workspace
Do we have option to query delta table using Standard Workspace as a endpoint instead of JDBC
- 2244 Views
- 3 replies
- 2 kudos
- 2 kudos
@somanath Sankaran - Would you be happy to mark @Hubert Dudek's answer as best if it resolved the problem? That helps other members who are searching for answers find the solution more quickly.
- 2 kudos