- 22820 Views
- 16 replies
- 10 kudos
Resolved! Error in databricks-sql-connector
from databricks import sql hostname = '<name>.databricks.com' http_path = '/sql/1.0/endpoints/<endpoint_id>' access_token = '<personal_token>' connection = sql.connect(server_hostname=hostname, http_path=http_path, access_token=access_token) cu...
- 22820 Views
- 16 replies
- 10 kudos
- 10 kudos
Hi @Borislav Blagoev​ ,Thanks very much for taking the time to collect these logs.The problem here (as indicated by the `IpAclValidation` message) is that IP allow listing (enabled for your workspace) will not allow arbitrary connections from Spark c...
- 10 kudos
- 12553 Views
- 3 replies
- 1 kudos
Resolved! Delta Table is not available in the Databricks SQL
Hi, there!I'm trying to read a data (simple SELECT * FROM schema.tabl_a) from the "Queries" Tab inside the Databricks SQL platform, but always getting "org.apache.spark.sql.AnalysisException: dbfs:/.../.. doesn't exist" DescribeRelation true, [col_na...
- 12553 Views
- 3 replies
- 1 kudos
- 7634 Views
- 7 replies
- 3 kudos
Resolved! Issue with quotes in struct type columns when using ODBC
I'm trying to connect to Databricks using pyodbc and I'm running into an issue with struct columns. As far as I understand, struct columns and array columns are not supported by pyodbc, but they are converted to JSON. However, when there are nested c...
- 7634 Views
- 7 replies
- 3 kudos
- 3 kudos
@Derk Crezee​ - I learned something today. Apparently ODBC does not convert to JSON. There is no defined spec on how to return complex types, in fact that was added only in SQL 2016. That's exactly what you are running into!End of history lesson Her...
- 3 kudos
- 18626 Views
- 4 replies
- 5 kudos
Resolved! parquet file to include partitioned column in file
HI,I have a daily scheduled job which processes the data and write as parquet file in a specific folder structure like root_folder/{CountryCode}/parquetfiles. Where each day job will write new data for countrycode under the folder for countrycodeI am...
- 18626 Views
- 4 replies
- 5 kudos
- 5 kudos
Most external consumers will read partition as column when are properly configured (for example Azure Data Factory or Power BI).Only way around is that you will duplicate column with other name (you can not have the same name as it will generate conf...
- 5 kudos
- 3774 Views
- 2 replies
- 1 kudos
Resolved! Cannot Get Databricks SQL to read external Hive Metastore
I have followed the documentation and using the same metastore config that is working in the Data Engineering context. When attempting to view the Databases, I get the error:Encountered an internal errorThe following information failed to load:The li...
- 3774 Views
- 2 replies
- 1 kudos
- 1 kudos
@Bilal Aslam​ I didn't think to look there before since I hadn't tried to run any queries. I see the failed SHOW DATABASES queries in history and they identify the error: Builtin jars can only be used when hive execution version == hive metastore v...
- 1 kudos
- 4938 Views
- 5 replies
- 2 kudos
Resolved! Hello Experts - I am facing one technical issue with Databricks SQL - IF-ELSE or CASE statement implementation when trying to execute two separate set of queries based on a valued of a column of the Delta table.
Hi Experts,I'm reading a pipe delimited source file where first row does not contain the data but contains the - REPLACE or UPDATE values which indicate the refresh type if it is Full refresh Or Upsert. The second row is nothing but header and the a...
- 4938 Views
- 5 replies
- 2 kudos
- 2 kudos
Thanks a ton Gurpreet, Suggested approach was helpful in resolution of the issue. Appreciate your help !!
- 2 kudos
- 6733 Views
- 4 replies
- 3 kudos
what is the difference between weekday and dayofweek function in spark SQL?
dayofweek: https://docs.databricks.com/sql/language-manual/functions/dayofweek.htmlweekday : https://docs.databricks.com/sql/language-manual/functions/weekday.htmlAccording to the documentation , they both are synonym functions. But when I use it I n...
- 6733 Views
- 4 replies
- 3 kudos
- 3 kudos
That's correct for weekday moday=0 for dayofweek Sunday=1.​​You can also look for documentation here https://spark.apache.org/docs/latest/api/python/reference/api/pyspark.sql.functions.dayofweek.html​https://spark.apache.org/docs/latest/api/sql/index...
- 3 kudos
- 4923 Views
- 3 replies
- 2 kudos
%SQL Append null values into a SQL Table
Hi All, I am new to Databricks and am writing my first program.Note: Code Shown Below:I am creating a table with 3 columns to store data. 2 of the columns will be appended in from data that I have in another table.When I run my append query into the...
- 4923 Views
- 3 replies
- 2 kudos
- 2 kudos
Hi Hubert,Your answer moves me closer to being able to update pieces of a 26 field MMR_Restated table in pieces are the correct fields values are calculated Thru the process. I have been looking for a way to be able to update in "pieces"...... 2 fie...
- 2 kudos
- 7034 Views
- 5 replies
- 12 kudos
The database and tables disappears when I delete the cluster
Hi guys,I have a trial databricks account, I realized that when I shutdown the cluster my databases and tables is disappear .. that is correct or thats is because my account is trial ?
- 7034 Views
- 5 replies
- 12 kudos
- 12 kudos
@William Scardua​ if it's an external hive metastore or Glue catalog you might be missing the configuration on the cluster. https://docs.databricks.com/data/metastores/index.htmlAlso as mentioned by @Hubert Dudek​ , if it's a community edition then t...
- 12 kudos
- 5787 Views
- 2 replies
- 10 kudos
Resolved! When trying to use pyodbc connector to write files to SQL server receiving error. java.lang.ClassNotFoundException Any alternatives or ways to fix this?
jdbcUsername = ******** jdbcPassword = *************** server_name = "jdbc:sqlserver://***********:******" database_name = "********" url = server_name + ";" + "databaseName=" + database_name + ";" table_name = "PatientTEST" try: df.write \ ...
- 5787 Views
- 2 replies
- 10 kudos
- 10 kudos
please check following code:df.write.jdbc( url="jdbc:sqlserver://<host>:1433;database=<db>;user=<user>;password=<password>;encrypt=true;trustServerCertificate=false;hostNameInCertificate=*.database.windows.net;loginTimeout=30;driver=com.microsof...
- 10 kudos
- 4876 Views
- 4 replies
- 3 kudos
read percentage values in spark ( no casting )
I have a xlsx file which has a single column ;percentage30%40%50%-10%0.00%0%0.10%110%99.99%99.98%-99.99%-99.98%when i read this using Apache-Spark out put i get is,|percentage|+----------+| 0.3|| 0.4|| 0.5|| -0.1|| 0.0|| ...
- 4876 Views
- 4 replies
- 3 kudos
- 3 kudos
Affirmative. This is how excel stores percentages. What you see is just cell formatting.Databricks notebooks do not (yet?) have the possibility to format the output.But it is easy to use a BI tool on top of Databricks, where you can change the for...
- 3 kudos
- 5834 Views
- 9 replies
- 4 kudos
Resolved! Configure Databricks (spark) context from PyCharm
Hello. I'm trying to connect to Databricks from my IDE (PyCharm) and then run delta table queries from there. However, the cluster I'm trying to access has to give me permission. In this case, I'd go to my cluster, run the cell which gives me permiss...
- 5834 Views
- 9 replies
- 4 kudos
- 4 kudos
"I'm trying to connect to Databricks from my IDE (PyCharm) and then run delta table queries from there."If you are going to deploy later your code to databricks the only solutions which I see is to use databricks-connect or just make development envi...
- 4 kudos
- 5924 Views
- 4 replies
- 1 kudos
- 5924 Views
- 4 replies
- 1 kudos
- 1 kudos
Hi @Vamsee krishna kanth Arcot​ Yes, currently you will have to download the JDBC from https://databricks.com/spark/jdbc-drivers-download and connect from other applications with JDBC URL just like you mentioned in your example. There is an internal ...
- 1 kudos
- 2856 Views
- 4 replies
- 3 kudos
- 2856 Views
- 4 replies
- 3 kudos
- 3 kudos
You may have noticed that the local SQL endpoint is not listed in the options for getting started with APEX. The local SQL endpoint is an extremely useful feature for getting ADO.NET web services started. I say check this uk-dissertation.com review f...
- 3 kudos
- 4182 Views
- 5 replies
- 11 kudos
Resolved! Runtime SQL Configuration - how to make it simple
Hi, I'm running couple of Notebooks in my pipeline and I would like to set fixed value of 'spark.sql.shuffle.partitions' - same value for every notebook. Should I do that by adding spark.conf.set.. code in each Notebook (Runtime SQL configurations ar...
- 4182 Views
- 5 replies
- 11 kudos
- 11 kudos
Hi, Thank you all for the tips. I tried before to set this option in Spark Config but didn't work for some reason. Today I tried again and it's working :).
- 11 kudos