Databricks

Upload file to DBFS fails with error code 0

When trying to upload to DBFS from local machine getting error as "Error occurred when processing file ... : Server responded with 0 code" DBR 7.3 LTSSpark 3.0.1 Scala 2.12 Uploading the file using the "upload" in the Databricks cloud console, the c...

Code to automate provisioning of databricks in Azure

Hi, I would like to provision a Databricks environment in Azure and looking at options to create a workspace, cluster, notebook using code. Could you please point me to the documentation around this. Thank you.

What are the advantages of using Delta Live tables over dbt?

Can Databricks Connect work on a cluster with port 443?

I am attempting to use Databricks Connect with a cluster in Azure Government with a port of 443 but get the following error when running databricks-connect test.The port you specified is either being used already or invalid. Port: The port that Data...

EXISTS statement works incorrectly

Hi everybody. Looks like EXISTS statement works incorrectly. If i execute the following statement in SQL Server it returns one row, as it should WITH a AS ( SELECT '1' AS id, 'Super Company' AS name UNION SELECT '2' AS id, 'SUPER COMPANY...

Databricks - autostart from jdbc query

Hi team, New to Databricks and trying to understand if there is a "True" auto-start capability with Databricks. We are evaluating Databricks Delta lake as an alternative cloud based datawarehouse solution but the biggest problem I see is the inabili...

How to set font-size of values in each cell of dataframe?

I am working on pandas and python.After processing a particular dataframe in my program , I am appending that dataframe below an existing Excel file. Now problem is my excel has font size of 11 pt but dataframe has font size of 12 pt.I want to set f...

filter push down into redis when querying using spark connector

HiI'm loading df from redis using this code:df = (spark.read.format("org.apache.spark.sql.redis") .option("table", f"state_store_ready_to_sell") .option("key.column", "msid").option("infer.schema", "true").load()and then i'm running f...

S3 connection reset error :: Removing Spark Config on Cluster

Hi guys, I am running a production pipeline (Databricks Runtime 7.3 LTS) that keeps failing for some delta file reads with the error: 21/07/19 09:56:02 ERROR Executor: Exception in task 36.1 in stage 2.0 (TID 58) com.databricks.sql.io.FileReadExcept...

sparkR.session() from web terminal

Question: sparkR.session() gives an error when run on web terminal, while it runs in a notebook. What parameters should be provided to create a spark session on web terminal? PS: I am trying to run a .R file using Rscript call on terminal instead ...

Access an external table from another workspace

What's the best way to add an external table so another cluster/workspace can access an existing external table on S3? I need to redeploy my workspace into a new VPC, so I am not expecting any collisions of the warehouses. Is it as simple as adding ...

Is there a way to programmatically clear the notebook state in Azure databricks?

I have a scenario where I have a series of jobs that are triggered in ADF, the jobs are not linked as such but the resulting temporally tables from each job takes up memory of the databricks cluster. If I can clear the notebook state, that would fre...

How does FSCK work and does it have any negative effects on subsequent notebook executions?

In my environment, there are 3 groups of notebooks that run on their own schedules, however they all use the same underlying transaction logs (auditlogs, as we call them) in S3. From time to time, various notebooks from each of the 3 groups fail wit...

Load an explicit schema from an external metadata.csv file or a json file for reading csv's into dataframe

Hi, I have a metadata csv file which contains column name, and datatype such as Colm1: INT Colm2: String. I can also get the same in a json format as shown: I can store this on ADLS. How can I convert this into a schema like: "Myschema" that I can ...

Not able to fetch data from Simba Spark Jdbc Driver

We are getting below error when we tried to set the date in preparedstatement using Simba Spark Jdbc Driver. Exception: Query execution failed: [Simba][SparkJDBCDriver](500051) ERROR processing query/statement. Error Code: 0, SQL state: org.apache.h...