Databricks Community

Resolved! Run stored bash in Databricks with %sh

Hi, I made bash file in databricks and I can see that the file is stored as the following picture. I was supposed to run this bash file through %sh cell, but as you see the following picture, I could not find bash file, which I could find through d...

Resolved! Databricks-Connect shows different partitions than Databricks for the same delta table

Hello,here is a small code-snippet:from pyspark.sql import SparkSession spark = SparkSession.builder.appName('example_app').getOrCreate()   spark.sql('SHOW PARTITIONS database.table').show() The output inside the Databricks-Notebook:+-------------+--...

Resolved! [Simba][SparkJDBCDriver](500177) Error getting http path from connection string

I'm trying to use a very basic java program to connect to Databricks using spark jdbc driver (SparkJDBC42.jar), but I get the error (mentioned above): [Simba][SparkJDBCDriver](500177) Error getting http path from connection stringHere is my code snip...

Resolved! Error writing parquet files

Hi, we are having this chain of errors every day in different files and processes:An error occurred while calling o11255.parquet.: org.apache.spark.SparkException: Job aborted.Caused by: org.apache.spark.SparkException: Job aborted due to stage failu...

Resolved! Is it possible to write tables to delta lake using upsert mode? Would it be more efficiant than overwrite?

Resolved! Delete the file

While exporting data from Databricks to Azure blob storage how can I delete the committed, started and success file? ​

Resolved! Is is possible to Mount multiple ADLS Gen2 Storage paths in single workspace

Hello Experts,We are looking on feasibility of mounting more that one ADLS Gen2 storages on a single workspace of databricks.Best RegardsPraveen

Resolved! Cluster does not start (Coursera Training Distributed Computing with Spark SQL)

Following the instruction on Week 1 > The Databricks Environment, it is supposed to create a new cluster. However, the cluster is not starting or able to attached notebook and due that I can not continue the tasks/assignments. related documents not...

Resolved! AutoML : data set for problem type "Classification"

HI,I am working on AutoML Experiment. Could you plz help me with data set for problem type "Classification"Regards.

Resolved! list users having access to scope credentials

Hello!How do I list all the users or groups having access to the key-vault backed scope credentials?Let's say, I have a scope called MyScope for which all the secrets are stored in MyKeyVault.I would like to see what users have access there and ideal...

Resolved! What's the best architecture for Structured Streaming and why?

I am building an ETL pipeline which reads data from a Kafka topic ( data is serialized in Thrift format) and writes it to Delta Table in databricks. I want to have two layersBronze Layer -> which has raw Kafka dataSilver Layer -> which has deserializ...

Resolved! How temporary is dbfs:/tmp/? Are files periodically deleted from there?

Resolved! How to pass variables into query string?

I have two variables StartTimeStmp and EndTimeStmp, i am going to assign the Start timestamp to it based on Last Successful Job Runtime and EndTimeStamp would be current time of system.SET StartTimeStmp = '2022-03-24 15:40:00.000';SET EndTimeStmp = '...

Resolved! How to create a drop-down widget in Databricks notebook with choices retrieved using subquery over a table in SQL, python?