Databricks Community

Hi @shubham_007 , Thanks for the question. You can install libraries with Serverless Compute using an "Environment". This also provides the option to configure an environment specification into a YAML file. More detail can be found here: https://docs...

Hi @MAHANK,If the Repos are not an option for you at the moment, you can use Visual Studio Code to compare 2 files. It's pretty straightforward. All you have to do after installing VS Code is to download both files to your local environment, open the...

We have the same error. It does not seem to be related to the Current or Preview runtimes in the DLT settings. region: Azure East US 2For us, the pipeline completes successfully but the corresponding connection to the underlying data referenced from ...

@TomBrick Greetings! Are you trying to establish a connection from a VM on Azure? Can you add below logging and add the log file here. On MacOS and Linux, the odbc.ini file will need to be used to set the logging parameters at the DSN level. Linux: /...

Seems like a duplicate: https://community.databricks.com/t5/data-engineering/urgent-need-information-details-and-reference-link-on-below-two/td-p/107260

Have you tried setting it up in your code as: from pyspark.sql import SparkSession # Create a Spark session spark = SparkSession.builder \ .appName("YourAppName") \ .config("spark.files.ignoreCorruptFiles", "true") \ .getOrCreate() # Yo...

To handle the scenario where your pipeline fails after loading some records into the first gold table or if one gold table loads successfully while the second fails, you can implement a failure handling mechanism that ensures already inserted records...

Hello folks,I also opened a ticket with Databricks Academy, and it seems that Partner Learning doesn't include the code anymore. You need a Databricks labs subscription.Quote: "Are you referring to the labs that are not available?If so, We are sorry ...

Getting Same issue with Runtime 16.1.

I don't get any output when running the following, I have the destination set to dbfs . But it was only supposed to be for cluster logs and not for query execution outputs to be stored in DBFS. Any idea if this is expected behavior.spark.conf.get("sp...

Let me rephrase it. You can't use Message as the rowTag, because it's the root element. rowTag implies that it's a tag within the root element, which might occur multiple times. Check the docs around reading and write XML files, there you'll find exa...