Databricks Community

I've created a Lakeflow job to run 5 notebook tasks, one for each silver table- Customers, Accounts, Transactions, Loans and Branches.In Customers notebook, after writing the data to delta table using auto loader, I'm applying the non null and primar...

Hi!I'm trying to create an ETL pipeline. It reads data from a UC volume, however, Databricks is not allowing me to do so. The following error is generated:AnalysisException: [RequestId=a11e017b-61db-4c30-a03a-d7cce55e5aea ErrorClass=INVALID_PARAMETER...

I'm trying to add _metadata column while reading a json file: from pyspark.sql.functions import colfrom pyspark.sql.types import StructType, StructField, LongType, TimestampTypedf_accounts_read = spark.readStream.format("cloudFiles").\    option("clo...

 Hi!I'm facing an error related to Checkpoint whenever I try to display a dataframe using auto Loader in Databricks free edition. Please refer the screenshot. To combat this, I have to delete the checkpoint folder and then execute the display or writ...