Databricks Community

AanchalSoni

I'm trying to add _metadata column while reading a json file: from pyspark.sql.functions import colfrom pyspark.sql.types import StructType, StructField, LongType, TimestampTypedf_accounts_read = spark.readStream.format("cloudFiles").\ option("clo...

AanchalSoni

Hi!I'm facing an error related to Checkpoint whenever I try to display a dataframe using auto Loader in Databricks free edition. Please refer the screenshot. To combat this, I have to delete the checkpoint folder and then execute the display or writ...

AanchalSoni · 09-30-2025

Hi!I'm trying to stream some files using read_files.format("cloudFiles"). However, when new files arrive, the subsequent SQL query and monitoring graphs are not getting updated. Please suggest.

AanchalSoni · 09-18-2025

Using Community EditionI'm trying to create a pipeline(streaming) using auto loader (accessing external location) and each time my select query is thrown this error- " An error occurred while calling t.analyzeAndFormatResult. : java.lang.UnsupportedO...

AanchalSoni · 09-12-2025

Hi,I'm trying to create a pipeline using Azure, however, Azure is not getting listed in the drop down of Catalog Explorer -> Create External Location. I'm using community version for practice. Please advice.

AanchalSoni

Thanks Ashwin! And yes your explanation about Checkpoints was clear. I could comprehend the relevance of Checkpoints.

AanchalSoni

Hi Ashwin!Thanks for your response. Before the screenshot step, I'm just reading the file with explicit schema. When you say 'The recommendation is not to reuse a checkpoint path across different query shapes or output modes. Give each logical stream...

AanchalSoni

Thanks Ashwin! This seems to have worked.

AanchalSoni · 09-30-2025

Hi Saurabh!If I'm not explicitly mentioning the trigger then by default the trigger should be 500 ms and there should be a quick check for new files. however, even after a few minutes there is no expected activity.

AanchalSoni · 09-30-2025

df_super = ( spark.readStream.format("cloudFiles") .option("cloudFiles.format", "csv") .option("cloudFiles.validateOptions", "false") .option( "cloudFiles.schemaHints", "ROW_ID int, Order_Date date, Ship_Date date, Postal_Co...

Databricks Community

User Stats

User Activity

NULL rows getting inserted in delta table- Schema mismatch

Checkpoint Location Error

Streaming- Results not getting updated on arrival of new files

Unable to use Auto Loader for External Location in Community Edition

Azure not getting listed in create external location

Re: Checkpoint Location Error

Re: Checkpoint Location Error

Re: NULL rows getting inserted in delta table- Schema mismatch

Re: Streaming- Results not getting updated on arrival of new files

Re: Streaming- Results not getting updated on arrival of new files