Databricks Community

kulkpd · 11-17-2023

I am using DLT with filenotification and DLT job is just fetching 1 notification from SQS queue at a time. My pipeline is expected to process 500K notifications per day but it running hours behind. Any recommendations?spark.readStream.format("cloudFi...

kulkpd · 02-17-2024

@zero234 ,Adding some suggestion based on answers from @Retired_mod. Important point to note here: "To define a materialized view in Python, apply @table to a query that performs a static read against a data source. To define a streaming table, apply...

kulkpd · 12-14-2023

Can you please paste the exact errors and check below things:check following if its related to KMS:1. IAM role policy and KMS policy should have allow permissions2. Did you use extraConfig while mounting the source-s3 bucket:If you have used IAM role...

kulkpd · 12-14-2023

below option used while performing spark.readStream:::.option('cloudFiles.format', 'json').option('cloudFiles.inferColumnTypes', 'true').option('cloudFiles.schemaEvolutionMode', 'rescue').option('cloudFiles.useNotifications', True).option('skipChange...

kulkpd · 12-14-2023

My bad, somewhere in the screenshot I saw that but not able to find it now.Which source you are using to load the data, delta table, aws-s3, or azure-storage?

kulkpd · 12-13-2023

@esalohs Unfortunately even you specify includeExistingFiles as false still its going to do full scan. Probably some bug at Databricks side @Retired_mod ? My use-case was similar that process only latest files and not exisiting 150million files but n...

Databricks Community

User Stats

User Activity

Autoloader with filenotification

Re: i have created a materialized view table using delta live table pipeline and its not appending d

Re: Databricks-Autoloader-S3-KMS

Re: Databricks Autoloader - list only new files in an s3 bucket/directory

Re: Filtering files for query

Re: Databricks Autoloader - list only new files in an s3 bucket/directory