Databricks

BenLambert · 11-30-2023

I have a dataframe with a schema similar to the following:id: stringarray_field: array element: struct field1: string field2: string array_field2: array element: struct nested_field: stri...

BenLambert · 05-17-2023

Within the UI it is possible to "Select tables for refresh" for a specific Delta Live Tables Workflow. I often use it to make a full refresh on smaller tables during development. Unfortunately, when an error occurs during the full refresh on selected...

BenLambert · 04-28-2023

We have a DLT pipeline that uses the autoloader to detect files added to a source storage bucket. It reads these updated files and adds new records to a bronze streaming table. However we would also like to automatically delete records from the bronz...

BenLambert · 04-28-2023

We are using Delta Live Tables for running ingestion pipelines and have come across the two options for the autoloader "file notification" vs "directory listing" this is reflected in the option cloudFiles.useIncrementalListing. We are wondering what ...

BenLambert · 09-06-2022

I have a delta live tables pipeline that is loading and transforming data. Currently I am having a problem that the schema inferred by DLT does not match the actual schema of the table. The table is generated via a groupby.pivot operation as follows:...

BenLambert · 12-04-2023

It turns out that if the exploded fields don't match the schema that was defined when reading the JSON in the first place that all the data that doesn't match is silently dropped. This is not really nice default behaviour.

BenLambert · 11-30-2023

@Kaniz additionally I am getting a message that says something to the effect of:"This query is on a non-Delta table with many small files. To improve the performance of queries, convert to Delta and run the OPTIMIZE command on the table dbfs:/mnt/buc...

BenLambert · 11-30-2023

Just to make it clear, I have about 19,000 ids and when I explode I only get 4 rows that correspond to the first id.

BenLambert · 11-30-2023

@Kaniz thanks for the tip. This is the approach I am trying to take, but the issue is that exploding the outer array fails. It only returns the 4 values associated with the first id and not all possible rows.

BenLambert · 06-14-2023

In this case we just need to cause an error of any type where the pipeline doesn't execute and then in the UI it no longer shows an option to refresh a specific table, leaving only a full refresh as an option. In this case we removed the specificatio...

Databricks

User Stats

User Activity

Explode is giving unexpected results.

Table Refresh UI Error

How to deal with deleted files in source directory in DLT?

When should you use the directory listing vs file notification

Delta Live Tables not inferring table schema properly.

Re: Explode is giving unexpected results.

Re: Explode is giving unexpected results.

Re: Explode is giving unexpected results.

Re: Explode is giving unexpected results.

Re: Table Refresh UI Error