Databricks

SM · 02-24-2022

Hello, I am working with Delta Live Tables, I am trying to create a DLT from a combination of Dataframes from a 'for loop' which are unioned and then DLT is created over the Unioned Dataframe. However I noticed that the delta table has duplciates. An...

SM · 01-20-2022

I am trying write data into Azure Datalake. I am reading files from Azure Blob Storage however when I try to create the Delta Live Table to Azure Datalake I get error the following errorshaded.databricks.azurebfs.org.apache.hadoop.fs.azurebfs.contrac...

SM · 02-28-2022

I use 'toLocalIterator' to iterate over a dataframe to make a dictionary item which is then for looped. For each key value in the Dictionary, a dataframe is returned after processing. That dataframe is appended to a list of dataframes. Using Reduce I...

SM · 02-27-2022

def map_explode(df): metric = create_map(list(chain(*( (lit(name), col(name).cast(StringType())) for name in df.columns if name not in ['timestamp', 'path'] )))) df = df.withColumn('mapCol', metric) result = df.select('timestamp', 'path',...

Databricks

User Stats

User Activity

Delta Live Tables has duplicates created by multiple workers

How to use Azure Data lake as a storage location to store the Delta Live Tables?

Re: Delta Live Tables has duplicates created by multiple workers

Re: Delta Live Tables has duplicates created by multiple workers