Databricks

Ovi · 05-19-2023

Hello everyone, I have an intermittent issue when trying to create a Delta table for the first time in Databricks: all the data gets converted into parquet at the specified location but the _delta_log is not created or, if created, it's left empty, t...

Ovi · 03-14-2023

Hello everyone,From a list of folders on s3, how can I filter which ones are Delta tables, without trying to read each one at a time?Thanks,Ovi

Ovi · 10-18-2022

Hi all! I have an S3 bucket with Delta parquet files/folders with different schemas each. I need to create an RDD or DataFrame from all those Delta Tables that should contain the path, name and different schema of each.How could I do that?Thank you!P...

Ovi · 03-14-2023

Nice solution @Nandini N !Thanks a lot for pointing it out to me!Have a nice day,Ovi

Ovi · 03-14-2023

Hi @Nandini N,Thank you for your answer which includes also an example.But for my use case, I need to do it using Scala. Could you please point me in the direct direction of how to implement this?Thank you again,Ovi

Ovi · 10-19-2022

Thank you @Debayan Mukherjee but I think I was misunderstood. Let me give you more details:I need to compare several Delta tables with different schema each with their analogue avro schemasI've managed to build a dataframe with the avro schemas usin...

Databricks

User Stats

User Activity

Spark Dataframe write to Delta format doesn't create a _delta_log

Filter only Delta tables from an S3 folders list

Construct Dataframe or RDD from S3 bucket with Delta tables

Re: Filter only Delta tables from an S3 folders list

Re: Filter only Delta tables from an S3 folders list

Re: Construct Dataframe or RDD from S3 bucket with Delta tables