Databricks Community

Hello, I am running a job that requires reading in files of different sizes, each one representing a different dataset, and loading them into a delta table. Some files are as big as 100Gib and others as small as 500 MiB. I want to repartition each fi...

Hello,I have non-incremental data landing in a storage account. This data contains old data from before as well as new data. I would like to avoid doing a complete table deletion and table creation just to upload the data from storage and have an upd...

Hello,Everyday a new file of the same name gets sent to my storage account with old and new data appended at the end.  Columns may also be added during one of these file updates. This file does a complete overwrite of the previous file. Is it possibl...

Hello, Currently I have files landing in a storage account. They are all located in subfolders of a common directory. Some subdirectories may contain files, others may not. Each file name is unique and corresponds to a unique table as well. No two fi...