Databricks Community

scvbelle · 08-30-2023

In my DLT pipeline outlined below which generically cleans identifier tables, after successfully creating initial streaming tables from the append-only sources, fails when trying to create the second cleaned tables witht the following:It'**bleep** cl...

scvbelle · 07-12-2023

I have multiple transactional sources feeding into my Azure Databricks env (MySQL, MSSQL, MySQL datadumps) for a client company-wide DataLake. They are basically all "managed sources" (using different DBMSs, receiving application dumps, etc), but I d...

scvbelle · 06-05-2023

The client receives data from a third party as weekly "datadumps" of a MySQL database copied into an Azure Blob Storage account container (I suspect this is done manually, I also suspect the changes between the approx 7GB files are very small). I nee...

scvbelle · 09-04-2023

Hi @Priyanka_Biswas Thanks so much for responding!I dove into the metastore and saw that because of my partitioning (?), directories are created using actual values from the table. On checking the source data, I see that indeed one of the values ends...

scvbelle · 08-30-2023

UPDATE: See the code below (for some reason couldn't post it yesterday.)I decided to revert back to an earlier version of the DLT pipeline which defined each table one by one rather than programatically, which had run without issue. That ran fine.I a...

scvbelle · 07-19-2023

This is the implementaiton of the original pos, for interest (excluded from the original post to reduce length) # some blacklisting and schema renamings where necessary (due to conflicts and/or illegal characters) SourceName=str SchemaName=str TableN...

scvbelle · 07-19-2023

I made the "mistake" of talking to my software engineer partner who majored in pure math about the problem.They agreed that 1) bringing both tables into memory is not a good idea, especially as then the diff tables will then also be held in memory2) ...

scvbelle · 07-07-2023

Hi!Thanks so much for this response - sorry I'm so delayed in responding.I ended up spinning up a dedicated Azure flexible MySQL serverrather than trying to set one up on the cluster machines (too many conflicting dependencies). This worked fine, but...

Databricks Community

User Stats

User Activity

DLT failure: ABFS does not allow files or directories to end with a dot

Optimising the creation of a change log for transactional sources in an ETL pipeline

Recommended ETL workflow for weekly ingestion of .sql.tz "database dumps" from Blob Storage into Unity Catalogue-enabled Metastore

Re: DLT failure: ABFS does not allow files or directories to end with a dot

Re: DLT failure: ABFS does not allow files or directories to end with a dot

Re: Optimising the creation of a change log for transactional sources in an ETL pipeline

Re: Optimising the creation of a change log for transactional sources in an ETL pipeline

Re: Recommended ETL workflow for weekly ingestion of .sql.tz "database dumps" from Blob St