Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-02-2023 05:12 AM
When using Delta Live Tables with SQL-syntax ignoreMissingFiles-option is not working and you are getting error anyway.(See picture below)
Link to feature:
https://learn.microsoft.com/en-us/azure/databricks/ingestion/auto-loader/options#generic-options
Here is the SQL –code for Bronze -layer
CREATE OR REFRESH TEMPORARY STREAMING LIVE TABLE bronze_InventSite
AS
SELECT
_c0 as LastProcessedChange_DateTime
,_c1 as DataLakeModified_DateTime
,_c2 as Start_LSN
,_c3 as End_LSN
,_c4 as DML_Action
,_c5 as Seq_Val
,_c6 as Update_Mask
,_c7 as RECID
,_c8 as DefaultDimension
,_c9 as DefaultInventStatusId
,_c10 as Name
,_c11 as OrderEntryDeadlineGroupId
,_c12 as SiteId
,_c13 as TaxBranchRefRecId
,_c14 as Timezone
,_c15 as IsReceivingWarehouseOverrideAllowed
,_c16 as DataAreaId
,_c17 as PARTITION
,_c18 as RECVERSION
,_c19 as SYSROWVERSIONNUMBER
FROM cloud_files("/mnt/d365fscm/ChangeFeed/InventSite", "csv",map("delimiter", ",", "header", "false", "ignoreMissingFiles", "true"))And here is SQL- code for SCD2 table:
CREATE OR REFRESH STREAMING LIVE TABLE silver_InventSite
;
APPLY CHANGES INTO
live.silver_InventSite
FROM
stream(live.bronze_InventSite)
KEYS
(RECID)
IGNORE NULL UPDATES
SEQUENCE BY
Start_LSN /*Start_LSN*/
STORED AS
SCD TYPE 2;
Labels:
- Labels:
-
Delta Live Tables
-
SQL