Databricks Community

anhnnguyen

My delta table are stored at HANA data lake file and I have ETL configured like below@DP.materialized_view(temporary=True) def source(): return spark.read.format("delta").load("/data/source") @dp.materialized_view def sink(): return spark.re...

anhnnguyen

Hello guys,I'm building ETL pipeline and need to access HANA data lake file system. In order to do that I need to have sap-hdlfs library in compute environment, library is available in maven repository.My job will have multiple notebook task and ETL ...

anhnnguyen

1 more note that I'm not using Unity Catalog here, not sure if it's relevant

anhnnguyen

hi @Yogesh_Verma_ @GaweL after removing temporary=True, pipeline still full recompute every run even though there is no change in source

anhnnguyen

thanks @XP, it worked like a charmactually I did try with policy before but the one I tried is usage policy so that I could not find where to add library lol

anhnnguyen

I tried and it does not work. After saving config, Databricks will revert it back.The way seem possible is load library from init script but as @szymon_dybczak mentioned, it's not a good way since it will cause unexpected behavior

Databricks Community

User Stats

User Activity

Materialized view always load full table instead of incremental

Adding maven dependency to ETL pipeline

Re: Materialized view always load full table instead of incremental

Re: Materialized view always load full table instead of incremental

Re: Adding maven dependency to ETL pipeline

Re: Adding maven dependency to ETL pipeline