topic Re: Delta Live Table name dynamically in Data Engineering

Delta Live Table name dynamically

Phani1 — Tue, 28 Jun 2022 04:48:48 GMT

Hi Team,

Can we pass Delta Live Table name dynamically [from a configuration file, instead of hardcoding the table name]? We would like to build a metadata-driven pipeline.

Re: Delta Live Table name dynamically

Hubert-Dudek — Tue, 28 Jun 2022 13:12:10 GMT

Yes, it is possible. Just pass the variable to @dlt.table(name=variable)

for name in ['table1', 'table2']:
   @dlt.table(name=name)
   def delta_live_table():
      return (
         spark.range(1, 10)
       )

Re: Delta Live Table name dynamically

Phani1 — Tue, 28 Jun 2022 13:56:42 GMT

Thanks, @Hubert Dudek for your quick response on this, I can able to create DLT dynamically.

Can we pass the Database name while creating DLT tables instead of passing the database name in the pipeline configuration?

Error message :

org.apache.spark.sql.AnalysisException: Materializing tables in custom schemas is not supported. Please remove the database qualifier from table 'default.Delta_table3'.

Re: Delta Live Table name dynamically

DanR — Fri, 08 Jul 2022 14:21:13 GMT

I hope this limitation is resolved - storing everything from one pipeline in a single database is not ideal. Preferably I'd like to be able to store bronze level data in it's own "Database" rather than mix with silver/gold.

Re: Delta Live Table name dynamically

Noopur_Nigam — Mon, 25 Jul 2022 05:15:21 GMT

Hi @Dan Richardson There is a feature request for this limitation already in queue.This is the feature request ID: DB-I-5073. We do not have any ETA on it yet and will be implemented once prioritized . Please note that you won't be able to access the feature request as it is internal to Databricks, however you can always follow-up with above ID for the status update on this.

Re: Delta Live Table name dynamically

jose_gonzalez — Mon, 15 Aug 2022 20:55:58 GMT

Hi @Dan Richardson,

Just a friendly follow-up. do you have any follow-up questions or did Noopur's response helped you? please let us know

Re: Delta Live Table name dynamically

cpayne_vax — Wed, 17 Jan 2024 22:12:33 GMT

Hi, have there been any updates on this feature or internal ticket? This would be a great addition. Thanks!

Re: Delta Live Table name dynamically

Azure_dbks_eng — Tue, 27 Feb 2024 22:50:59 GMT

I am observing same error while I adding dataset.tablename.

org.apache.spark.sql.catalyst.ExtendedAnalysisException: Materializing tables in custom schemas is not supported. Please remove the database qualifier from table 'streaming.dlt_read_test_files'

@Dlt.table(name="streaming.dlt_read_test_files") def raw_data(): return spark.readStream.format("delta").load(abfss_location) @dlt.table(name="streaming.dlt_clean_test_files") def filtered_data(): return dlt.readStream("streaming.dlt_read_test_files").select(F.col("data"))

Do we have update on this topic?

Re: Delta Live Table name dynamically

Vic01 — Fri, 12 Jul 2024 21:20:18 GMT

Hello,

I wonder if there is any update for this feature?

Thanks

Re: Delta Live Table name dynamically

Wilson_Haenisch — Thu, 17 Oct 2024 15:04:03 GMT

This would be a great improvement to DLT. Majority of the architecture requirements I see, do separate bronze, silver and gold at the schema level. We can get around the issue separating the DLT pipelines into 3 separate ones, but you loose the ability to follow the pipeline end-to-end and add delays in processing.

Re: Delta Live Table name dynamically

bmhardy — Wed, 11 Dec 2024 12:09:03 GMT

Is this post referring to Direct Publishing Mode? As we are multi-tenanted we have to have separate schema per client, which currently means a single pipeline per client. This is not cost effective at all, so we are very much reliant on DPM. I believe it is not due to go in to public preview now until February at the earliest. It would be good if this can be fast-tracked if more people need it.