โ06-27-2022 09:48 PM
Hi Team,
Can we pass Delta Live Table name dynamically [from a configuration file, instead of hardcoding the table name]? We would like to build a metadata-driven pipeline.
โ06-28-2022 06:12 AM
Yes, it is possible. Just pass the variable to @dlt.table(name=variable)
for name in ['table1', 'table2']:
@dlt.table(name=name)
def delta_live_table():
return (
spark.range(1, 10)
)
โ06-28-2022 06:56 AM
Thanks, @Hubert Dudek for your quick response on this, I can able to create DLT dynamically.
Can we pass the Database name while creating DLT tables instead of passing the database name in the pipeline configuration?
Error message :
org.apache.spark.sql.AnalysisException: Materializing tables in custom schemas is not supported. Please remove the database qualifier from table 'default.Delta_table3'.
โ07-08-2022 07:21 AM
I hope this limitation is resolved - storing everything from one pipeline in a single database is not ideal. Preferably I'd like to be able to store bronze level data in it's own "Database" rather than mix with silver/gold.
โ07-24-2022 10:15 PM
Hi @Dan Richardsonโ There is a feature request for this limitation already in queue.This is the feature request ID: DB-I-5073. We do not have any ETA on it yet and will be implemented once prioritized . Please note that you won't be able to access the feature request as it is internal to Databricks, however you can always follow-up with above ID for the status update on this.
โ08-15-2022 01:55 PM
Hi @Dan Richardsonโ,
Just a friendly follow-up. do you have any follow-up questions or did Noopur's response helped you? please let us know
โ01-17-2024 02:12 PM
Hi, have there been any updates on this feature or internal ticket? This would be a great addition. Thanks!
โ02-27-2024 02:50 PM
I am observing same error while I adding dataset.tablename.
org.apache.spark.sql.catalyst.ExtendedAnalysisException: Materializing tables in custom schemas is not supported. Please remove the database qualifier from table 'streaming.dlt_read_test_files'
@Dlt.table(name="streaming.dlt_read_test_files")
def raw_data():
return spark.readStream.format("delta").load(abfss_location)
@dlt.table(name="streaming.dlt_clean_test_files")
def filtered_data():
return dlt.readStream("streaming.dlt_read_test_files").select(F.col("data"))
Do we have update on this topic?
โ07-12-2024 02:20 PM
Hello,
I wonder if there is any update for this feature?
Thanks
Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโt want to miss the chance to attend and share knowledge.
If there isnโt a group near you, start one and help create a community that brings people together.
Request a New Group