Ryan_Chynoweth
Databricks Employee
Databricks Employee

Hi @Gimwell Young​ ,

Happy that it helped! With Unity Catalog you specify a default location in ADLS for all your data so it should be handled for you. Please reach out to your Databricks account team to learn more.

Creating and writing to a managed table would look something like the following. Note that there are many ways to do this. The only difference to your code is the "toTable" in the write stream.

df = (spark
   .readStream
   .format("cloudFiles") 
   .option("cloudFiles.format", source_format)
   .option('header', 'true')  
   .option("cloudFiles.schemaLocation", schema_path)
   .load(adls_raw_file_path)
)
 
(df.writeStream
    .format('delta')
    .outputMode('append')
    .queryName(query_name)
    .option('checkpointLocation', checkpoint_path)
    .option("mergeSchema", "true")
    .trigger(availableNow=True)
    .toTable("<TABLE NAME HERE>")
)