Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-06-2021 06:05 AM
So one way of doing this is to simple iterate over the directories at that level and create a table for each. Assuming that the schema is the same in all files you could simple do something like
subjects = dbutils.fs.ls("abfss://landing@account.dfs.core.windows.net/sc/raw")
for subject in subject:
table_name = subject.name.strip("/")
spark.sql(f"CREATE TABLE IF NOT EXISTS `{table_name}` USING PARQUET LOCATION '{subject.path}'")