- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-04-2023 02:57 AM
Is there a way to ingest Azure CosmosDB data via Delta Live Tables? If I use regular workflows it works well, but with DLT I'm not able to set CosmosDB Connector on a cluster.
- Labels:
-
Azure
-
ChangeFeed
-
DLT
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-04-2023 03:18 AM
In DLT, you can install libraries only through pip. The problem is that the CosmosDB driver is installed through Maven / JAR. I don't think it is preinstalled.
Maybe you can send the change data feed to Event Hub and read Event Hub in DLT as Kafka.
https://learn.microsoft.com/en-us/azure/cosmos-db/nosql/changefeed-ecommerce-solution
My blog: https://databrickster.medium.com/
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-04-2023 01:59 PM
I would likely do the same, but instead send the change data feed to Event Hub and materialize in abfss as json or parquet or some format and consume via DLT using autoloader.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-01-2023 05:31 AM
Thanks a lot! Just wanted to doublecheck whether this natively exists.
