topic Re: Azure CosmosDB change feed ingestion via DLT in Data Engineering

Azure CosmosDB change feed ingestion via DLT

Ogi — Wed, 04 Jan 2023 10:57:08 GMT

Is there a way to ingest Azure CosmosDB data via Delta Live Tables? If I use regular workflows it works well, but with DLT I'm not able to set CosmosDB Connector on a cluster.

Re: Azure CosmosDB change feed ingestion via DLT

Hubert-Dudek — Wed, 04 Jan 2023 11:18:12 GMT

In DLT, you can install libraries only through pip. The problem is that the CosmosDB driver is installed through Maven / JAR. I don't think it is preinstalled.

Maybe you can send the change data feed to Event Hub and read Event Hub in DLT as Kafka.

https://learn.microsoft.com/en-us/azure/cosmos-db/nosql/changefeed-ecommerce-solution

Re: Azure CosmosDB change feed ingestion via DLT

Jfoxyyc — Wed, 04 Jan 2023 21:59:59 GMT

I would likely do the same, but instead send the change data feed to Event Hub and materialize in abfss as json or parquet or some format and consume via DLT using autoloader.

Re: Azure CosmosDB change feed ingestion via DLT

Ogi — Wed, 01 Feb 2023 13:31:42 GMT

Thanks a lot! Just wanted to doublecheck whether this natively exists.