Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
Is there a way to ingest Azure CosmosDB data via Delta Live Tables? If I use regular workflows it works well, but with DLT I'm not able to set CosmosDB Connector on a cluster.
In DLT, you can install libraries only through pip. The problem is that the CosmosDB driver is installed through Maven / JAR. I don't think it is preinstalled.
Maybe you can send the change data feed to Event Hub and read Event Hub in DLT as Kafka.
In DLT, you can install libraries only through pip. The problem is that the CosmosDB driver is installed through Maven / JAR. I don't think it is preinstalled.
Maybe you can send the change data feed to Event Hub and read Event Hub in DLT as Kafka.
I would likely do the same, but instead send the change data feed to Event Hub and materialize in abfss as json or parquet or some format and consume via DLT using autoloader.
Thanks a lot! Just wanted to doublecheck whether this natively exists.
Connect with Databricks Users in Your Area
Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.
If there isn’t a group near you, start one and help create a community that brings people together.
