Databricks Community

aliacovella

I have this behavior that I'm not understanding. I have a notebook that defines a DLT from a Kinesis stream and a view from that DLT. This works when I run it from within workflow, configured using the DLT pipeline. If, however, I create a workflow a...

aliacovella

I'm trying to create a pipeline containing a view from a federated source. In this case, I'd like to just create materialized views from the federation and and schedule the pipeline for execution. If I define a pipeline with only something like the ...

aliacovella

Thanks, I do have one more question. In the following scenario:A new migration via AWS database migration service has been configured with the CDC going to KinesisA new DLT has been configured to read from the Kinesis streamAn initial load of potenti...

aliacovella

Thanks, I originally tried the Kinesis route, and it worked well but was hoping there was a simpler solution. I'll go with that then.

aliacovella

From what I can see, it looks like it queries all the records again. I tried looking into the apply changes API, but that seems to require a streaming table, and it appears that streaming tables are not supported from jdbc sources and in this case it...

aliacovella

Thanks, that makes sense. So if I create a table against a federated data source, when changes occur in the source table, does it automatically handle the change data capture or does it perform a table scan on the federated source to determine change...

Databricks Community

User Stats

User Activity

DLT Vs Notebook runs

DLT Pipleline with only views

Re: DLT Pipleline with only views

Re: DLT Pipleline with only views

Re: DLT Pipleline with only views

Re: DLT Pipleline with only views