Databricks Community

aliacovella

I'm trying to debug a task that is a DLT workflow and I've tried putting in log statements and print statements but I can't seem to see the output in the event log after the run nor can I see the print statements anywhere. Can someone point me to whe...

aliacovella

Here I have a table named organizations_silver that was build from a bronze table created from a Kinesis change data capture feed.@dlt.table(name="kinesis_raw_stream", table_properties={"pipelines.reset.allowed": "false"})def kinesis_raw_stream(): ...

aliacovella

I have this behavior that I'm not understanding. I have a notebook that defines a DLT from a Kinesis stream and a view from that DLT. This works when I run it from within workflow, configured using the DLT pipeline. If, however, I create a workflow a...

aliacovella

I'm trying to create a pipeline containing a view from a federated source. In this case, I'd like to just create materialized views from the federation and and schedule the pipeline for execution. If I define a pipeline with only something like the ...

aliacovella

I'm not actually getting a failure, but I want to see that the value of a parameter I set up in the workflow is being set correctly where it is used. In this case I'd like to parameterize the Kinesis stream I'll be polling from, so I'd like to pass t...

aliacovella

Thanks, I have it working now.

aliacovella

Thanks, I do have one more question. In the following scenario:A new migration via AWS database migration service has been configured with the CDC going to KinesisA new DLT has been configured to read from the Kinesis streamAn initial load of potenti...

aliacovella

Thanks, I originally tried the Kinesis route, and it worked well but was hoping there was a simpler solution. I'll go with that then.

aliacovella

From what I can see, it looks like it queries all the records again. I tried looking into the apply changes API, but that seems to require a streaming table, and it appears that streaming tables are not supported from jdbc sources and in this case it...

Databricks Community

User Stats

User Activity

How can I get logging or print output from a Delta Live Table workflow.

How can I dedupe from a table created from a Kinesis change data capture feed.

DLT Vs Notebook runs

DLT Pipleline with only views

Re: How can I get logging or print output from a Delta Live Table workflow.

Re: DLT Vs Notebook runs

Re: DLT Pipleline with only views

Re: DLT Pipleline with only views

Re: DLT Pipleline with only views