Hello JD,
Here is a suggested approach:
Within the single pipeline that owns dim_customer :
1. Declare the streaming table with __START_AT and __END_AT .
2. Add a backfill flow:
CREATE FLOW backfill_dim_customer ASINSERT INTO ONCE dim_customer ...
Hello @LSIMS ,
Based on my understanding you are seeing slow reads in general including the count(*) from Oracle itself.. based on that I'd suggest the following:
- If fetch(read) is slow that means the upstream data is not optimized, a good optimiza...
Hello @Upendra_Dwivedi ,
This is potentially a missing package in your local Python setup, kindly can you check troubleshooting steps here and let me know
In the alternative this didn't work please share the output of the following commands:
python ...
Hello @Abser786,
There is a difference between Dynamic Resource Allocation and the Scheduler policy
Dynamic Resource Allocation means getting more compute as needed if current compute is totally consumed, this can be achieved by autoscaling feature/c...
Hello @sshssh
Databricks support Notebook scoped environments for databricks runtime (regular and ML)
Kindly please check blog post here for ways to install Notebook scoped libraries in databricks
Regards
