Hi @dyusuf ,It could be because AQE (Adaptive Query Execution) is enabled....AQE, dynamically handles skew...Please refer below link for more details:https://docs.databricks.com/aws/en/optimizations/aqeCan you please disable AQE and check if this wor...
Hi @glevin ,Can I check what is your data source (Azure SQL, Oracle, MySQL etc.), may be the driver of your source is just picking the default fetch value?Can you share your profiler output which shows, it's returning 1000 records not 5K as desired?
Hi @ShivangiB,Can you share the definition of your table - ucdata.brz.liquidcluster_table_data?Curious - why your source and target are the same table? Are you trying to store the different grains of data in the same table?I think you are referring t...
Hi @Rachana2,As @Alberto_Umana has mentioned I'd check table_lineage / column_lineage tables, as maintaining a lineage through a bespoke pipeline/tooling may not be a right approach.Can you please explain your use case which explains why you don't wa...
