Hi!

I have a problem with user memory on driver (I have almost several mb of storage memory, 0 Execution memory and more than 7GB of JVM Memory on Heap in use).

How it can be? I don't have any broadcast variables, joins or aggregations.

All the pipeline:

1. I create a list of 20 string values

2. I iterate throw this list and do the following:

a. Create dlt.view from streaming source

b. Create streaming_live_table

c. Use apply_changes for scd-1

What this 7GB of Heap Memory on driver can be? As I understand, it's an user memory, but why so huge amount?

How I can reduce it? Is it a metadata for dlt or smth like that?

Is it a good approach to iterate throw the python list to create dlt?

Hope someone can give answers to some questions. Thank you in advance