Yes, you're correct! When using dropDuplicates within foreachBatch, it operates only on the current micro-batch, so it removes duplicates in a stateless manner for each batch independently. Since there's no continuous state tracking across batches, y...
Instead of copying your files from ADLS to UC volumes, you can create a storage credential and an external location. This allows you to access all your ADLS data directly through the catalog explorer under external locations. For guidance on creating...
