This is exactly the scenario apply_changes_from_snapshot was designed for. It compares consecutive full snapshots and automatically derives inserts, updates, and deletes by absence no delete indicator column needed.
Great question the core principle is: credentials should never travel through the agent itself. Instead of PATs or embedding a client_secret in your agent, use your Identity Provider (Entra ID / Okta) with the OBO flow:User authenticates via your IdP...
Got it! So those three bugs were the culprits:Typo — inferSchemaccc instead of inferSchemaYou stored the path in data but forgot to pass it into .load(data)res.sh is incomplete — should be res.show()from pyspark.sql.functions import *data = "workspac...
