Hi community,

I'm exploring ways to perform low-level, programmatic operations on Delta tables directly from a PySpark environment.

The standard delta.tables.DeltaTable Python API is excellent for high-level DML, but it seems to abstract away the core transactional engine. My goal is to interact with this engine directly.

My research suggests that it might be possible to access the underlying Scala/Java APIs by using the spark._jvm gateway. I'd like to ask for community guidance on whether this is the correct approach to access the internal org.apache.spark.sql.delta.DeltaLog and OptimisticTransaction objects.

Specifically, if spark._jvm is indeed the right path:

1. What is the canonical way to obtain a DeltaLog instance for a given table, especially considering tables in Unity Catalog?
2. Once a DeltaLog object is obtained, is calling its startTransaction() method the standard way to get an OptimisticTransaction?
3. Is the DeltaLog API intentionally kept separate from the high-level DeltaTable wrapper to maintain a stable public API?

I'm essentially looking for the best practices and potential pitfalls when between the PySpark API and the core JVM engine for transactional control.

Any advice or confirmation would be highly appreciated.

Thank you!