On (1): the docs don't gate Scala foreachBatch on serverless. Limitations with Databricks Connect for Scala lists streaming foreachBatch as unavailable only on Databricks Runtime 13.3 LTS and below, and serverless limitations names only Trigger.ProcessingTime and Trigger.Continuous as blocked triggers. Serverless Scala and Java JAR jobs are in Public Preview. I'm reading published docs here, not speaking for support.

On (2): I can't find any published Databricks doc or KB that names RST_STREAM or PROTOCOL_ERROR, so I wouldn't put a label on it from out here. One thing worth raising with support: [UDF_ERROR](https://docs.databricks.com/aws/en/error-messages/udf-error-error-class) has a dedicated ENV_LOST sub-condition for an execution environment that was lost mid-run, and you're getting INTERNAL instead. Whether that distinction is meaningful here is a question for whoever picks up the ticket. Your empty no-op result already rules out the merge logic.

On (3), three things before the case:

1. Make sure your application logs are actually being emitted. On serverless, the SLF4J logging API defaults to a no-op backend and application log output is silently discarded unless you add the log4j-slf4j2-impl bridge matching the log4j-api version in your environment version (2.20.0 on environment version 5). See Configure logging for serverless compute. One note on the log4j.logger.org.apache.spark.sql.connect=DEBUG suggestion above: Spark logs aren't available on serverless, only client-side application logs, only six Spark properties are settable and none are log4j-related, and changing the log4j level through SparkContext is listed as unavailable. Use the query profile in place of the Spark UI.

2. Match Scala, JDK, and your Spark API dependency to your environment version. You mentioned you're not building against Databricks Connect; spark-sql-api is a documented alternative, but Databricks recommends databricks-connect marked provided, so that's worth ruling out. Don't bundle log4j-api or log4j-core.

3. Isolate foreachBatch itself. Swap Kinesis for a small Delta table, Trigger.AvailableNow(), empty foreachBatch, single-class JAR. Delta is a documented serverless streaming source, so a failure there isolates foreachBatch cleanly.

If that still fails, it's a clean repro to attach to a support case. On fallback, note that standard access mode also runs on Spark Connect; dedicated compute is the mode that uses the classic Spark architecture.