You’re right that the behaviour is weird at first glance (“5k rows on a 64 GB cluster and I blow up on write”), but your stack trace is actually very revealing: this isn’t a classic Delta write / shuffle OOM – it’s SciSpaCy/UMLS falling over when loading its ANN index on the executors.

This is happening inside get_pipeline() when you construct the EntityLinker. That’s where SciSpaCy loads the big UMLS ANN index from disk (via cached_path).

Some points to consider:

• df_entities.show() only triggers enough partitions to show the first N rows (by default 20), so maybe 1–2 Python workers actually run get_pipeline() and load the index.

• df_entities.write... needs to process all partitions, so more Python worker processes spin up across your autoscaling cluster.

• Each new Python worker sees global_nlp is None and tries to load SciSpaCy + UMLS index again.

• The ANN index for UMLS is big (multi-GB). Multiple concurrent loads → heavy disk and memory pressure and/or a partially read index file → basic_ios::clear: iostream error.

So the “OOM on write” is a side effect of many workers loading huge models and a huge index at once, not the Delta write itself.

Can you please try the following?

1. Stop autoscaling and run a single worker (or even single-node cluster):

   • Driver: 64 GB, 8 cores

   • Workers: 0–1 (or use a single-node cluster with driver only)

2. Force fewer Python workers by lowering partitions (just 1 or max 2)

3. Rerun. 

I have some other ideas but let's first see this one.