Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-12-2023 07:37 AM
This is due to the distributed/parallel nature of spark.
You could add a .coalesce(1) to your write statement. Like that only one file will be generated, BUT it will still reside in a directory and have a funny name.
Frankly, I would use Data Factory for ingest. Cheap, fast, a pain to use but see the previous two points 🙂