how to parallel n number of process in databricks
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-19-2025 12:32 AM
Requirement: I have a volume in which random txt file coming from MQ with random numbers. In my workspace I have python script. Also, i have created job which, when new file will come in volume it will trigger automatically.
My requirement is, I need some thing in middle which will run or execute n number of times when n number of files will come in volume with n number of python script, meaning python script is only one but it should call n number of times with n number of files. I do not want python scrpit as a multithreading or multiprocess to do this work is there any other way to do it.
I am attaching flow chart of understading my requirement.