Thanks @szymon_dybczak for the detailed steps. In addition to the above I had to write the logic to create directory if not present to get it working -
parent_dir = os.path.dirname(experiment_name)
dbutils.fs.mkdirs(parent_dir)
To summarise -
1. Created the ML experiment from within the directory.
2. Verified and corrected the full path of the experiment.
To summarise -
1. Created the ML experiment from within the directory.
2. Verified and corrected the full path of the experiment.
3. Added permission at the experiment level.
4. Added logic to create the directory if not present.
