Hello @saleem_shady!

Have you tried including the parameter ddp_find_unused_parameters=False in your TrainingArguments? Here's an example of how to include it: https://github.com/databricks/databricks-ml-examples/blob/master/llm-models/llamav2/llamav2-7b/06_fi...

If you have already included this parameter and are still encountering issues, please share the error message you are receiving as a reply in this post.

Best Regards,

Jéssica Santos