I got in touch with Microsoft support and they mentioned it is not possible as the Azure Databricks app registration is managed by Databricks and changing the manifest to exclude group claims on that application is not possible and it impacts all users.