To be transparent, we didn't know we were using SCIM either, which was one of the reasons it took us a bit to figure it out.  Basically what happened, is that an update to one group removed access from people who weren't part of that group, even though they were under another group that should have had access.  Basically Azure Databricks doesn't work with nested groups properly.  So to fix it, we removed the nested groups, and added only the specific groups to the relevant permissions.  There were only two of us who weren't a member of the second group, so it seemed like a specific permission setting, instead of a group setting.