Migrating Service Principals from Non-Unity to Unity-Enabled Databricks Workspaces - Entitlements Mi

[Sudheer2]

 Hello Databricks Community,

I am currently in the process of migrating Service Principals from a non-Unity workspace to a Unity-enabled workspace in Databricks. While the Service Principals themselves seem to be migrating correctly, I am facing an issue where the permissions (entitlements) associated with the Service Principals in the source workspace are not matching in the target Unity-enabled workspace.

The Service Principals in the source workspace have specific permissions such as:

• Allow cluster creation
• Databricks SQL access
• Workspace access

  However, after migrating these Service Principals, the entitlements (permissions) in the target Unity-enabled workspace are not transferred correctly, resulting in a mismatch.

  Code:

  Here’s the code I’m using for migrating the Service Principals:

  python

  Copy code

  def import_service_principal(sp, target_host, target_token): headers = get_headers(target_token) url = f'{target_host}/api/2.0/preview/scim/v2/ServicePrincipals' sp_check_url = f'{url}/{sp["applicationId"]}' # Check if service principal already exists by application ID # Check if the service principal already exists in the target system if resource_exists(sp_check_url, headers): logging.warning(f"Service Principal with ID {sp['applicationId']} already exists in target system. Skipping import.") return False data = { "schemas": sp.get("schemas", []), "applicationId": sp["applicationId"], "displayName": sp.get("displayName", ""), "description": sp.get("description", "") } try: make_request_with_error_handling(url, headers, method='POST', data=data) logging.info(f"Service Principal {sp['applicationId']} imported successfully.") return True except KeyError as ke: logging.error(f"KeyError importing service principal: {sp}. Missing key: {ke}") except requests.exceptions.HTTPError as err: if err.response.status_code == 409: logging.warning(f"Service Principal with ID {sp['applicationId']} already exists in target system. Skipping import.") else: logging.error(f"HTTP error occurred importing service principal: {sp}. Error: {err}") except Exception as e: logging.error(f"Error importing service principal: {sp}. Error: {e}") return False

  Issue:

  • Permissions like "Allow cluster creation", "Databricks SQL access", and "Workspace access" are not migrating properly from the source non-Unity workspace to the Unity-enabled target workspace.
  • The entitlements (permissions) are not transferred as expected, causing mismatches between the source and target workspaces.

    Has anyone encountered a similar issue when migrating Service Principals from a non-Unity to a Unity-enabled Databricks workspace? If so, could you provide any suggestions, solutions, or steps to ensure that the permissions and entitlements are correctly migrated as well?

    Thank you in advance for your help!

[Walter_C]

Hello, are this Service Principals Account Level Service principals? If this is the case what you are currently doing is assign the service principals to the new workspace, but I cannot see on the script and API call JSON that you are setting up the entitlement part of the SP, the entitlement is handle differently on each workspace so it is a manual process to be enabled.

Have you got this script from an specific source that you can share with us?

[Walter_C]

Please try the following:

def import_service_principal(sp, target_host, target_token):
headers = get_headers(target_token)
url = f'{target_host}/api/2.0/preview/scim/v2/ServicePrincipals'
sp_check_url = f'{url}/{sp["applicationId"]}'

# Check if service principal already exists by application ID
if resource_exists(sp_check_url, headers):
logging.warning(f"Service Principal with ID {sp['applicationId']} already exists in target system. Skipping import.")
return False

data = {
"schemas": sp.get("schemas", []),
"applicationId": sp["applicationId"],
"displayName": sp.get("displayName", ""),
"description": sp.get("description", ""),
"entitlements": sp.get("entitlements", [])
}

try:
make_request_with_error_handling(url, headers, method='POST', data=data)
logging.info(f"Service Principal {sp['applicationId']} imported successfully.")
return True
except KeyError as ke:
logging.error(f"KeyError importing service principal: {sp}. Missing key: {ke}")
except requests.exceptions.HTTPError as err:
if err.response.status_code == 409:
logging.warning(f"Service Principal with ID {sp['applicationId']} already exists in target system. Skipping import.")
else:
logging.error(f"HTTP error occurred importing service principal: {sp}. Error: {err}")
except Exception as e:
logging.error(f"Error importing service principal: {sp}. Error: {e}")

return False

[Sudheer2]

Hi @Walter_C ,

Thanks for the solution! It's working around 90%, but I am encountering an issue with 10% of the cases, specifically regarding entitlements.

For example, in one service principal in the source workspace:

• Display Name: test_sp
• Status: Active
• Entitlements:
  • Allow cluster creation is not enabled
  • Databricks SQL access
  • Workspace access

However, when I import this service principal into the target workspace, the Allow cluster creation entitlement is enabled by default, even though it wasn't enabled in the source.

It seems like there's a discrepancy between how entitlements are handled in the source and target systems. I would appreciate any advice on why this might be happening or if there's a way to prevent the Allow cluster creation entitlement from being enabled in the target system unless it’s explicitly set in the source.

Thanks again for your help!

Best regards,
Sudheer Ch.

[Walter_C]

Would be good to do a print pf the values that are getting received on the entitlements for that SP to understand if the value is being retrieved as true

[Sudheer2]

HI @Walter_C ,

Thanks for the suggestion. I've already tried using print statements to debug the entitlements being received, but unfortunately, it still doesn't seem to work correctly. The issue persists, where the Allow cluster creation entitlement that is not enabled in the source is unexpectedly getting enabled in the target system after the import.

Also, to clarify, I'm not using account-level service principals; I'm working with workspace-level service principals only. This could potentially be a key difference in behavior compared to account-level service principals.

If you have any further suggestions, I'd be happy to try them!

Thanks again!