https://community.databricks.com/t5/data-engineering/configuring-airflow/m-p/26535#M18568 <P>I would recommend having the 'user' the Databricks Jobs are triggered by as a dedicated user. This is what I would consider a 'Service Account' and I'll drop a definition for that type of user below.</P><P></P><P>Seeing that you have SSO enabled, I might create this user in the IDP system in gsuite, and propagate this newly created user into the Databricks workspace. I would ensure this user has the appropriate Job Create permissions, and then generate a PAT for integration with Airflow.</P><P></P><P>Having a dedicated account will help with monitoring its usage, and also ensure CI/CD best practices are observed for code promotion. This also ensures that the 'normal user' PAT is not in the loop, because if that 'normal user' were to leave the company, they could be de-provisioned from the Databricks workspace, which invalidates the PAT, which then breaks your Airflow integration.</P><P></P><P>Service accounts are a special type of non-human <A href="https://www.beyondtrust.com/resources/glossary/privileged-access-management-pam" alt="https://www.beyondtrust.com/resources/glossary/privileged-access-management-pam" target="_blank">privileged account</A> used to execute applications and run automated services, virtual machine instances, and other processes.</P> Mon, 07 Jun 2021 15:25:00 GMT User16783855117 2021-06-07T15:25:00Z https://community.databricks.com/t5/data-engineering/configuring-airflow/m-p/26534#M18567 <P>Should we create a Databricks user for airflow and generate a personal access token for it? We also have gsuite SSO enabled, does that mean I need to create a gsuite account for the user as well?</P> Wed, 02 Jun 2021 23:57:55 GMT https://community.databricks.com/t5/data-engineering/configuring-airflow/m-p/26534#M18567 Anonymous 2021-06-02T23:57:55Z https://community.databricks.com/t5/data-engineering/configuring-airflow/m-p/26535#M18568 <P>I would recommend having the 'user' the Databricks Jobs are triggered by as a dedicated user. This is what I would consider a 'Service Account' and I'll drop a definition for that type of user below.</P><P></P><P>Seeing that you have SSO enabled, I might create this user in the IDP system in gsuite, and propagate this newly created user into the Databricks workspace. I would ensure this user has the appropriate Job Create permissions, and then generate a PAT for integration with Airflow.</P><P></P><P>Having a dedicated account will help with monitoring its usage, and also ensure CI/CD best practices are observed for code promotion. This also ensures that the 'normal user' PAT is not in the loop, because if that 'normal user' were to leave the company, they could be de-provisioned from the Databricks workspace, which invalidates the PAT, which then breaks your Airflow integration.</P><P></P><P>Service accounts are a special type of non-human <A href="https://www.beyondtrust.com/resources/glossary/privileged-access-management-pam" alt="https://www.beyondtrust.com/resources/glossary/privileged-access-management-pam" target="_blank">privileged account</A> used to execute applications and run automated services, virtual machine instances, and other processes.</P> Mon, 07 Jun 2021 15:25:00 GMT https://community.databricks.com/t5/data-engineering/configuring-airflow/m-p/26535#M18568 User16783855117 2021-06-07T15:25:00Z