- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
04-01-2023 10:14 PM
@Martin Huige :
To enable Photon for your Databricks Python workbooks that are scheduled/invoked by Azure Data Factory, you will need to use an existing Databricks cluster that is configured with Photon. At this time, creating a new job cluster in Azure Data Factory does not provide an option to enable Photon.
To use an existing Databricks cluster that is configured with Photon, you can specify the cluster ID in the Databricks Linked Service configuration in Azure Data Factory. To do this, follow these steps:
- In your Azure Data Factory pipeline, click on the Databricks activity that runs the Python workbook.
- Click on the "Settings" tab, and then click on the "Databricks Linked Service" dropdown.
- Select your Databricks Linked Service from the dropdown, or create a new one if you haven't already.
- In the "Cluster ID" field, enter the ID of the Databricks cluster that is configured with Photon. You can find the cluster ID in the Databricks workspace UI, or you can use the Databricks API to retrieve it.
- Save your changes and run your pipeline.
With this configuration, the Databricks activity in your Azure Data Factory pipeline will use the specified Databricks cluster, which is configured with Photon, to run your Python workbook.