Updating python version from 3.8 to 3.12 for s3 ingestion

[sandipkumar]

I went to AWS Cloudformation stack and edited the template from python 3.8 to 3.12 and updated. I did this for both the workspace stack and the s3 ingestion stack. Will it break anything? Do I need to make any changes in the python code in the template so that there are no problems? If so, what do I need to change?

What is the best way to update cloud formation template automatically created by Databricks while still not breaking it?

All the s3 connections I made were made in the last 3 months. Why is Databricks still using python 3.8 in its templates?

[Kaniz_Fatma]

Hi @sandipkumar, Updating the Python version in your AWS CloudFormation templates from 3.8 to 3.12 should not break anything, as long as your existing Python code is compatible with Python 3.12.

1. Ensure that your existing Python code in the CloudFormation templates is compatible with Python 3.12. While Python 3.12 is largely backwards compatible with 3.8, you may need to address some minor syntax or library changes.

2. To automatically update the CloudFormation templates created by Databricks, you can use AWS CloudFormation StackSets. StackSets allow you to create, update, or delete stacks across multiple AWS accounts and Regions with a single operation.

   • You can create a StackSet with the updated Python 3.12 version and apply it to the existing Databricks-generated stacks.
   • This allows you to centrally manage the Python version update without having to manually edit each individual stack.

[sandipkumar]

Hi @Kaniz_Fatma ,

Thanks a lot! I will look at StackSets.

As I mentioned the code is not written by me but Databricks. Why does Databricks not use a new python version in its default stacks? We are low on resources and heavily rely on the default Databricks code and minimise devops interference. Moreover, even if we use StackSets to update the version, how do we know that the Databricks code will work and not break? Are we sure that code written by databricks is backward compatible?