Cannot find an AWS Cloudwatch init script that supports runtime 11.x

[IdanYaffe]

Hi all,

I'm using the AWS CW init global script in order to monitor my clusters' instances.

I'm also using data live tables with some autoloader jobs.

Unfortunately, the data live tables are now running runtime version 11.

As a result, newly created pipelines are failing and the init script needs some adjustments to runtime 11.

Is there an init script for version 11? (I couldn't fine one)

Is there a way to force newly created data live tables to use an older runtime for the time being? (I saw an option for current=version 11 and preview=a newer version).

Thanks

[Hubert-Dudek]

Unfortunately, in delta live tables, you can not specify runtime (except current and preview, which you mentioned). It could be helpful that DLT runtimes releases are mentioned on the databricks side the same way as SQL, ML, and standard ones @Kaniz Fatma​ @Prabakar Ammeappin​ @Lindsay Olson​ 

Regarding the cloudwatch script, I know that one from the AWS blog https://aws.amazon.com/blogs/mt/how-to-monitor-databricks-with-amazon-cloudwatch/

I think the issue is that since 11.0 log4j is in version 2:

Log4j is upgraded from Log4j 1 to Log4j 2

Log4j 1 is being upgraded to Log4j 2. Legacy Log4j 1 dependencies are being removed.

If you depend on the Log4j 1 classes that were previously included inside the Databricks Runtime, those classes no longer exist. You should upgrade your dependencies to Log4j 2.

If you have custom plugins or configuration files that depend on Log4j 2, they may no longer work with the version of Log4j 2 in this release. For assistance, contact your Databricks representative.