topic Re: Avoid AWS SDK 1.x dependencies in cluster classpaths in Data Engineering

Avoid AWS SDK 1.x dependencies in cluster classpaths

Mathias_Peters — Mon, 26 Oct 2026 15:29:53 GMT

Hi,

As part of an AWS well architected review, we received a recommendation to stop using AWS SDK 1.x since this version has been deprecated since end of 2025. A cloud trail search showed that this lib is still in use in our data AWS accounts. In the Databricks resources I found this: https://docs.databricks.com/aws/en/release-notes/runtime/18.2#aws-sdk-v1-dependencies-are-shaded

Question: when will the migration to v2 be done and which DBR version should we use here?

Thank you and kind regards

Mathias

Re: Avoid AWS SDK 1.x dependencies in cluster classpaths

harry085drop — Tue, 27 Oct 2026 05:14:35 GMT

Hello, As shading SDK v1 dependencies doesn’t guarantee a complete switch to v2. Also, verify whether the CloudTrail events come from Databricks itself or your own workloads before deciding which DBR version to use.

Re: Avoid AWS SDK 1.x dependencies in cluster classpaths

johnsmith9912 — Tue, 27 Oct 2026 06:54:30 GMT

Please refer to the website for details on AWS SDK v1 dependencies in Databricks Runtime 18.2. For the migration timeline and recommended DBR version, we should confirm the latest guidance with Databricks Support.

Re: Avoid AWS SDK 1.x dependencies in cluster classpaths

data_pulse — Tue, 27 Oct 2026 07:54:00 GMT

One important distinction here is AWS SDK v1 classpath exposure vs actual SDK v1 usage.

DBR 18.2+: Databricks shaded the bundled AWS SDK v1 dependencies, so they're no longer directly exposed on the classpath. However, this doesn't mean all internal usage has migrated to SDK v2.
Migration status: Databricks docs describes shading AWS SDK v1 dependencies in DBR 18.2 as a preparatory step toward SDK v2 migration, not confirmation that the migration is completd. No timeline as well been specified.
CloudTrail findings: Upgrading alone may not eliminate SDK v1 activity. Check the CloudTrail userAgent, IAM role and affected operations to identify the caller.
Custom dependencies: Also inspect cluster libraries and application JARs for transitive aws-java-sdk-* dependencies.

My suggestion: Test a representative workload on DBR 18.2+ and compare CloudTrail results. If SDK v1 calls persist after ruling out application dependencies, share evidence with Databricks Support to identify the responsible component.

Key takeaway: Shading SDK v1 dependencies is not the same as eliminating SDK v1 API calls.

References: DBR 18.2 release notes | AWS SDK v1 detection guide

Re: Avoid AWS SDK 1.x dependencies in cluster classpaths

datarich — Tue, 27 Oct 2026 12:36:14 GMT

Hi Mathias, good question. A lot of AWS customers are seeing this after their reviews.

When will the v2 migration be done? Databricks hasn't published a date. The DBR 18.2 change you found is the first step. AWS SDK v1 is now shaded inside the runtime, so it's no longer on your classpath. That lets Databricks move its own code to v2 without breaking customer code. The rest of the migration is still in progress.

What this means for your CloudTrail findings Shading moves v1 off your classpath, but the runtime still uses it internally for now. So even on the latest runtime, you'll keep seeing aws-sdk-java/1.x user agents in CloudTrail from Databricks compute until the migration is finished. Changing the DBR version won't make them disappear yet.

Which DBR version to use

Use Databricks Runtime 18 LTS or later, or serverless. These include the shading change, so v1 isn't exposed to your code.
If your own jobs or libraries still import AWS SDK v1, move them to v2. Or, as a temporary measure, add v1 as an explicit dependency, since the runtime no longer provides it.

For your review

Separate your own v1 usage from Databricks'. Check the CloudTrail userAgent and the calling role. Calls from your own code or libraries are the ones you can fix now.
Note the rest as a dependency on Databricks, with the 18.2 shading as evidence that the migration is underway.
If you need a timeline for your risk register, ask your Databricks account team. They can share updates as they become available.

Docs: