cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results forย 
Search instead forย 
Did you mean:ย 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results forย 
Search instead forย 
Did you mean:ย 

Avoid AWS SDK 1.x dependencies in cluster classpaths

Mathias_Peters
Contributor II

Monday

Hi, 

As part of an AWS well architected review, we received a recommendation to stop using AWS SDK 1.x since this version has been deprecated since end of 2025. A cloud trail search showed that this lib is still in use in our data AWS accounts. In the Databricks resources I found this: https://docs.databricks.com/aws/en/release-notes/runtime/18.2#aws-sdk-v1-dependencies-are-shaded

Question: when will the migration to v2 be done and which DBR version should we use here?

Thank you and kind regards

Mathias

 

0 Kudos
4 REPLIES 4

harry085drop
New Contributor

Monday

Hello, As shading SDK v1 dependencies doesnโ€™t guarantee a complete switch to v2. Also, verify whether the  CloudTrail events come from Databricks itself or your own workloads before deciding which DBR version to use.

calfresh com

johnsmith9912
New Contributor

Monday

Please refer to the website for details on AWS SDK v1 dependencies in Databricks Runtime 18.2. For the migration timeline and recommended DBR version, we should confirm the latest guidance with Databricks Support.

0 Kudos

data_pulse
Contributor

yesterday

One important distinction here is AWS SDK v1 classpath exposure vs actual SDK v1 usage.

  • DBR 18.2+: Databricks shaded the bundled AWS SDK v1 dependencies, so they're no longer directly exposed on the classpath. However, this doesn't mean all internal usage has migrated to SDK v2.
  • Migration status: Databricks docs describes shading AWS SDK v1 dependencies in DBR 18.2 as a preparatory step toward SDK v2 migration, not confirmation that the migration is completd. No timeline as well been specified.
  • CloudTrail findings: Upgrading alone may not eliminate SDK v1 activity. Check the CloudTrail userAgent, IAM role and affected operations to identify the caller.
  • Custom dependencies: Also inspect cluster libraries and application JARs for transitive aws-java-sdk-* dependencies.

My suggestion: Test a representative workload on DBR 18.2+ and compare CloudTrail results. If SDK v1 calls persist after ruling out application dependencies, share evidence with Databricks Support to identify the responsible component.

Key takeaway: Shading SDK v1 dependencies is not the same as eliminating SDK v1 API calls.

References: DBR 18.2 release notes | AWS SDK v1 detection guide

datarich
Databricks Employee
Databricks Employee

yesterday

Hi Mathias, good question. A lot of AWS customers are seeing this after their reviews.

When will the v2 migration be done? Databricks hasn't published a date. The DBR 18.2 change you found is the first step. AWS SDK v1 is now shaded inside the runtime, so it's no longer on your classpath. That lets Databricks move its own code to v2 without breaking customer code. The rest of the migration is still in progress.

What this means for your CloudTrail findings Shading moves v1 off your classpath, but the runtime still uses it internally for now. So even on the latest runtime, you'll keep seeing aws-sdk-java/1.x user agents in CloudTrail from Databricks compute until the migration is finished. Changing the DBR version won't make them disappear yet.

Which DBR version to use

  • Use Databricks Runtime 18 LTS or later, or serverless. These include the shading change, so v1 isn't exposed to your code.

  • If your own jobs or libraries still import AWS SDK v1, move them to v2. Or, as a temporary measure, add v1 as an explicit dependency, since the runtime no longer provides it.

For your review

  • Separate your own v1 usage from Databricks'. Check the CloudTrail userAgent and the calling role. Calls from your own code or libraries are the ones you can fix now.

  • Note the rest as a dependency on Databricks, with the 18.2 shading as evidence that the migration is underway.

  • If you need a timeline for your risk register, ask your Databricks account team. They can share updates as they become available.

Docs:

 

0 Kudos
Announcements