Databricks Community

Alena · 08-05-2025

I’m running an ingestion pipeline with a Databricks job:A file lands in S3A Lambda is triggeredThe Lambda runs a Databricks jobThe incoming files vary a lot in size, which makes processing times vary as well. My job cluster has autoscaling enabled, b...

Alena · 08-05-2025

Thank you so much for your help—it works! One thing I’m trying to do is authenticate hadoop-opensearch using a different role than the one my cluster is mapped to. Environment variables only seem to work if they’re set in the cluster configuration. I...

Alena · 07-25-2025

Thank you for posting this. What did you use to authenticate? I keep getting the error below. I tried passing env vars - did not work. EnvironmentVariableCredentialsProvider: Unable to load AWS credentials from environment variables (AWS_ACCESS_KEY_...

Databricks Community

User Stats

User Activity

Programmatically set minimum workers for a job cluster based on file size?

Re: Write Spark DataFrame into OpenSearch

Re: Write Spark DataFrame into OpenSearch