AWS Config price raised after change to Job Compute

NickCBZ — Fri, 11 Oct 2024 16:04:56 GMT

I was looking for opportunities to decrease the cost of my Databricks ETLs and, following the documentation, I started to use Job Computes on my ETLs.

Earlier, I used only all-purpose compute to do the ETLs because I needed them to run every 10minutes and the time to start a cluster wouldn't allow me to do so.

So I changed my ETLs to use the all-purpose from 7:00 am to 11:00 pm in workdays and use a job compute on weekend or late night, changing schedule to 1 hour.

After that, my Databricks and my EC2 cost dropped a significantly, but the "config" cost on AWS spiked, going from $0.10 to $3.80 per day. I searched a lot and this simply doesn't make sense, because I only have 3 jobs running with a 1-hour schedule. In AWS Billing, this cost is described as "AWS Config ConfigurationItemRecorded", with thousands of configurations in just 2 days.

How can I fix this?

Re: AWS Config price raised after change to Job Compute

NickCBZ — Thu, 24 Oct 2024 21:39:43 GMT

If someone have this problem in the future, the solution is simple: just disable AWS Config. That's all.

topic AWS Config price raised after change to Job Compute in Data Engineering

AWS Config price raised after change to Job Compute

Re: AWS Config price raised after change to Job Compute