I was looking for opportunities to decrease the cost of my Databricks ETLs and, following the documentation, I started to use Job Computes on my ETLs.
Earlier, I used only all-purpose compute to do the ETLs because I needed them to run every 10minutes and the time to start a cluster wouldn't allow me to do so.
So I changed my ETLs to use the all-purpose from 7:00 am to 11:00 pm in workdays and use a job compute on weekend or late night, changing schedule to 1 hour.
After that, my Databricks and my EC2 cost dropped a significantly, but the "config" cost on AWS spiked, going from $0.10 to $3.80 per day. I searched a lot and this simply doesn't make sense, because I only have 3 jobs running with a 1-hour schedule. In AWS Billing, this cost is described as "AWS Config ConfigurationItemRecorded", with thousands of configurations in just 2 days.
How can I fix this?
