topic Re: For dummies: How to avoid 'bill shock' & control AWS charges while learning to use Databrick in Data Engineering

For dummies: How to avoid 'bill shock' & control AWS charges while learning to use Databricks?

hantha — Thu, 01 Aug 2024 00:58:03 GMT

Hi, I'm an out-of-work data analyst wanting to re-skill as a 'citizen data engineer'. By following how-to guides I was able to set up my own Databricks account etc along with a personal VPC in AWS. After 2 weeks of problem-free training I checked my month-to-date billing info in AWS. The "NAT Gateway Hour" cost (337 hours!) is the 2nd largest cost item but seems rather unnecessary, given how everything I need is co-located in the same AWS regional data center (Sydney). What would be the likely impact on my Databricks training experience supposing I just delete my NAT gateway? Or is that billing item just included as a notional charge, one of several that will be waived by AWS provided I stay within the 'free tier' limits? (The AWS invoice status is "pending" so I guess I'll find out soon enough 🙂

Re: For dummies: How to avoid 'bill shock' & control AWS charges while learning to use Databrick

holly — Thu, 01 Aug 2024 12:54:44 GMT

Hi Hantha

Databricks needs VPCs to work, but there are the default ones and customer managed ones: https://docs.databricks.com/en/security/network/classic/customer-managed-vpc.html
Customer managed ones are optional, but many tutorials include them as that's standard security for lots of companies. I have made the mistake of this and cost Databricks $1200 🤡
If you start deleting your network pieces, it'll delete things associated with it
Recommendation: take a look at the docs linked. If it looks like you set up a managed one, you'll have to recreate everything with the default in a new Databricks workspace (sorry)
As for the AWS charges, I couldn't guarantee what's part of the free tier and what isn't. I do know their support team is quite good so if you drop them an email they should be able to give you guidance.