For dummies: How to avoid 'bill shock' & control AWS charges while learning to use Databricks?

hantha
New Contributor

‎07-31-2024 05:58 PM

Hi, I'm an out-of-work data analyst wanting to re-skill as a 'citizen data engineer'.  By following how-to guides I was able to set up my own Databricks account etc along with a personal VPC in AWS. After 2 weeks of problem-free training I checked my month-to-date billing info in AWS. The "NAT Gateway Hour" cost (337 hours!) is the 2nd largest cost item but seems rather unnecessary, given how everything I need is co-located in the same AWS regional data center (Sydney). What would be the likely impact on my Databricks training experience supposing I just delete my NAT gateway? Or is that billing item just included as a notional charge, one of several that will be waived by AWS provided I stay within the 'free tier' limits? (The AWS invoice status is "pending" so I guess I'll find out soon enough 🙂  

0 Kudos

holly
Databricks Employee
Databricks Employee

‎08-01-2024 05:54 AM

Hi Hantha

  • Databricks needs VPCs to work, but there are the default ones and customer managed ones: https://docs.databricks.com/en/security/network/classic/customer-managed-vpc.html
  • Customer managed ones are optional, but many tutorials include them as that's standard security for lots of companies. I have made the mistake of this and cost Databricks $1200 🤡
  • If you start deleting your network pieces, it'll delete things associated with it 
  • Recommendation: take a look at the docs linked. If it looks like you set up a managed one, you'll have to recreate everything with the default in a new Databricks workspace (sorry)
  • As for the AWS charges, I couldn't guarantee what's part of the free tier and what isn't. I do know their support team is quite good so if you drop them an email they should be able to give you guidance.