Thanks for your follow-up! That’s a really good and fair question — especially for folks coming from traditional warehouses or on-prem environments.

In most projects I’ve worked on, it’s true that the platform or infra team is responsible for cost control, monitoring usage patterns, and setting alerts if a specific team or job suddenly spikes in consumption.

However, as a developer, DE or ML engineer, I personally still try to stay aware of my own resource usage, even if I don’t always know the exact €/$ cost of a given job or training run. Why? Because in cloud environments:

• Everything is elastic and on-demand, which is great for flexibility but it also means every compute second and every GB of memory has a price tag attached.

So my own rule of thumb is:
Always start with the smallest cluster possible, and only scale up if the code/job really needs it.

I avoid using the largest clusters "just in case" — it’s better to monitor memory usage, optimize logic, and grow gradually if needed.

Databricks also uses its own cost unit called DBUs (Databricks Units), which makes it easier to track costs in a normalized way across workloads — but keep in mind that the actual machines are running on your cloud provider (Azure, AWS, GCP). Also, serverless compute in Databricks can be great for certain use cases, but it’s not always cheaper than classic clusters — for example, if your job runs for a long time or needs tight control over the environment, classic compute may be more cost-effective.

So yes — ideally, the infra team should own cost governance. But from my side, I try to:

• Be cost-aware when designing pipelines or training model.
• Optimize Spark code (e.g. caching, avoiding wide transformations, etc.)
• Choose the right compute layer (classic vs serverless vs job clusters) depending on the workload

Cloud is a shift in mindset — it gives you power and flexibility, but also means everyone plays a small part in cost efficiency, especially at scale.

Gema 👩‍💻