Databricks Community

Poorva21

Hi everyone,I'm working on optimizing Databricks costs for a production-grade data pipeline (Spark + Delta Lake) on Azure. I’m looking for practical, field-tested strategies to reduce compute and storage spend without impacting performance.So far, I’...

Poorva21

Use the Databricks SQL system users tableSELECT * FROM system.usersOnly shows fully provisioned users Users pending invitation may not appear.

Poorva21

Your expectation is reasonable, but most dashboards are designed to recompute queries per visual. The best workaround is to precompute the base dataset in a table or view, and have all visuals query that instead. This reduces query load and ensures c...

Poorva21

Yes, it’s possible to have a Lakehouse table synced with a Delta table in Unity Catalog. You have a few options:Direct read: Register the Delta table in Unity Catalog and query it directly from your Streamlit app.Delta Live Tables (DLT): Create a DLT...

Poorva21

Managed tables are “owned” by a DLT pipeline. Re-deploying a pipeline that references the same managed tables will fail unless you either:Drop the existing tables firstUse external tables that are not owned by DLTUse a separate development schema/pip...

Poorva21

Auto Optimize = automatically reduces small files during writes. Best for ongoing ETL.OPTIMIZE = manual compaction + Z-ORDER for improving performance on existing data.They are complementary, not competing. Most teams use Auto Optimize for daily inge...

Databricks Community

User Stats

User Activity

Best Practices for Optimizing Databricks Costs in Production Workloads?

Re: Extract all users from Databricks Groups

Re: Databricks Dashboard Optimization

Re: How to create a lakebase table ?

Re: Declarative Pipeline Re-Deployment and existing managed tables exception

Re: What is the difference between OPTIMIZE and Auto Optimize?