Databricks Community

Islam_hoti

Hi everyone, We are trying to settle on an environment strategy and keep going back and forth, so I would like to hear how other teams actually landed on this rather than what the reference architecture suggests. Our situation. One Databricks account...

Islam_hoti

Hi everyone,We moved most of our workloads to serverless over the last few months and the performance side has been fine. The part we did not plan for is cost attribution.On classic compute this was easy. One cluster per team, tags on the cluster, an...

Islam_hoti

Hi everyone,Unity Catalog managed Iceberg tables went GA earlier this year, and they arrived with the features that used to be the reason to stay on Delta: liquid clustering, predictive optimization, materialized views, streaming tables. On paper the...

Islam_hoti

Hi,Yes, this is a known pattern, and the root cause you describe is correct: Lakeflow Connect ingestion tables are AUTO CDC (SCD Type 1) targets, so they are mutable streaming tables rather than append-only. A normal streaming read against them eithe...

Islam_hoti

Databricks Connect is just a Python package, so you can bypass the extension entirely and keep your pixi setup.Add databricks-connect to your pixi environment, select that interpreter as your kernel in VS Code, and create the session yourself:from d...

Islam_hoti

Hey ,Parquet stores data column by column instead of row by row, so a query only reads the columns it asks for. If your table has 20 columns and the analyst needs just order date and order amount, Parquet reads those two and skips the rest. CSV would...

Islam_hoti

Hey @gowri_databrick ,Think of the checkpoint as a bookmark for your stream.As your query runs, it writes down two things: which data it has already read from the source, and the state it has built up so far, like running counts or aggregations.So wi...

Islam_hoti

Hey, @airbots The index status just mirrors the sync pipeline behind it, so "syncing" forever usually means that pipeline is stuck, not your job. Go to Jobs & Pipelines, find the system-managed pipeline named after your index, and check its event log...

Databricks Community

User Stats

User Activity

How are you separating dev, staging and prod in Unity Catalog without duplicating everything?

How are you attributing serverless costs back to individual jobs and teams?

Now that Unity Catalog manages Iceberg natively, is anyone actually switching?

Re: Incremental Load Issue with SDP

Re: VS Code and Connect - Manual Environment Setup

Re: Understanding Parquet File Storage for Large Datasets

Re: What is a Checkpoint in Structured Streaming?

Re: Vector Store update stale in Syncing status even the actual sync task is done.