cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

cristian-vasile
by • New Contributor II
  • 47 Views
  • 0 replies
  • 0 kudos

Hybrid CSV – Self-describing .CSV files with inline schemas and hex-encoded IEEE 754 floats

Anyone who has built ingestion pipelines in Spark/Databricks knows the classic CSV headache: CSV files carry raw values, but zero context. Without an external schema definition, engines are forced to run expensive schema inference—and still risk misi...

  • 47 Views
  • 0 replies
  • 0 kudos
faruk
by • Contributor II
  • 212 Views
  • 2 replies
  • 0 kudos

Lakeflow Connect – Oracle Connector vs Foreign Catalog Connector

Hi everyone,I'm currently evaluating the best approach to ingest large volumes of data from an Oracle database into Databricks.I see two possible approaches:Oracle Connector.Foreign Catalog Connector.I'd like to understand a few things:What are the m...

  • 212 Views
  • 2 replies
  • 0 kudos
Latest Reply
indradbxlearn
New Contributor III
  • 0 kudos

Hi @faruk  - You are talking about two different approaches of data ingestion which are totally different in nature:Lakeflow Connect Oracle connector: a managed ingestion pipeline that copies data into Delta. It is the log-based integrated CDC connec...

  • 0 kudos
1 More Replies
krishgarikipati
by • Databricks Partner
  • 348 Views
  • 4 replies
  • 3 kudos

Delta Tables: Is there any possibility on Delta tables Data file formats other than Parquet

Hi Team,Since Parquet files are immutable and Delta tables framework writes data changes into new data files, Is there any option to specify any other file format and try to use same data file for minor data changes.(Below 10 Record transaction chang...

  • 348 Views
  • 4 replies
  • 3 kudos
Latest Reply
saisaranv
Contributor II
  • 3 kudos

Hi Krishna,Delta Lake tables are tightly coupled with the Parquet file format, so you cannot switch a Delta table to use a different underlying file format while retaining Delta features such as ACID transactions, time travel, and schema enforcement....

  • 3 kudos
3 More Replies
spiralmantra07
by • New Contributor II
  • 565 Views
  • 1 replies
  • 2 kudos

How can Data Fabric streamline real-time and batch data processing for data engineering services?

Data Fabric helps data engineering teams by providing a unified architecture that integrates data across sources, automates pipeline creation, and ensures governance. It enables scalable data pipelines, efficient data warehouse management, and seamle...

  • 565 Views
  • 1 replies
  • 2 kudos
Latest Reply
saisaranv
Contributor II
  • 2 kudos

Well said. One additional consideration is workload predictability and scaling requirements.In my experience, Microsoft Fabric can be a strong choice for organizations with well-understood and relatively stable workloads, where capacity planning can ...

  • 2 kudos
techinlayers
by • New Contributor III
  • 1034 Views
  • 1 replies
  • 3 kudos

How I explain account vs workspace vs Unity Catalog metastore to new platform engineers

Sharing the explanation that finally stopped the same question coming back every onboarding.The account is the organisation-level container, normally one per cloud provider, and it owns billing, account admins and identities. A workspace is a single ...

  • 1034 Views
  • 1 replies
  • 3 kudos
Latest Reply
saisaranv
Contributor II
  • 3 kudos

Excellent explanation. The distinction between account, workspace, and metastore is one of the most common areas of confusion for new Databricks users, and you've broken it down in a very practical way. I especially liked the emphasis on the metastor...

  • 3 kudos
renzwoo
by • New Contributor II
  • 339 Views
  • 1 replies
  • 1 kudos

Built my first Databricks App with Lakebase

Screenshot of my deployed app attached. Thanks for the great quest!#Databricks #Lakebase #AppKit

  • 339 Views
  • 1 replies
  • 1 kudos
Latest Reply
saisaranv
Contributor II
  • 1 kudos

Great work and thanks for sharing! It's always inspiring to see real applications being built with AppKit, Lakebase, and the Databricks ecosystem.A next interesting step would be to expose the application through external APIs and evaluate it from a...

  • 1 kudos
amitsharma1707
by • Databricks Partner
  • 256 Views
  • 1 replies
  • 1 kudos

Structured Streaming: Choose Watermarks Around Real Arrival Delays

Structured Streaming: Choose Watermarks Around Real Arrival DelaysA practical streaming lesson: measure how late business events arrive before choosing a watermark threshold.Consider a hypothetical retailer using Azure Databricks to calculate sales i...

  • 256 Views
  • 1 replies
  • 1 kudos
Latest Reply
saisaranv
Contributor II
  • 1 kudos

In one of my previous implementations, we took a slightly different approach. Rather than relying on watermarks at the ingestion layer, we treated the Bronze layer as a complete system of record and used a blind append pattern to capture every event,...

  • 1 kudos
DevashishK26
by • New Contributor II
  • 207 Views
  • 1 replies
  • 1 kudos

From the Databricks Hackathon to the Databricks Community!

Back in March 2026, I participated in a Databricks hackathon, where I had the opportunity to build a solution using Databricks and explore its capabilities in solving real-world problems.I'm excited to share that my project secured a place among the ...

  • 207 Views
  • 1 replies
  • 1 kudos
Latest Reply
saisaranv
Contributor II
  • 1 kudos

Great one and good to see over a community with your contributions. Welcome to Databricks Community

  • 1 kudos
Denis12345
by • New Contributor
  • 156 Views
  • 2 replies
  • 0 kudos

Dynamic Continuous Data type and new column arrival adopt in silver in DLT

My source is : Oracle db - Golden gate - kafka - Flink - S3 - bronze - silver - gold It's a live cdc data, in S3 we have delta tables in string format, bronze also having string format for all tables, it's a DLT pipeline Now we have to map datatypes ...

  • 156 Views
  • 2 replies
  • 0 kudos
Latest Reply
anuj_lathi
Databricks Employee
  • 0 kudos

Short answer: Drive the silver layer from your metadata snapshot with metaprogramming. Generate the typed SELECT and CDC apply for each table from that metadata, use try_cast so bad values never fail the stream, and decide up front how new columns g...

  • 0 kudos
1 More Replies
elgeo
by • Valued Contributor II
  • 13951 Views
  • 10 replies
  • 12 kudos

Clean up _delta_log files

Hello experts. We are trying to clarify how to clean up the large amount of files that are being accumulated in the _delta_log folder (json, crc and checkpoint files). We went through the related posts in the forum and followed the below:SET spark.da...

  • 13951 Views
  • 10 replies
  • 12 kudos
Latest Reply
stephen4
Contributor
  • 12 kudos

The distinction between `VACUUM` removing obsolete data files and checkpoint creation handling transaction log cleanup is particularly helpful. It also shows why changing retention settings alone may not be enough without new commits and checkpoints....

  • 12 kudos
9 More Replies
MikeGo
by • Valued Contributor
  • 495 Views
  • 9 replies
  • 4 kudos

Zerobus to lakebase

Hi,Is there a plan to make lakebase as destination of Zerobus?Thanks.

  • 495 Views
  • 9 replies
  • 4 kudos
Latest Reply
anuj_lathi
Databricks Employee
  • 4 kudos

Thanks, that clarifies it. My earlier assumption was in the right neighborhood, but LTAP changes one part of what I told you. Your flow of OTel → Kafka → Lakebase, with LTAP handling the analytical side, is a coherent design, and it means you don't ...

  • 4 kudos
8 More Replies
User10
by • New Contributor III
  • 406 Views
  • 5 replies
  • 6 kudos

Resolved! Preserving Salesforce connector SCD2 history across Full Refreshes

We use the managed Lakeflow Connect Salesforce connector with SCD2 history tracking enabled in Bronze, with Silver materialized views serving downstream consumers.Salesforce schema changes are propagated across environments at different times. Removi...

  • 406 Views
  • 5 replies
  • 6 kudos
Latest Reply
User10
New Contributor III
  • 6 kudos

Hi everyone @anuj_lathi, @SumeshKashyap , @niteshm thank you for your responses! I really appreciate the guidance.I'm now on the cusp of deciding whether reversing our SCD layers would be a more maintainable long term approach. It would look like: in...

  • 6 kudos
4 More Replies
ifra_inam
by • New Contributor II
  • 544 Views
  • 4 replies
  • 4 kudos

Databricks Free Edition: AWS S3 Unity Catalog Storage Credential Fails AssumeRole Validation

I'm using Databricks Free Edition and trying to configure an AWS S3 external location through Unity Catalog.My storage credential validation consistently fails at Assume Role, while Self Assume Role and External ID Condition are skipped.Error: Failed...

  • 544 Views
  • 4 replies
  • 4 kudos
Latest Reply
anuj_lathi
Databricks Employee
  • 4 kudos

The error means Databricks' cross-account sts:AssumeRole call is being denied by AWS before any of the later checks run, which is why Self Assume and External ID show as skipped. Everything you've verified is about the IAM role's configuration, so t...

  • 4 kudos
3 More Replies
vamsi_simbus
by • Databricks Partner
  • 255 Views
  • 2 replies
  • 1 kudos

Anyone using Kinaxis Data Publish (Delta Sharing) from Databricks Marketplace?

Hi all,We're looking at Kinaxis Data Publish on the Databricks Marketplace. It shares Kinaxis Maestro planning data into Databricks through Delta Sharing. We requested access a couple of weeks ago and haven't heard back, so I'd like to hear from anyo...

  • 255 Views
  • 2 replies
  • 1 kudos
Latest Reply
anuj_lathi
Databricks Employee
  • 1 kudos

Short version: Kinaxis decides what goes into the share and who gets approved, and from the Databricks side the share is read-only. I can't give you first-hand Kinaxis-specific details, but the platform mechanics answer most of your questions. Data ...

  • 1 kudos
1 More Replies
Labels