cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

techinlayers
by • New Contributor III
  • 903 Views
  • 0 replies
  • 2 kudos

How I explain account vs workspace vs Unity Catalog metastore to new platform engineers

Sharing the explanation that finally stopped the same question coming back every onboarding.The account is the organisation-level container, normally one per cloud provider, and it owns billing, account admins and identities. A workspace is a single ...

  • 903 Views
  • 0 replies
  • 2 kudos
kabiromohd
by • New Contributor
  • 831 Views
  • 2 replies
  • 0 kudos

Databricks Free Edition

Hi,I just signed up for Databricks free edition to start my learning journey in Data Engineering on it.Want to know for how long it remains free?Thank you.

  • 831 Views
  • 2 replies
  • 0 kudos
Latest Reply
Ashwin_DSA
Databricks Employee
  • 0 kudos

Hi @kabiromohd,   Welcome to Databricks, and a great choice for starting your Data Engineering journey!   The short answer is that Databricks Free Edition is forever free. There is no trial period or expiration date. You can use it indefinitely as lo...

  • 0 kudos
1 More Replies
emorgoch
by • Contributor
  • 2078 Views
  • 9 replies
  • 2 kudos

Resolved! Using autoloader with multiple object types in load path

My data source is going to generate csv files for multiple objects all into the same directory that I need to load from. The files will have names in the format along the lines of <objecttype>_YYYY_MM_DD_guid.csv.gz. Each objecttype will have it's ow...

  • 2078 Views
  • 9 replies
  • 2 kudos
Latest Reply
stephen4
Contributor
  • 2 kudos

A single Auto Loader stream with filename-based routing looks much cleaner here, especially since new object types can appear without updating the pipeline.

  • 2 kudos
8 More Replies
LiresaFerizaj
by • Contributor
  • 1300 Views
  • 2 replies
  • 2 kudos

Liquid Clustering vs Z-Order vs Partitioning

 Liquid clustering: what it actually replaced, and the caveats worth knowingI see a lot of questions here that boil down to "should I partition, Z-order, or use liquid clustering." Posting how I reason about it, partly to help and partly because I'd ...

  • 1300 Views
  • 2 replies
  • 2 kudos
Latest Reply
aayush_410
Contributor
  • 2 kudos

This lines up well with what Databricks' own docs and the 2026 refresh of their partitioning guidance say — a few things worth adding as confirmation/extra data points rather than pushback:On the key-count cap: confirmed at 4 clustering keys max. Wor...

  • 2 kudos
1 More Replies
seanpmcn
by • New Contributor III
  • 1680 Views
  • 5 replies
  • 5 kudos

Resolved! Issue in "Build a Declarative Pipeline with Spark Declarative Pipelines"

I am trying to complete the "Get Started with Data Engineering" course, but I have been running into an issue.I have gotten to this step in "Build a Declarative Pipeline with Spark Declarative Pipelines":Demo: Create and Run the PipelineNow you'll co...

  • 1680 Views
  • 5 replies
  • 5 kudos
Latest Reply
seanpmcn
New Contributor III
  • 5 kudos

I have added the source now. I went to the root folder > Create pipeline asset > New pipeline source code folder. I was then able to go to the right of the pipeline name at the top of the screen and edit the catalogue and schema.However, after runnin...

  • 5 kudos
4 More Replies
javasquez
by • New Contributor II
  • 591 Views
  • 3 replies
  • 1 kudos

"Run now with different settings” is disabled for Continuous Jobs even when the trigger is paused.

Problem:When a Databricks Job is configured with a Continuous trigger, the “Run now with different settings” option remains disabled even after the Continuous trigger is paused.Limitation:This prevents us from manually running the same job with diffe...

  • 591 Views
  • 3 replies
  • 1 kudos
Latest Reply
javasquez
New Contributor II
  • 1 kudos

 

  • 1 kudos
2 More Replies
ralphchan
by • New Contributor II
  • 7748 Views
  • 7 replies
  • 1 kudos

Connect Oracle Fusion (ERP / HCM) to Databricks

Any suggestion to connect Oracle Fusion (ERP/HCM) to Databricks?I have explored a few options including the use of Oracle Integration Cloud but it requires a lot of customization.

  • 7748 Views
  • 7 replies
  • 1 kudos
Latest Reply
Shankar-Raj
New Contributor II
  • 1 kudos

Check out BI Connector(https://www.biconnector.com/oracle-fusion-data-warehouse-integration/), which offers direct connectivity from Oracle Fusion/FDI/OAC to DW/Lakehouse/Power BI/Tableau and more.   It's widely deployed among Oracle Fusion & Analyti...

  • 1 kudos
6 More Replies
Gianfranco
by • Databricks Partner
  • 3909 Views
  • 5 replies
  • 2 kudos

Deleting Records from DLT Bronze and Silver Tables

I have a pipeline that generates two DLT streaming tables: a Bronze table and a Silver table. I need to delete specific records from both tables. I've read an article (https://www.databricks.com/blog/handling-right-be-forgotten-gdpr-and-ccpa-using-de...

  • 3909 Views
  • 5 replies
  • 2 kudos
Latest Reply
aayush_410
Contributor
  • 2 kudos

The core idea: the reason deleting from Bronze breaks the Silver streaming read isn't Delta itself — it's that Structured Streaming (which DLT streaming tables use under the hood) requires an append-only source by default. Any DELETE/UPDATE on Bronze...

  • 2 kudos
4 More Replies
RaviRocksAI
by • New Contributor
  • 421 Views
  • 2 replies
  • 0 kudos

Training Notebooks/ Access to Lab

Hi, I am looking for training notebooks or access to lab

  • 421 Views
  • 2 replies
  • 0 kudos
Latest Reply
data_pulse
Contributor
  • 0 kudos

Also can check Databricks Academy Labs https://www.databricks.com/databricks-academy-labsDb Demos Tutorials: https://www.databricks.com/resources/demos/tutorials Db Demos cover various use cases by just doing below in any databricks Workspace. Try th...

  • 0 kudos
1 More Replies
jamson
by • New Contributor
  • 34873 Views
  • 3 replies
  • 0 kudos

What are the best practices for optimizing Power BI reports and dashboards for performance in the PL

I’m studying for the PL-300 exam and would love some advice on how to optimize Power BI reports and dashboards for better performance. Specifically, I’m interested in:Techniques for improving report load times and responsiveness.Best practices for ma...

  • 34873 Views
  • 3 replies
  • 0 kudos
Latest Reply
AliceGray
New Contributor II
  • 0 kudos

A few practices can make a significant difference when optimizing Power BI reports and dashboards for performance, especially when Power BI is connected to Databricks or other large data sources.1. Optimize the data model firstUse a simple star schem...

  • 0 kudos
2 More Replies
david_aspegren
by • New Contributor III
  • 1538 Views
  • 7 replies
  • 12 kudos

create_auto_cdc_from_snapshot_flow Python session resolution fails if having multiple snapshot flows

When a pipeline contains more than one create_auto_cdc_from_snapshot_flow flow (each driven by a custom Python next_snapshot_and_version function), flow resolution fails intermittently/consistently with: RuntimeError: The original Spark session is be...

  • 1538 Views
  • 7 replies
  • 12 kudos
Latest Reply
david_aspegren
New Contributor III
  • 12 kudos

Hi again, it seems to have regressed on me, my pipeline is failing again, anybody else seeing this?

  • 12 kudos
6 More Replies
Islam_hoti
by • Contributor
  • 1397 Views
  • 6 replies
  • 10 kudos

Resolved! Photon enabled but a large share of the plan is falling back, cost up and runtime flat

Hi everyone,Trying to work out whether this is expected or whether I have misconfigured something.We enabled Photon on a job cluster running a nightly aggregation over roughly 2TB. The expectation was the usual improvement. What we got instead was ru...

  • 1397 Views
  • 6 replies
  • 10 kudos
Latest Reply
ThiamLee
Contributor III
  • 10 kudos

Great questions! Would love to hear from someone who’s dealt with Photon fallbacks, especially around Python UDFs. Curious to know if rewriting them actually made a noticeable difference in runtime and cost.

  • 10 kudos
5 More Replies
Sen
by • New Contributor
  • 22228 Views
  • 11 replies
  • 2 kudos

Resolved! Performance enhancement while writing dataframes into Parquet tables

Hi,I am trying to write the contents of a dataframe into a parquet table using the command below.df.write.mode("overwrite").format("parquet").saveAsTable("sample_parquet_table")The dataframe contains an extract from one of our source systems, which h...

  • 22228 Views
  • 11 replies
  • 2 kudos
Latest Reply
getcustomessayh
New Contributor II
  • 2 kudos

Writing an effective essay takes more than putting ideas on paper; it requires solid research, logical structure, clear arguments, and careful proofreading. Students working on challenging assignments may find essay writing help useful for organizing...

  • 2 kudos
10 More Replies
HZ
by • New Contributor III
  • 1209 Views
  • 7 replies
  • 4 kudos

Resolved! Compatibility Mode for AWS Redshift

Hello,I have several tables loaded from S3 using Lakeflow pipelines which generates streaming table and materialized views. Want to be able to read these tables in AWS redshift.Based on the Databricks documentation for compatibility mode:Databricks C...

  • 1209 Views
  • 7 replies
  • 4 kudos
Latest Reply
HZ
New Contributor III
  • 4 kudos

Thanks @ivanvyd , from describe extended table, i can see metadata location as s3-compat, detailed table information has streaming table, location s3-uc. In theory, the databricks unity catalog connector with AWS should read metadata from the s3-comp...

  • 4 kudos
6 More Replies
Labels