cancel
Showing results for 
Search instead for 
Did you mean: 
Databricks Platform Discussions
Dive into comprehensive discussions covering various aspects of the Databricks platform. Join the conversation to deepen your understanding and maximize your usage of the Databricks platform.
cancel
Showing results for 
Search instead for 
Did you mean: 

Browse the Community

Data Engineering

Join discussions on data engineering best practices, architectures, and optimization strategies with...

11794 Posts

Data Governance

Join discussions on data governance practices, compliance, and security within the Databricks Commun...

509 Posts

Generative AI

Explore discussions on generative artificial intelligence techniques and applications within the Dat...

330 Posts

Machine Learning

Dive into the world of machine learning on the Databricks platform. Explore discussions on algorithm...

994 Posts

Warehousing & Analytics

Engage in discussions on data warehousing, analytics, and BI solutions within the Databricks Communi...

660 Posts

Activity in Databricks Platform Discussions

abhijit007
by > New Contributor II
  • 23 Views
  • 1 replies
  • 0 kudos

Databricks Asset Bundles capability for cross cloud migration

Hi everyone,We are planning a migration from Azure Databricks to GCP Databricks and would like to understand whether Databricks Asset Bundles (DAB) can be used to migrate workspace assets such as jobs, pipelines, notebooks, and custom serving endpoin...

  • 23 Views
  • 1 replies
  • 0 kudos
Latest Reply
iyashk-DB
Databricks Employee
  • 0 kudos

DAB's are useful but not sufficient. They work well for re-creating control-plane assets such as jobs, notebooks, DLT/Lakeflow pipelines, and model serving endpoints in a target workspace, even across clouds, by using environment-specific targets and...

  • 0 kudos
Suheb
by > Contributor
  • 25 Views
  • 1 replies
  • 0 kudos

What are the practical differences between bagging and boosting algorithms?

How are bagging and boosting different when you use them in real machine-learning projects?

  • 25 Views
  • 1 replies
  • 0 kudos
Latest Reply
iyashk-DB
Databricks Employee
  • 0 kudos

Bagging and boosting differ mainly in how they reduce error and when you’d choose them: Bagging (e.g., Random Forest) trains many models independently in parallel on different bootstrap samples to reduce variance, making it ideal for unstable, high-v...

  • 0 kudos
piotrsofts
by > New Contributor III
  • 19 Views
  • 1 replies
  • 1 kudos

Data segregated by Domain - how to do that

In the Databricks one demo, and even in official website I seehttps://www.databricks.com/blog/introducing-databricks-one"Domains", how to enable it ? Or add "Domain" to data?

piotrsofts_0-1765990740616.png
  • 19 Views
  • 1 replies
  • 1 kudos
Latest Reply
Hubert-Dudek
Databricks MVP
  • 1 kudos

I think domains are a new concept in Private Preview. If Unity Catalog has to become an enterprise catalog, then they will need to add that concept.

  • 1 kudos
piotrsofts
by > New Contributor III
  • 15 Views
  • 0 replies
  • 0 kudos

Accessing Knowledge Base from Databricks One

Is it possible to use Knowledge Assistant from Databricks one ? 

  • 15 Views
  • 0 replies
  • 0 kudos
RodrigoE
by > Visitor
  • 40 Views
  • 3 replies
  • 0 kudos

Vector search index initialization very slow

Hello,I am creating a vector search index and selected Compute embeddings for a delta table with 19M records.  Delta table has only two  columns: ID (selected as index) and Name (selected for embedding). Embedding model is databricks-gte-large-en.Ind...

Machine Learning
index
search
vector
vector index
Vector Search
  • 40 Views
  • 3 replies
  • 0 kudos
Latest Reply
iyashk-DB
Databricks Employee
  • 0 kudos

@RodrigoE please follow this document - https://docs.databricks.com/aws/en/machine-learning/foundation-model-apis/deploy-prov-throughput-foundation-model-apis#create-your-provisioned-throughput-endpoint-using-the-ui

  • 0 kudos
2 More Replies
seefoods
by > Valued Contributor
  • 20 Views
  • 0 replies
  • 0 kudos

spark conf for serveless jobs

Hello Guys, I use serveless on databricks Azure, so i have build a decorator which instanciate a SparkSession. My job use autolaoder / kafka using mode availableNow. Someone Knows which spark conf is required beacause i want to add it  ? Thanx import...

  • 20 Views
  • 0 replies
  • 0 kudos
seefoods
by > Valued Contributor
  • 169 Views
  • 2 replies
  • 2 kudos

Resolved! setup databricks connect on VsCode and PyCharm

Hello Guyz,Someone Know what's is the best pratices to setup databricks connect for Pycharm and VsCode using Docker, Justfile and .env file Cordially, Seefoods

  • 169 Views
  • 2 replies
  • 2 kudos
Latest Reply
Gecofer
Contributor II
  • 2 kudos

Hi @seefoods!I’ve worked with Databricks Connect and VSCode in different projects, and although your question mentions Docker, Justfile and .env, the “best practices” really depend on what you’re trying to do. Here’s what has worked best for me:1.- D...

  • 2 kudos
1 More Replies
Joost1024
by > Visitor
  • 55 Views
  • 3 replies
  • 0 kudos

Read Array of Arrays of Objects JSON file using Spark

Hi Databricks Community! This is my first post in this forum, so I hope you can forgive me if it's not according to the forum best practices After lots of searching, I decided to share the peculiar issue I'm running into in this community.I try to lo...

  • 55 Views
  • 3 replies
  • 0 kudos
Latest Reply
Joost1024
Visitor
  • 0 kudos

I guess I was a bit over enthusiastic by accepting the answer.When I run the following on the single object array of arrays (as shown in the original post) I get a single row with column "value" and value null. from pyspark.sql import functions as F,...

  • 0 kudos
2 More Replies
rc10000
by > New Contributor
  • 55 Views
  • 2 replies
  • 3 kudos

Resolved! Data Bricks Engineer - DEA Exam vs Training

Hi, I love the Databricks resources but I'm a little confused on what training to take. My focus is studying and practicing for the Databricks Engineer Associate exam, but when I hear of the 'training', I'm not sure which training people are referrin...

  • 55 Views
  • 2 replies
  • 3 kudos
Latest Reply
Advika
Community Manager
  • 3 kudos

Hello @rc10000!+1 to what @Louis_Frolio  mentioned above.The Learning Plan is designed for users preparing for the Databricks Certified Data Engineer Associate and Professional exams. Also below are a few paths, depending on what you’re looking for: ...

  • 3 kudos
1 More Replies
rc10000
by > New Contributor
  • 54 Views
  • 1 replies
  • 1 kudos

Resolved! Lakeflow Connect - Databricks Data Engineer Associate Exam Post-July 2025

Hi, I'm asking another Databricks Data Engineer Associate Exam Dec 2025 question. For those who have taken the DEA exam, is Lakeflow Connect a relevant topic for the test? Been a little confused on what resource to rely on besides the official study ...

  • 54 Views
  • 1 replies
  • 1 kudos
Latest Reply
SP_6721
Honored Contributor II
  • 1 kudos

Hi @rc10000,Lakeflow Connect is mentioned in the exam guide under training, but it’s more about the ingestion concepts. These topics come under the Development & Ingestion section. I’d suggest following the official exam guide first and Databricks Ac...

  • 1 kudos
excavator-matt
by > Contributor
  • 48 Views
  • 0 replies
  • 0 kudos

How do I grant access to find a table in Databricks, without giving access to query the table?

Hi!By default it seems users can only see tables and views in Unity Catalog that they have SELECT permission/privilege on. However, we would like to use Unity Catalog as a data catalog of tables we have. They wouldn't then be able to request access t...

Data Governance
permission
privilege
Unity Catalog
  • 48 Views
  • 0 replies
  • 0 kudos
Richard3
by > New Contributor II
  • 323 Views
  • 6 replies
  • 5 kudos

IDENTIFIER in SQL Views not supported?

Dear community,We are phasing out the dollar param `${catalog_name}` because it has been deprecated since runtime 15.2.We use this parameter in many queries and should now be replaced by the IDENTIFIER clause.In the query below where we retrieve data...

Richard3_0-1765199283388.png Richard3_1-1765199860462.png
  • 323 Views
  • 6 replies
  • 5 kudos
Latest Reply
Hubert-Dudek
Databricks MVP
  • 5 kudos

I have good news: in runtime 18, IDENTIFIER and parameter markers are supported everywhere! We need to wait a month or two as the SQL warehouse and serverless are still on runtime 17.

  • 5 kudos
5 More Replies
s_agarwal
by > Visitor
  • 31 Views
  • 0 replies
  • 0 kudos

Queries from Serverless compute referring to older/deleted/vacuumed version of the delta tables.

Hi Team,I have a unity catalog based managed delta table which I am able to successfully query using the regular compute/cluster options.But when I try to query the same table using a Serverless/SQL Warehouse, they are referring to an older version /...

  • 31 Views
  • 0 replies
  • 0 kudos
RobFer1985
by > New Contributor
  • 144 Views
  • 2 replies
  • 0 kudos

Databricks pipeline fails expectation on execute python script, throws error: Update FAILES

Hi Community,I'm new to Databricks and am trying to make and implement pipeline expectations, The pipelines work without errors and my job works. I've tried multiple ways to implement expectations, sql and python. I keep resolving the errors but end ...

  • 144 Views
  • 2 replies
  • 0 kudos
Latest Reply
carlo968rojer
  • 0 kudos

Hello, @RobFer1985 The primary cause of your error is a circular reference in your logic: you are defining a table named orders_2 while simultaneously trying to readStream from that same table. In Delta Live Tables (DLT), the function acts as the "wr...

  • 0 kudos
1 More Replies
lindsey
by > New Contributor II
  • 2665 Views
  • 1 replies
  • 1 kudos

"Error: cannot read mws credentials: invalid Databricks Account configuration" on TF Destroy

I have a terraform project that creates a workspace in Databricks, assigns it to an existing metastore, then creates external location/storage credential/catalog. The apply works and all expected resources are created. However, without touching any r...

  • 2665 Views
  • 1 replies
  • 1 kudos
Latest Reply
eduardo_287
  • 1 kudos

I have the same problem, were you able to solve it?

  • 1 kudos