cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
mscsu
by • New Contributor
  • 867 Views
  • 0 replies
  • 0 kudos

Unity catalog

Great learning on serverless compute, Unity catalog, etc

Data Engineering
Reply
  • 867 Views
  • 0 replies
  • 0 kudos
Trilleo
by • New Contributor III
  • 8683 Views
  • 4 replies
  • 2 kudos

Resolved! Handle updates from bronze to silver table stream

Hi Databricks Community,  I am trying to stream from a bronze to a silver table, however, I have the problem that there may be updates in the bronze table. Delta table streaming reads and write does not support skipChangeCommits=false, i.e. handle mo...

Data Engineering
Reply
  • 8683 Views
  • 4 replies
  • 2 kudos
View Replies
Latest Reply
Himali_K
New Contributor II
  • 2 kudos

Hi, You can use dlt apply changes to deal with changing source.Delta Live Tables Python language reference | Databricks on AWSThank you

  • 2 kudos
3 More Replies
aliehs0510
by • New Contributor II
  • 1348 Views
  • 1 replies
  • 1 kudos

DLT Pipeline does not create the view but it shows up on the DLT graph

I wanted a more filtered data set from a materialized view so I figured a view might be the solution but it doesn't get created under the target schema  however it shows upon in the graph as a part of the pipeline. Can't we use MVs as data source for...

Data Engineering
Reply
  • 1348 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
Rishabh-Pandey
Databricks MVP
  • 1 kudos

Issue at Hand:You mentioned that a view is not created under the target schema but appears in the DLT graph. This situation arises due to how DLT manages views and materialized views.Possible Causes and Solutions:DLT Execution and Target Schema:In DL...

  • 1 kudos
sanket-kelkar
by • New Contributor II
  • 10962 Views
  • 4 replies
  • 1 kudos

Databricks costing - Need details of the Azure VM costs

Hi All,We are using the Azure Databricks platform for one of our Data Engg needs. Here's my setup -1. Job compute that uses Cluster of size - 1 driver and 2 workers - all are of 'Standard_DS3_v2' type. (Photon is disabled).2. The job compute takes th...

Data Engineering
Reply
  • 10962 Views
  • 4 replies
  • 1 kudos
View Replies
Latest Reply
GuillermoM
New Contributor II
  • 1 kudos

To calculate the real cost of an Azure Cluster or Job, there are two ways: DIY, which means querying the Microsoft Cost API and Databricks API and then combining the information to get the exact cost, or you can use a tool such as KopiCloud Databrick...

  • 1 kudos
3 More Replies
CAN
by • New Contributor
  • 2944 Views
  • 1 replies
  • 0 kudos

Security Threats in Databricks for File Upload

Dear community, we are using the Azure Databricks service and wondering if uploading a file to the DBFS (or to a storage accessed directly from a notebook in Databricks) could be a potential security threat. Imagine you upload some files with 'malici...

Data Engineering
Reply
  • 2944 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
KrunalMedapara
New Contributor II
  • 0 kudos

Uploading a file to the Databricks File System (DBFS) or accessing storage directly from a notebook in Azure Databricks could pose potential security risks if not managed properly. Here are some considerations:Sensitive Data Exposure: Uploading sensi...

  • 0 kudos
User16790091296
by • Databricks Employee
  • 1832 Views
  • 1 replies
  • 0 kudos

How to efficiently read the data lake files' metadata?

I want to read the last modified datetime of the files in data lake in a databricks script. If I could read it efficiently as a column when reading data from data lake, it would be perfect.Thank you:)

Data Engineering
Reply
  • 1832 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
KrunalMedapara
New Contributor II
  • 0 kudos

Efficiently reading data lake files involves:Choosing the Right Tools: Select tools optimized for data lake file formats (e.g., Parquet, ORC) and distributed computing frameworks (e.g., Apache Spark, Apache Flink).Partitioning and Indexing: Partition...

  • 0 kudos
Tuan
by • New Contributor
  • 669 Views
  • 0 replies
  • 0 kudos

Vector Search

Do you guys think vector search will be cheaper and perform better than Azure AI Search?

Data Engineering
Reply
  • 669 Views
  • 0 replies
  • 0 kudos
Johnc
by • New Contributor
  • 769 Views
  • 0 replies
  • 0 kudos

Connecting to SAP ECC

I'm looking to meet with others who have successfully connected databricks to SAP ECC.

Data Engineering
Reply
  • 769 Views
  • 0 replies
  • 0 kudos
MaOthman2110
by • New Contributor
  • 1919 Views
  • 1 replies
  • 0 kudos

can't login back to databricks community

Hello,I'm experiencing difficulty logging into the Databricks community despite using the correct username and password. Additionally, when attempting to reset my password, I haven't received any email notifications.

Data Engineering
Reply
  • 1919 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Aurelius
New Contributor II
  • 0 kudos

I am facing the same issue.

  • 0 kudos
Labels