cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

mscsu
by New Contributor
  • 620 Views
  • 0 replies
  • 0 kudos

Unity catalog

Great learning on serverless compute, Unity catalog, etc

  • 620 Views
  • 0 replies
  • 0 kudos
Manjula_Ganesap
by Contributor
  • 433 Views
  • 0 replies
  • 0 kudos

Autoloader on ADLS blobs with archival enabled

Hi All, I'm trying to change our Ingestion process to use Autoloader to identify new files landing in a directory on ADLS. The ADLS directory has access tier enabled to archive files older than a certain time period. When I'm trying to set up Autoloa...

  • 433 Views
  • 0 replies
  • 0 kudos
Trilleo
by New Contributor III
  • 3782 Views
  • 4 replies
  • 2 kudos

Resolved! Handle updates from bronze to silver table stream

Hi Databricks Community,  I am trying to stream from a bronze to a silver table, however, I have the problem that there may be updates in the bronze table. Delta table streaming reads and write does not support skipChangeCommits=false, i.e. handle mo...

  • 3782 Views
  • 4 replies
  • 2 kudos
Latest Reply
Himali_K
New Contributor II
  • 2 kudos

Hi, You can use dlt apply changes to deal with changing source.Delta Live Tables Python language reference | Databricks on AWSThank you

  • 2 kudos
3 More Replies
aliehs0510
by New Contributor II
  • 535 Views
  • 1 replies
  • 1 kudos

DLT Pipeline does not create the view but it shows up on the DLT graph

I wanted a more filtered data set from a materialized view so I figured a view might be the solution but it doesn't get created under the target schema  however it shows upon in the graph as a part of the pipeline. Can't we use MVs as data source for...

  • 535 Views
  • 1 replies
  • 1 kudos
Latest Reply
Rishabh-Pandey
Esteemed Contributor
  • 1 kudos

Issue at Hand:You mentioned that a view is not created under the target schema but appears in the DLT graph. This situation arises due to how DLT manages views and materialized views.Possible Causes and Solutions:DLT Execution and Target Schema:In DL...

  • 1 kudos
sanket-kelkar
by New Contributor II
  • 6635 Views
  • 4 replies
  • 1 kudos

Databricks costing - Need details of the Azure VM costs

Hi All,We are using the Azure Databricks platform for one of our Data Engg needs. Here's my setup -1. Job compute that uses Cluster of size - 1 driver and 2 workers - all are of 'Standard_DS3_v2' type. (Photon is disabled).2. The job compute takes th...

  • 6635 Views
  • 4 replies
  • 1 kudos
Latest Reply
GuillermoM
New Contributor II
  • 1 kudos

To calculate the real cost of an Azure Cluster or Job, there are two ways: DIY, which means querying the Microsoft Cost API and Databricks API and then combining the information to get the exact cost, or you can use a tool such as KopiCloud Databrick...

  • 1 kudos
3 More Replies
CAN
by New Contributor
  • 1112 Views
  • 1 replies
  • 0 kudos

Security Threats in Databricks for File Upload

Dear community, we are using the Azure Databricks service and wondering if uploading a file to the DBFS (or to a storage accessed directly from a notebook in Databricks) could be a potential security threat. Imagine you upload some files with 'malici...

  • 1112 Views
  • 1 replies
  • 0 kudos
Latest Reply
KrunalMedapara
New Contributor II
  • 0 kudos

Uploading a file to the Databricks File System (DBFS) or accessing storage directly from a notebook in Azure Databricks could pose potential security risks if not managed properly. Here are some considerations:Sensitive Data Exposure: Uploading sensi...

  • 0 kudos
User16790091296
by Contributor II
  • 1040 Views
  • 1 replies
  • 0 kudos

How to efficiently read the data lake files' metadata?

I want to read the last modified datetime of the files in data lake in a databricks script. If I could read it efficiently as a column when reading data from data lake, it would be perfect.Thank you:)

  • 1040 Views
  • 1 replies
  • 0 kudos
Latest Reply
KrunalMedapara
New Contributor II
  • 0 kudos

Efficiently reading data lake files involves:Choosing the Right Tools: Select tools optimized for data lake file formats (e.g., Parquet, ORC) and distributed computing frameworks (e.g., Apache Spark, Apache Flink).Partitioning and Indexing: Partition...

  • 0 kudos
Tuan
by New Contributor
  • 308 Views
  • 0 replies
  • 0 kudos

Vector Search

Do you guys think vector search will be cheaper and perform better than Azure AI Search?

  • 308 Views
  • 0 replies
  • 0 kudos
Johnc
by New Contributor
  • 331 Views
  • 0 replies
  • 0 kudos

Connecting to SAP ECC

I'm looking to meet with others who have successfully connected databricks to SAP ECC.

  • 331 Views
  • 0 replies
  • 0 kudos
MaOthman2110
by New Contributor
  • 1413 Views
  • 1 replies
  • 0 kudos

can't login back to databricks community

Hello,I'm experiencing difficulty logging into the Databricks community despite using the correct username and password. Additionally, when attempting to reset my password, I haven't received any email notifications.

  • 1413 Views
  • 1 replies
  • 0 kudos
Latest Reply
Aurelius
New Contributor II
  • 0 kudos

I am facing the same issue.

  • 0 kudos

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels