cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

brendan-mahler
by New Contributor
  • 1433 Views
  • 2 replies
  • 0 kudos

Databricka 2023

What is your favorite tool for monitoring your pipelines?

  • 1433 Views
  • 2 replies
  • 0 kudos
Latest Reply
kknebel
New Contributor II
  • 0 kudos

There are a variety of third party apps that can help monitor your data and pipelines. Some big ones include calibra and bigeye

  • 0 kudos
1 More Replies
dholmgreen
by New Contributor
  • 1537 Views
  • 0 replies
  • 0 kudos

Unity Catalog ABAC

Is there a roadmap for when ABAC (Attribute-based access control) wilö be available in Unity Catalog?

  • 1537 Views
  • 0 replies
  • 0 kudos
Mac3
by New Contributor II
  • 1023 Views
  • 1 replies
  • 1 kudos

Data Engineering

What is a common problem you face when ingesting data in hot storage?

  • 1023 Views
  • 1 replies
  • 1 kudos
Latest Reply
Seb
New Contributor II
  • 1 kudos

Finding the best partitioning strategy and also cost issue if you ingest huge amount of data

  • 1 kudos
DebanjanB
by New Contributor
  • 615 Views
  • 0 replies
  • 0 kudos

Autloader processing semantics

Queue based Autoloader processes files in the order they are received only when the job is up and running. However when the job is down, the files that queue up are processed in lexical order once the job is up. Since autoloader jobs need to be stopp...

  • 615 Views
  • 0 replies
  • 0 kudos
whleeman
by New Contributor III
  • 776 Views
  • 0 replies
  • 0 kudos

Does Databricks get cached result for a subquery?

If I run a query as "SELECT fare_amount FROM nyctaxi.trips where fare_amount > 1.5".  The query results will be cached for 24 hours.I then compose a second query using the previous query as a subquery "SELECT * FROM nyctaxi.trips WHERE fare_amount IN...

  • 776 Views
  • 0 replies
  • 0 kudos
Tidaldata
by New Contributor
  • 866 Views
  • 0 replies
  • 0 kudos

Loveing Databricks Summit

Loving the summit so far, awesome keynote speakers, great trainers and paid courses. Finished certification #databrickslearning

  • 866 Views
  • 0 replies
  • 0 kudos
JohanBringsdal
by New Contributor
  • 1093 Views
  • 0 replies
  • 0 kudos

Migrating old solution to new optimal delta lake setup

Hi Databricks community!I have previsouly worked on a project that easily could be optimized with Databricks. It is currently running on Azure Synapse, but the premise is the same.Ill describe the scenario here:1. Data owners send a constant flow of ...

  • 1093 Views
  • 0 replies
  • 0 kudos
christo_M
by New Contributor
  • 1321 Views
  • 1 replies
  • 0 kudos

Data Governance

How can I propagate a deletion to all tables where a customer requests to be removed from the database as part of the GDPR compliance ?  

  • 1321 Views
  • 1 replies
  • 0 kudos
Latest Reply
Datajoe
Contributor
  • 0 kudos

We use a python script that enables and removes access to tables based on role-group, but can be user as well. Also have a script that removes all access- can be executes in seconds.

  • 0 kudos
teja_7
by New Contributor
  • 881 Views
  • 0 replies
  • 0 kudos

DataSummit 2023

Happy to be part of the data summit 2023. Wondering if DBSQL is enabled for DLT tables

  • 881 Views
  • 0 replies
  • 0 kudos

Join Us as a Local Community Builder!

Passionate about hosting events and connecting people? Help us grow a vibrant local community—sign up today to get started!

Sign Up Now
Labels