cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
subashdsouza
by • New Contributor
  • 734 Views
  • 1 replies
  • 0 kudos

How will Iceberg be integrated into Delta Lake

Data Engineering
Reply
  • 734 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
mhiltner
Databricks Employee
  • 0 kudos

Delta Uniform is the way to go at the moment: https://www.databricks.com/blog/delta-lake-universal-format-uniform-iceberg-compatibility-now-ga In one word, tables are written in Delta but have compatibility with Iceberg as they will also save iceberg...

  • 0 kudos
Peter-M
by • New Contributor II
  • 2335 Views
  • 3 replies
  • 2 kudos

Resolved! Is there a way to have multiple triggers for a single workflow?

Data Engineering
Reply
  • 2335 Views
  • 3 replies
  • 2 kudos
View Replies
Latest Reply
mhiltner
Databricks Employee
  • 2 kudos

You could have a workflow with two tasks, one being a "trigger checker" that could be a super light task scheduled to run every X hours/minutes. This first task would check for your different triggers and define a success criteria for your next task....

  • 2 kudos
2 More Replies
Husainyusuf
by • New Contributor
  • 669 Views
  • 1 replies
  • 0 kudos

DataAIsummit 2024

Having a wonderful learning experience and sharing experience and knowledge with fellow data engineers/scientists/architects

Data Engineering
Reply
  • 669 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Ryangough37
New Contributor II
  • 0 kudos

Interesting use cases so far. Looking forward ot the sessions tomorrow!

  • 0 kudos
301444
by • New Contributor
  • 603 Views
  • 0 replies
  • 0 kudos

dbr

overwhelming and need to learn a lot

Data Engineering
Reply
  • 603 Views
  • 0 replies
  • 0 kudos
Kaviprakash_S
by • New Contributor III
  • 2103 Views
  • 3 replies
  • 1 kudos

SQL compilation error while connecting to snowflake from Databricks

Hi All,I'm trying to connect to the snowflake database from databricks notebook either to read the data or write the data. However I'm getting an weird error. The code and error are provided as follows, snowflake_table = (spark.read  .format("snowfla...

Kaviprakash_S_0-1717986833387.png
Data Engineering
Reply
  • 2103 Views
  • 3 replies
  • 1 kudos
View Replies
Latest Reply
Kaviprakash_S
New Contributor III
  • 1 kudos

@Retired_mod Could you please help with this ? 

  • 1 kudos
2 More Replies
jsperson
by • New Contributor
  • 444 Views
  • 0 replies
  • 0 kudos

Partition Pruning with Non-Delta Files

Howdy - first time caller here.I'm trying to figure out how/if partition pruning works with non-CSV files. I have files landing in bronze in physical partitions of the form dlk_load_dtm=<load date time>. I'd like to load the partitions that don't yet...

Data Engineering
Reply
  • 444 Views
  • 0 replies
  • 0 kudos
heron
by • New Contributor
  • 2247 Views
  • 2 replies
  • 2 kudos

Get metadata information about Delta Table using only SQL Editor (Query)?

I'm trying to obtain the basic information and the storage location of the delta table, but without success.Is there a way to get: storage location, type, catalog, schema, table name using SQL Editor through a query?I can get the basic information (c...

Data Engineering
Reply
  • 2247 Views
  • 2 replies
  • 2 kudos
View Replies
Latest Reply
jacovangelder
Honored Contributor
  • 2 kudos

I believe what you're looking for is DESCRIBE EXTENDED <table_name>.This returns both delta storage location aswell as detailed table information such as type, table properties, catalog, schema, etc. 

  • 2 kudos
1 More Replies
WWoman
by • Contributor
  • 1561 Views
  • 3 replies
  • 1 kudos

Looking for descriptions of action_name column in system.access.audit.. specifically getTable

Hi all,I am looking for description of action_name column in system.access.audit. I am specifically interested in getTable, deleteTable and createTable. I believe the latter 2 are self descriptive but I'd like to confirm.If getTable is related to acc...

Data Engineering
Reply
  • 1561 Views
  • 3 replies
  • 1 kudos
View Replies
Latest Reply
jacovangelder
Honored Contributor
  • 1 kudos

Have you read this? https://docs.databricks.com/en/admin/system-tables/audit-logs.htmlI do agree that is a bit vague. But getTable seems to me when you do a DESCRIBE TABLE <table>, or view table metadata in the UI, so not accessing data in the table,...

  • 1 kudos
2 More Replies
felix_counter
by • New Contributor III
  • 4157 Views
  • 8 replies
  • 5 kudos

Resolved! DBR 14.3 LTS auto-optimizes liquid cluster tables with "clusterBy" empty

Hello,I have a structured stream job writing every 5 mins into a table with liquid clustering enabled. After migrating from DBR 13.3 LTS to DBR 14.3 LTS I observe that the table is newly regularly optimized despite I have not set the "spark.databrick...

Data Engineering
Reply
  • 4157 Views
  • 8 replies
  • 5 kudos
View Replies
Latest Reply
raphaelblg
Databricks Employee
  • 5 kudos

Hello @felix_counter , It seems you're referring to Predictive optimization for Delta Lake, a relatively new feature. In contrast to Optimized writes for Delta Lake on Databricks (basically `spark.databricks.delta.autoCompact.enabled ` and `spark.dat...

  • 5 kudos
7 More Replies
tejas8196
by • New Contributor II
  • 310 Views
  • 0 replies
  • 0 kudos

DAB not updating zone_id when redeployed through GitHub CI/CD

Can anyone please look into https://community.databricks.com/t5/community-discussions/dab-not-updating-zone-id-when-redeployed/td-p/71655  zone_id will update fromauto --> ap-south-1aauto --> ap-south-1bap-south-1a <--> ap-south-1b But not fromap-sou...

Data Engineering
Reply
  • 310 Views
  • 0 replies
  • 0 kudos

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels