cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Engineering & Streaming

Forum Posts

Start a conversation
Husainyusuf
by • New Contributor
  • 274 Views
  • 1 replies
  • 0 kudos

DataAIsummit 2024

Having a wonderful learning experience and sharing experience and knowledge with fellow data engineers/scientists/architects

Data Engineering
Reply
  • 274 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Ryangough37
New Contributor II
  • 0 kudos

Interesting use cases so far. Looking forward ot the sessions tomorrow!

  • 0 kudos
301444
by • New Contributor
  • 527 Views
  • 0 replies
  • 0 kudos

dbr

overwhelming and need to learn a lot

Data Engineering
Reply
  • 527 Views
  • 0 replies
  • 0 kudos
Kaviprakash_S
by • New Contributor III
  • 1473 Views
  • 3 replies
  • 1 kudos

SQL compilation error while connecting to snowflake from Databricks

Hi All,I'm trying to connect to the snowflake database from databricks notebook either to read the data or write the data. However I'm getting an weird error. The code and error are provided as follows, snowflake_table = (spark.read  .format("snowfla...

Kaviprakash_S_0-1717986833387.png
Data Engineering
Reply
  • 1473 Views
  • 3 replies
  • 1 kudos
View Replies
Latest Reply
Kaviprakash_S
New Contributor III
  • 1 kudos

@Retired_mod Could you please help with this ? 

  • 1 kudos
2 More Replies
jsperson
by • New Contributor
  • 333 Views
  • 0 replies
  • 0 kudos

Partition Pruning with Non-Delta Files

Howdy - first time caller here.I'm trying to figure out how/if partition pruning works with non-CSV files. I have files landing in bronze in physical partitions of the form dlk_load_dtm=<load date time>. I'd like to load the partitions that don't yet...

Data Engineering
Reply
  • 333 Views
  • 0 replies
  • 0 kudos
heron
by • New Contributor
  • 1248 Views
  • 2 replies
  • 2 kudos

Get metadata information about Delta Table using only SQL Editor (Query)?

I'm trying to obtain the basic information and the storage location of the delta table, but without success.Is there a way to get: storage location, type, catalog, schema, table name using SQL Editor through a query?I can get the basic information (c...

Data Engineering
Reply
  • 1248 Views
  • 2 replies
  • 2 kudos
View Replies
Latest Reply
jacovangelder
Honored Contributor
  • 2 kudos

I believe what you're looking for is DESCRIBE EXTENDED <table_name>.This returns both delta storage location aswell as detailed table information such as type, table properties, catalog, schema, etc. 

  • 2 kudos
1 More Replies
WWoman
by • New Contributor III
  • 893 Views
  • 3 replies
  • 1 kudos

Looking for descriptions of action_name column in system.access.audit.. specifically getTable

Hi all,I am looking for description of action_name column in system.access.audit. I am specifically interested in getTable, deleteTable and createTable. I believe the latter 2 are self descriptive but I'd like to confirm.If getTable is related to acc...

Data Engineering
Reply
  • 893 Views
  • 3 replies
  • 1 kudos
View Replies
Latest Reply
jacovangelder
Honored Contributor
  • 1 kudos

Have you read this? https://docs.databricks.com/en/admin/system-tables/audit-logs.htmlI do agree that is a bit vague. But getTable seems to me when you do a DESCRIBE TABLE <table>, or view table metadata in the UI, so not accessing data in the table,...

  • 1 kudos
2 More Replies
felix_counter
by • New Contributor III
  • 2755 Views
  • 8 replies
  • 5 kudos

Resolved! DBR 14.3 LTS auto-optimizes liquid cluster tables with "clusterBy" empty

Hello,I have a structured stream job writing every 5 mins into a table with liquid clustering enabled. After migrating from DBR 13.3 LTS to DBR 14.3 LTS I observe that the table is newly regularly optimized despite I have not set the "spark.databrick...

Data Engineering
Reply
  • 2755 Views
  • 8 replies
  • 5 kudos
View Replies
Latest Reply
raphaelblg
Databricks Employee
  • 5 kudos

Hello @felix_counter , It seems you're referring to Predictive optimization for Delta Lake, a relatively new feature. In contrast to Optimized writes for Delta Lake on Databricks (basically `spark.databricks.delta.autoCompact.enabled ` and `spark.dat...

  • 5 kudos
7 More Replies
tejas8196
by • New Contributor II
  • 230 Views
  • 0 replies
  • 0 kudos

DAB not updating zone_id when redeployed through GitHub CI/CD

Can anyone please look into https://community.databricks.com/t5/community-discussions/dab-not-updating-zone-id-when-redeployed/td-p/71655  zone_id will update fromauto --> ap-south-1aauto --> ap-south-1bap-south-1a <--> ap-south-1b But not fromap-sou...

Data Engineering
Reply
  • 230 Views
  • 0 replies
  • 0 kudos
JamesY
by • New Contributor III
  • 632 Views
  • 1 replies
  • 0 kudos

Resolved! Scala how to write data to sqlmi table that has identity col, auto-initialized col DATETIME DEFAULT

Scala, I am trying to write dataframe to sqlmi table, it works for all other table when dataframe columns matches table columns, and I created a new table with the identity and auto-initialized column, from what I understand, by default, I don't need...

Data Engineering
Databricks
Identity column
Scala
SqlMi
Reply
  • 632 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
daniel_sahal
Esteemed Contributor
  • 0 kudos

@JamesY Try adding the missing columns as a dummy columns. I've seen this issue happening some time ago - it looks like it's still not fixed.

  • 0 kudos
gtaspark
by • New Contributor II
  • 50027 Views
  • 9 replies
  • 5 kudos

Resolved! How to get the total directory size using dbutils

Is there a way to get the directory size in ADLS(gen2) using dbutils in databricks? If I run this dbutils.fs.ls("/mnt/abc/xyz") I get the file sizes inside the xyz folder( there are about 5000 files), I want to get the size of the XYZ folder how ca...

Data Engineering
Reply
  • 50027 Views
  • 9 replies
  • 5 kudos
View Replies
Latest Reply
User16788316720
New Contributor III
  • 5 kudos

File size is only specified for files. So, if you specify a directory as your source, you have to iterate through the directory. The below snippet should work (and should be faster than the other solutions).import glob   def get_directory_size_in_byt...

  • 5 kudos
8 More Replies
Sas
by • New Contributor II
  • 1745 Views
  • 3 replies
  • 0 kudos

Resolved! Table is being dropped when cluster terminates in comunity edition

Hi ExpertI have created an external table in databricks community edition. Table is external table. But when i cluster is terminated, i am not able to query the table any more. What is the reason? What i need to do so that table is not dropped. Table...

Data Engineering
Reply
  • 1745 Views
  • 3 replies
  • 0 kudos
View Replies
Latest Reply
venkateshgunda
New Contributor III
  • 0 kudos

Separation of Storage and Compute:Databricks separates storage from compute. Data stored in DBFS (Databricks File System) or external storage systems is persistent and not tied to the lifecycle of a cluster.When you create tables and databases in Dat...

  • 0 kudos
2 More Replies
Frustrated_DE
by • New Contributor III
  • 447 Views
  • 1 replies
  • 1 kudos

Delta live table segregation

Hi,   I've recently been prototyping on Databricks, I was hoping to develop using DLT pipelines in medallion architecture but with isolation of bronze/silver & gold layers in different catalogs in UC for security purposes.At the moment is there a lim...

Data Engineering
Reply
  • 447 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
tyler-xorbix
New Contributor III
  • 1 kudos

Hi @Frustrated_DE ,This seems to be a long-requested feature based on this previous post: Solved: Re: DLT pipeline - Databricks Community - 45740An alternative solution maybe to delegate permissions on the table level for this pipeline specifically. ...

  • 1 kudos

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels