cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
Datanoob123
by • Visitor
  • 28 Views
  • 2 replies
  • 0 kudos

Query to show column names in common between multiple tables

Hi all, I have a large amount of tables that I would like a query to pull the column names present in these tables that are common between all the tables.  I know about show columns, but can't seem to use this or another method to achieve this. This ...

Data Engineering
comparing tables
show columns
sql
Reply
  • 28 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
KaranamS
Contributor
  • 0 kudos

Hi @Datanoob123 ,You can use information_schema in system tables to get the details about columns and tables in Databricks. Here is a sample code that gives you the column names and number of tables with same column name.SELECT column_name, COUNT(DIS...

  • 0 kudos
1 More Replies
HarryRichard08
by • New Contributor
  • 23 Views
  • 1 replies
  • 0 kudos

Unable to Access S3 from Serverless but Works on Cluster

Hi everyone,I am trying to access data from S3 using an access key and secret. When I run the code through Databricks clusters, it works fine. However, when I try to do the same from a serverless cluster , I am unable to access the data.I have alread...

Data Engineering
Reply
  • 23 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
KaranamS
Contributor
  • 0 kudos

Hi @HarryRichard08 , Can you please provide the sample code you are using? And the error you get with serverless?

  • 0 kudos
Kayla
by • Valued Contributor II
  • 132 Views
  • 4 replies
  • 2 kudos

Unity Catalog "Sync" Question

I'm having a little trouble fully following the documentation on the SYNC command.I have a table in hive_metastore that still needs to be able to be updated daily for the next few months, but I also need to define a view in Unity Catalog based on tha...

Data Engineering
Reply
  • 132 Views
  • 4 replies
  • 2 kudos
View Replies
Latest Reply
Nivethan_Venkat
New Contributor III
  • 2 kudos

Hi @Kayla,SYNC command is to sync your hive EXTERNAL table to your Unity Catalog name space. If the table is external, the UC table will be in sync with your external location. If it is hive managed table, you can't use SYNC command to have your mana...

  • 2 kudos
3 More Replies
the_dude
by • New Contributor II
  • 84 Views
  • 1 replies
  • 0 kudos

Impossibility to have multiple versions of the same Python package installed

Hello, We package our Spark jobs + utilities in a custom package to be used in wheel tasks in Databricks. In my opinion, having several versions of this job (say "production" and "dev") run on the same cluster against different versions of this custo...

Data Engineering
Reply
  • 84 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
the_dude
New Contributor II
  • 0 kudos

If someone comes across this post - as per documentation, library/package installation can be Notebook-scoped. Thus, in order to overcome the limitation described in the initial post instead we are experimenting with Notebook tasks whose only respons...

  • 0 kudos
Phani1
by • Valued Contributor II
  • 35 Views
  • 1 replies
  • 0 kudos

Reading Multiple Data Formats

 Hi All, I'm looking to develop generic code that can read multiple data formats, such as Parquet, Delta, Iceberg and save it as delta. Can you provide some insights or guidance on how to achieve this?Regards,Phani

Data Engineering
Reply
  • 35 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Erika_Fonseca
Databricks Employee
  • 0 kudos

Take a look at these 2 projects that follow a metadata-driven approach: Lakehouse Engine DLT Meta

  • 0 kudos
zmsoft
by • New Contributor III
  • 262 Views
  • 1 replies
  • 0 kudos

How to copy file from UC volume to external location folder

Hi there, How to copy file from UC volume to external location folder Thanks&Regards, zmsoft

Data Engineering
Reply
  • 262 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Advika
Databricks Employee
  • 0 kudos

Hello @zmsoft! To copy a file from a UC volume to an external location, you can use:   dbutils.fs.cp( "UC_volume_path", "external_location_path" )   Ensure the external location is preconfigured in Unity Catalog and you have the necessary permission...

  • 0 kudos
Eduard
by • New Contributor II
  • 102104 Views
  • 3 replies
  • 1 kudos

Cluster xxxxxxx was terminated during the run.

Hello,I have a problem with the autoscaling of a cluster. Every time the autoscaling is activated I get this error. Does anyone have any idea why this could be?"Cluster xxxxxxx was terminated during the run (cluster state message: Lost communication ...

Data Engineering
Reply
  • 102104 Views
  • 3 replies
  • 1 kudos
View Replies
Latest Reply
louisgarza
New Contributor
  • 1 kudos

Hello Databricks Community,The error message indicates that the driver node was lost, which can happen due to network issues or malfunctioning instances. Here are a few possible reasons and solutions:Instance Instability: If your cloud provider has u...

  • 1 kudos
2 More Replies
Fatimah-Tariq
by • New Contributor III
  • 83 Views
  • 4 replies
  • 0 kudos

Schema update Issue in DLT

I have a pipeline in databricks with this flowSQL SERVER (Source) -> Staging (Parquet) -> Bronze (DLT) -> Silver(DLT) -> Gold (DLT)The pipeline is up and running smoothly for months but recently, there was a schema update at my source level and one o...

Data Engineering
Reply
  • 83 Views
  • 4 replies
  • 0 kudos
View Replies
Latest Reply
Fatimah-Tariq
New Contributor III
  • 0 kudos

Hi @Alberto_Umana, is there any word on how to fix my data and bring all the records back to the pipeline schema?

  • 0 kudos
3 More Replies
ShivangiB
by • New Contributor
  • 165 Views
  • 2 replies
  • 0 kudos

Liquid Clustering limitation clustering on write does not support source queries that include filter

I have a query :%sqlinsert into ucdata.brz.liquidcluster_table_data select sum(col1) as col1,col2,sum(col3) as col3 from ucdata.brz.liquidcluster_table_data  group by col2This query I am running with run time version 13.3 and it is still working. But...

Data Engineering
Reply
  • 165 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
ShivangiB
New Contributor
  • 0 kudos

Hey @SantoshJoshi , I am just trying too test the limitations specified for some case study. And while testing can't see these limitations so wanted to understand.it is having 3 columns , col1,col2,col3 having random numbers and col2 is cluster by co...

  • 0 kudos
1 More Replies
mjar
by • New Contributor III
  • 3847 Views
  • 9 replies
  • 3 kudos

ModuleNotFoundError when using foreachBatch on runtime 14 with Unity

Recently we have run into an issue using foreachBatch after upgrading our Databricks cluster on Azure to a runtime version 14 with Spark 3.5 with Shared access mode and Unity catalogue.The issue was manifested by ModuleNotFoundError error being throw...

Data Engineering
Reply
  • 3847 Views
  • 9 replies
  • 3 kudos
View Replies
Latest Reply
Abond
New Contributor
  • 3 kudos

Hi, Any news regarding that issue? I have the same one on job cluster with 15.4 LTS when using asset bundles with foreachBatch  in .py file and call it from notebook. When the same code is located in notebook - it works file.(prep_silver_df(bronze_ta...

  • 3 kudos
8 More Replies
cpayne_vax
by • New Contributor III
  • 17953 Views
  • 15 replies
  • 9 kudos

Resolved! Delta Live Tables: dynamic schema

Does anyone know if there's a way to specify an alternate Unity schema in a DLT workflow using the @Dlt.table syntax? In my case, I’m looping through folders in Azure datalake storage to ingest data. I’d like those folders to get created in different...

Data Engineering
Reply
  • 17953 Views
  • 15 replies
  • 9 kudos
View Replies
Latest Reply
abhishek_02
New Contributor
  • 9 kudos

Hi @kuldeep-in, Could you please provide the exact location how to disable DPM enabled option as i was not able to locate it in pipeline settings or Databricks settings.Thank you

  • 9 kudos
14 More Replies

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels