cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
Arunsundar
by • New Contributor III
  • 2697 Views
  • 5 replies
  • 4 kudos

The possibility of finding the workload dynamically and spin up the cluster based on the workload

Hi Team,Good morning. I would like to understand if there is a possibility to determine the workload automatically through code (data load from a file to a table, determine the file size, kind of a benchmark that we can check), based on which we can ...

Data Engineering
Reply
  • 2697 Views
  • 5 replies
  • 4 kudos
View Replies
Latest Reply
Kaniz_Fatma
Community Manager
  • 4 kudos

Hi @Arunsundar Muthumanickam​ ​​, We haven't heard from you since the last response from @Vigneshraja Palaniraj​ and @Debayan Mukherjee​​, and I was checking back to see if her suggestions helped you.Or else, If you have any solution, please share it...

  • 4 kudos
4 More Replies
AK032716
by • New Contributor
  • 2883 Views
  • 3 replies
  • 2 kudos

implement autoloader to ingest data into delta lake, i have 100 different tables with full load , append merge senarios

i want to implement autoloader to ingest data into delta lake from 5 different source systems and i have 100 different tables in each database how do we dynamically address this by using autoloader , trigger once option - full load , append merge sen...

Data Engineering
Reply
  • 2883 Views
  • 3 replies
  • 2 kudos
View Replies
Latest Reply
Kaniz_Fatma
Community Manager
  • 2 kudos

Hi @Anil Kovilakar​(Customer)​ , We haven’t heard from you on the last response from @Daniel Sahal​ ​ and @Jordan Fox​, and I was checking back to see if their suggestions helped you. Or else, If you have any solution, please do share that with the c...

  • 2 kudos
2 More Replies
Siddhesh2525
by • New Contributor III
  • 5277 Views
  • 2 replies
  • 6 kudos

How to pass dynamic value in databricks

I have separate column value defined in 13 diffrent notebook and i want merge into 1 databrick notebook and want to pass dynamic parameter using databrick so it will help me to run in single databricks notebook .

Data Engineering
Reply
  • 5277 Views
  • 2 replies
  • 6 kudos
View Replies
Latest Reply
Prabakar
Esteemed Contributor III
  • 6 kudos

Hi @siddhesh Bhavar​ you can use widgets with the %run command to achieve this. https://docs.databricks.com/notebooks/widgets.html#use-widgets-with-run%run /path/to/notebook $X="10" $Y="1"

  • 6 kudos
1 More Replies
max522over
by • New Contributor II
  • 14470 Views
  • 3 replies
  • 0 kudos

Resolved! I've set the partition mode to nonstrict in hive but spark is not seeing it

I've got a table I want to add some data to and it's partitoned. I want to use dynamic partitioning but I get this error org.apache.spark.SparkException: Dynamic partition strict mode requires at least one static partition column. To turn this off ...

Data Engineering
Reply
  • 14470 Views
  • 3 replies
  • 0 kudos
View Replies
Latest Reply
max522over
New Contributor II
  • 0 kudos

I got it working. This was exactly what I needed. Thank you @Peyman Mohajerian​ 

  • 0 kudos
2 More Replies
Labels