cancel
Turn on suggestions
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
alvaro_databric
by • New Contributor III
  • 2719 Views
  • 1 replies
  • 2 kudos

Resolved! Fastest Azure VM for Databricks Big Data workload

Hi All,It is well known that Azure provides a wide variety of VM for Databricks, some of which provide powerful features such as Photon and Delta Caching. I would like to ask the community which do you think is the fastests cluster for performing Big...

Data Engineering
Reply
  • 2719 Views
  • 1 replies
  • 2 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 2 kudos

@Alvaro Moure​ :The performance of a Databricks cluster for big data operations depends on many factors, such as the amount and structure of the data, the nature of the operations being performed, the configuration of the cluster, and the specific re...

  • 2 kudos
BkP
by • Contributor
  • 8087 Views
  • 14 replies
  • 9 kudos

Suggestion Needed for a Orchestrator/Scheduler to schedule and execute Jobs in an automated way

Hello Friends,We have an application which extracts dat from various tables in Azure Databricks and we extract it to postgres tables (postgres installed on top of Azure VMs). After extraction we apply transformation on those datasets in postgres tabl...

image
Data Engineering
Reply
  • 8087 Views
  • 14 replies
  • 9 kudos
View Replies
Latest Reply
VaibB
Contributor
  • 9 kudos

You can leverage Airflow, which provides a connector for databricks jobs API, or can use databricks workflow to orchestrate your jobs where you can define several tasks and set dependencies accordingly.

  • 9 kudos
13 More Replies
isaac_gritz
by • Databricks Employee
  • 1086 Views
  • 0 replies
  • 3 kudos

Optimize Azure VM / AWS EC2 / GKE Cloud Infrastructure Costs

Tips on Reducing Cloud Compute Infrastructure Costs for Azure VM, AWS EC2, and GCP GKE on DatabricksDatabricks takes advantage of the latest Azure VM / AWS EC2 / GKE VM/instance types to ensure you get the best price performance for your workloads on...

Data Engineering
Reply
  • 1086 Views
  • 0 replies
  • 3 kudos
brickster_2018
by • Databricks Employee
  • 1763 Views
  • 1 replies
  • 0 kudos

How to get Get tag values in Azure VM using metadata endpoint

Data Engineering
Reply
  • 1763 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
brickster_2018
Databricks Employee
  • 0 kudos

curl -H Metadata:true --noproxy "*" "http://169.254.169.254/metadata/instance?api-version=2020-09-01" | jq '.compute.tagsList[] | select(.name=="Creator") | .value'

  • 0 kudos
Labels
Top