Databricks Community

jacovangelder

Hello,Reading the documentation, it does not state it is possible to define libraries on job level instead of on task level. It feels really counter-intuitive putting libraries on task level in Databricks workflows provisioned by Asset Bundles. Is th...

jacovangelder

Couple of things:OPTIMIZE is a very compute intensive operation. Make sure you pick a VM that is compute optimized.I had to look into the AWS instances but it seems the r6g.large you're using is just a 2 CPU 16GB machine. This is by far not sufficien...

jacovangelder

I agree that consuming an event hub is not as straightforward, but it is doable by setting up a kafka stream in Spark. To be honest I find autoloader a bit cumbersome especially for this usecase, but hey if it works it works.

jacovangelder

Have you considered using Databricks Asset Bundles? Very easy to parameterize!

jacovangelder

It depends on what you find costly. I would ask yourself the question if you really need it in a 5 minute interval. If so, then there won't be much difference in pricing leaving a cheap cluster running and streaming it, compared to having a (serverle...

jacovangelder

How fancy do you want to go? You can send ADF diagnostic settings to an event hub and stream them into a delta table in Databricks. Or you can send them to a storage account and build a workflow with 5 minute interval that loads the storage blob into...

Databricks Community

User Stats

User Activity

How do you define PyPi libraries on job level in Asset Bundles?

Re: How to run OPTIMIZE to too big data set which has 11TB and more ?

Re: ADF logs into Databricks

Re: Passing parameters in Databricks workflows

Re: ADF logs into Databricks

Re: ADF logs into Databricks