Databricks Community

jacovangelder · 08-14-2024

I'm constantly getting frozen "Cancelling" or "Running" cells since a few days when prototyping some stuff in notebooks. Examples:And the same for instantiating functions (should be instant), neither finishing or able to cancelThis button doesn't wor...

jacovangelder · 06-28-2024

Hello,Reading the documentation, it does not state it is possible to define libraries on job level instead of on task level. It feels really counter-intuitive putting libraries on task level in Databricks workflows provisioned by Asset Bundles. Is th...

jacovangelder · 08-15-2024

Seems to have resolved itself today. Can't reproduce it anymore

jacovangelder · 07-26-2024

Couple of things:OPTIMIZE is a very compute intensive operation. Make sure you pick a VM that is compute optimized.I had to look into the AWS instances but it seems the r6g.large you're using is just a 2 CPU 16GB machine. This is by far not sufficien...

jacovangelder · 07-24-2024

I agree that consuming an event hub is not as straightforward, but it is doable by setting up a kafka stream in Spark. To be honest I find autoloader a bit cumbersome especially for this usecase, but hey if it works it works.

jacovangelder · 07-22-2024

Have you considered using Databricks Asset Bundles? Very easy to parameterize!

jacovangelder · 07-22-2024

It depends on what you find costly. I would ask yourself the question if you really need it in a 5 minute interval. If so, then there won't be much difference in pricing leaving a cheap cluster running and streaming it, compared to having a (serverle...

Databricks Community

User Stats

User Activity

Hanging/frozen cancelling/running cells in notebook

How do you define PyPi libraries on job level in Asset Bundles?

Re: Hanging/frozen cancelling/running cells in notebook

Re: How to run OPTIMIZE to too big data set which has 11TB and more ?

Re: ADF logs into Databricks

Re: Passing parameters in Databricks workflows

Re: ADF logs into Databricks