cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Erik
Valued Contributor II
since ‎09-20-2021
Friday

User Stats

  • 59 Posts
  • 0 Solutions
  • 17 Kudos given
  • 52 Kudos received

User Activity

Feature request: Run cluster in a different region

12-22-2023
We are one of those which unfortunately decided (earlier) to set up our infastructure in one of the sub-par Azure regions, and we happen to dont have any GPU clusters available.We would like to test out some GPU enhanced ML in databricks, but at the ...

Run driver on spot instance

11-18-2023
The traditional advice seems to be to run the driver on "on demand", and optionally the workers on spot. And this is indeed what happends if one chooses to run with spot instances in Databricks. But I am interested in what happens if we run with a dr...

Why not enable "decommissioning" in spark?

11-18-2023
You can enable "decommissioning" in spark, which causes it to remove work from a worker when it gets a notification from the cloud that the instance goes away (e.g. SPOT instances). This is dissabled by default, but it seems like such a no-brainer to...

Hot path event processing and serving in databricks

10-01-2023
We have a setup where we process sensor data in databricks using pyspark structured streaming from kafka streams, and continuisly write these to delta tables. These delta tables are served through a SQL warehouse endpoint to the users. We also store ...

Liquid clustering with structured streaming pyspark

09-17-2023
I would like to try out liquid clustering, but all the examples I see seem to be SQL tables created from selecting from other tables. Our gold tables are pyspark tables written directly to a table, e.g. like this: silver_df.writeStream.partitionBy(["...
View more