Showing results for 
Search instead for 
Did you mean: 
Valued Contributor II
since ‎09-20-2021

User Stats

  • 57 Posts
  • 0 Solutions
  • 16 Kudos given
  • 51 Kudos received

User Activity

We are one of those which unfortunately decided (earlier) to set up our infastructure in one of the sub-par Azure regions, and we happen to dont have any GPU clusters available.We would like to test out some GPU enhanced ML in databricks, but at the ...
The traditional advice seems to be to run the driver on "on demand", and optionally the workers on spot. And this is indeed what happends if one chooses to run with spot instances in Databricks. But I am interested in what happens if we run with a dr...
You can enable "decommissioning" in spark, which causes it to remove work from a worker when it gets a notification from the cloud that the instance goes away (e.g. SPOT instances). This is dissabled by default, but it seems like such a no-brainer to...
We have a setup where we process sensor data in databricks using pyspark structured streaming from kafka streams, and continuisly write these to delta tables. These delta tables are served through a SQL warehouse endpoint to the users. We also store ...
I would like to try out liquid clustering, but all the examples I see seem to be SQL tables created from selecting from other tables. Our gold tables are pyspark tables written directly to a table, e.g. like this: silver_df.writeStream.partitionBy(["...