Databricks

Erik

I would like to try out liquid clustering, but all the examples I see seem to be SQL tables created from selecting from other tables. Our gold tables are pyspark tables written directly to a table, e.g. like this: silver_df.writeStream.partitionBy(["...

Erik · 05-19-2023

We are using the "databricks_notebook" terraform resource to deploy our notebooks into the "Workspace" as part of our CICD run, and our jobs run notebooks from the workspace. For development we clone the repo into "Repos". At the moment the only modu...

Erik · 09-30-2022

We are distributing pbids files providing the connection info to databricks. It contains options passed to the "Databricks.Catalogs " function implementing the connection to databricks. It is my understanding that databricks has made this together wi...

Erik · 08-13-2022

As many of you, we have implemented a "medallion architecture" (raw/bronze/silver/gold layers), which are each stored on seperate storrage accounts. We only create proper hive tables of the gold layer tables, so our powerbi users connecting to the da...

Erik · 08-12-2022

It is our understanding that enabling unity catalog means that some metadata (for example usernames) will be sent to the USA for processing/storage. This is unfortunately a deal-breaker for us, and we need the data to reside solely in Europe for comp...

Erik

Here is the solution I ended up with. Any suggestions for improvements are welcome!I created a helper-function:import logging import uuid from typing import Dict logging.basicConfig() LOGGER = logging.getLogger(__name__) from pyspark.sql import Spar...

Erik

@KanizThanks for your reply. I know I can't use it with partitioning or zorder, thats ok. What I wonder about is if its possible to create a liquid clustering delta table using the pyspark writeStream API. For example like something like this:silver_...

Erik · 08-25-2022

No, there are no free ways yet.With the newly released databricks-sql-connector for golang it should be quite easy for anyone who known golang to create a grafana backend datasource plugin, so I hope someone does 

Erik · 08-19-2022

Thanks for the reply @Sivaprasad C S ! Is it possible to get a bit more information? Like, is it even planned to have unity in Europe at *some* point in time, or is it set up to always be US-only?

Erik · 07-05-2022

So, just to update people, since things have changed a bit! The good news is that Grafana has released a databricks data-connector! The bad news is that its "enterprise only". I have played a bit with it, and it works quite well. Here is an example s...

Databricks

User Stats

User Activity

Liquid clustering with structured streaming pyspark

Create python modules for both repos and workspace

Can we have the powerbi connector step into "hive_metastore" automatically?

How to combine medallion architecture and delta live-tables nicely?

Will Unity catalog come to Europe?

Re: Liquid clustering with structured streaming pyspark

Re: Liquid clustering with structured streaming pyspark

Re: Grafana + databricks = True?

Re: Will Unity catalog come to Europe?

Re: Grafana + databricks = True?