Databricks Community

excavator-matt

Hi!We're using Spark Lakeflow Declarative Pipelines for ingesting data from various data sources. However, in order to achieve compliance with GDPR, we are planning to start using ABAC tagging.However, I don't understand how we are supposed to use th...

excavator-matt · 10-06-2025

Hi!I am trying to replicate an AWS RDS PostgreSQL database in Databricks. I have successfully manage to enable CDC using AWS DMS that writes an initial load file and continuous CDC files in parquet.I have been trying to follow the official guide Repl...

excavator-matt · 09-03-2025

In Databricks Runtime 16.4 LTS for Machine Learning, I am used to be able to import sentence-transformers without any installation as it is part of the runtime with from sentence_transformers import SentenceTransformer.In this case I am running on a ...

excavator-matt · 09-03-2025

We're trying to run the bundled sentence-transformers library from SBert in a notebook running Databricks ML 16.4 on an AWS g4dn.2xlarge [T4] instance.However, we're experiencing out of memory crashes and are wondering what the optimal to run sentenc...

excavator-matt · 08-13-2025

Hi!We're trying to use the choropleth visualisation in the new dashboard. This works well for all countries we tried except for district level of Finland.The supported districts are clearly stated in the CSV on the documentation page titled county-di...

excavator-matt

One way to get version control might be to use the Terraform resource entity_tag_assignment. I am not sure if it supports governed_tags, but I'll experiment in the coming weeks.This separates the version control on where the tags are defined and wher...

excavator-matt

Also, upgrading to 17.3 ML still gives the same error.

excavator-matt

I now upgraded to the new 17.3 LTS ML and it now works. I didn't try 17.2 ML, but with 17.3 ML available, I don't see any reason to use it anymore.

excavator-matt

@Louis_Frolio I tried the Pandas on Spark approach.How do I from Delta table into Pandas on Spark DataFrame. Is this the best way? projects_df = spark.read.table("my_catalog.my_schema.my_project_table")projects_spdf = ps.from_pandas(projects_df.toPan...

excavator-matt · 10-27-2025

I discovered that there is a function in Spark to get the schema definition from file.So if you can assume there is a DMS load file, you can steal the schema from there and give it to the change feed view. schema = spark.read.parquet(dms_path + "LOAD...

Databricks Community

User Stats

User Activity

ABAC tag support for for Streaming tables (Spark Lakeflow Declarative Pipelines)?

How do use Databricks Lakeflow Declarative Pipeline on AWS DMS data?

Importing sentence-transformers no longer works on Databricks runtime 17.2 ML

What is the most efficient way of running sentence-transformers on a Spark DataFrame column?

Strange selection of districts for Finland in choropleth visualisation. Something to reconsider?

Re: ABAC tag support for for Streaming tables (Spark Lakeflow Declarative Pipelines)?

Re: What is the most efficient way of running sentence-transformers on a Spark DataFrame column?

Re: Importing sentence-transformers no longer works on Databricks runtime 17.2 ML

Re: What is the most efficient way of running sentence-transformers on a Spark DataFrame column?

Re: How do use Databricks Lakeflow Declarative Pipeline on AWS DMS data?