Databricks Community

excavator-matt

Hi!I am trying to replicate an AWS RDS PostgreSQL database in Databricks. I have successfully manage to enable CDC using AWS DMS that writes an initial load file and continuous CDC files in parquet.I have been trying to follow the official guide Repl...

excavator-matt · 09-03-2025

In Databricks Runtime 16.4 LTS for Machine Learning, I am used to be able to import sentence-transformers without any installation as it is part of the runtime with from sentence_transformers import SentenceTransformer.In this case I am running on a ...

excavator-matt · 09-03-2025

We're trying to run the bundled sentence-transformers library from SBert in a notebook running Databricks ML 16.4 on an AWS g4dn.2xlarge [T4] instance.However, we're experiencing out of memory crashes and are wondering what the optimal to run sentenc...

excavator-matt · 08-13-2025

Hi!We're trying to use the choropleth visualisation in the new dashboard. This works well for all countries we tried except for district level of Finland.The supported districts are clearly stated in the CSV on the documentation page titled county-di...

excavator-matt

Thank you for your reply!Maybe I am slow, but two things don't sense to me with this reply. If these are clarified, I consider the question resolved.You raise a good point that even if the you use AUTO CDC's sequence_by, you would still have the prob...

excavator-matt

Sorry for the lack of reply. I had to switch tasks, but I hope to be able to test this. You suggest that Pandas On Spark is more efficient implemented than Pandas UDF?

excavator-matt

If you find the author of the official documentation, there you can also send him bonus tasks1. The guide still dead names declarative pipelines dlt, but the modern import is of course "from pyspark import pipelines as dp" with identical syntax2. The...

excavator-matt · 09-08-2025

I am not sure if it is possible to replace sentence-transformers with transformers, but I think it should be possible to import sentence-transformers it is an official part of the runtime. In this case, I was more interested in the resulting vectors ...

excavator-matt · 09-03-2025

I simply run import in in the runtime, so it would it be the runtime default (CUDA 12.6, torch 2.70 and flash_attn 2.7.4.post1). I haven't tried overriding any of these. Not sure if there is a known incompatibility here.

Databricks Community

User Stats

User Activity

How do use Databricks Lakeflow Declarative Pipeline on AWS DMS data?

Importing sentence-transformers no longer works on Databricks runtime 17.2 ML

What is the most efficient way of running sentence-transformers on a Spark DataFrame column?

Strange selection of districts for Finland in choropleth visualisation. Something to reconsider?

Re: How do use Databricks Lakeflow Declarative Pipeline on AWS DMS data?

Re: What is the most efficient way of running sentence-transformers on a Spark DataFrame column?

Re: How do use Databricks Lakeflow Declarative Pipeline on AWS DMS data?

Re: Importing sentence-transformers no longer works on Databricks runtime 17.2 ML

Re: Importing sentence-transformers no longer works on Databricks runtime 17.2 ML