Databricks Community

excavator-matt

Hi!By default it seems users can only see tables and views in Unity Catalog that they have SELECT permission/privilege on. However, we would like to use Unity Catalog as a data catalog of tables we have. They wouldn't then be able to request access t...

excavator-matt

Hi!We're using Spark Lakeflow Declarative Pipelines for ingesting data from various data sources. However, in order to achieve compliance with GDPR, we are planning to start using ABAC tagging.However, I don't understand how we are supposed to use th...

excavator-matt · 10-06-2025

Hi!I am trying to replicate an AWS RDS PostgreSQL database in Databricks. I have successfully manage to enable CDC using AWS DMS that writes an initial load file and continuous CDC files in parquet.I have been trying to follow the official guide Repl...

excavator-matt · 09-03-2025

In Databricks Runtime 16.4 LTS for Machine Learning, I am used to be able to import sentence-transformers without any installation as it is part of the runtime with from sentence_transformers import SentenceTransformer.In this case I am running on a ...

excavator-matt · 09-03-2025

We're trying to run the bundled sentence-transformers library from SBert in a notebook running Databricks ML 16.4 on an AWS g4dn.2xlarge [T4] instance.However, we're experiencing out of memory crashes and are wondering what the optimal to run sentenc...

excavator-matt

Correction. Trying this will result in this error> ABAC policies are not supported on tables defined within a pipeline. Remove the policies or contact Databricks support.So it isn't supported 

excavator-matt

One way to get version control might be to use the Terraform resource entity_tag_assignment. I am not sure if it supports governed_tags, but I'll experiment in the coming weeks.This separates the version control on where the tags are defined and wher...

excavator-matt · 10-30-2025

Also, upgrading to 17.3 ML still gives the same error.

excavator-matt · 10-30-2025

I now upgraded to the new 17.3 LTS ML and it now works. I didn't try 17.2 ML, but with 17.3 ML available, I don't see any reason to use it anymore.

excavator-matt · 10-30-2025

@Louis_Frolio I tried the Pandas on Spark approach.How do I from Delta table into Pandas on Spark DataFrame. Is this the best way? projects_df = spark.read.table("my_catalog.my_schema.my_project_table")projects_spdf = ps.from_pandas(projects_df.toPan...

Databricks Community

User Stats

User Activity

How do I grant access to find a table in Databricks, without giving access to query the table?

ABAC tag support for for Streaming tables (Spark Lakeflow Declarative Pipelines)?

How do use Databricks Lakeflow Declarative Pipeline on AWS DMS data?

Importing sentence-transformers no longer works on Databricks runtime 17.2 ML

What is the most efficient way of running sentence-transformers on a Spark DataFrame column?

Re: ABAC tag support for for Streaming tables (Spark Lakeflow Declarative Pipelines)?

Re: ABAC tag support for for Streaming tables (Spark Lakeflow Declarative Pipelines)?

Re: What is the most efficient way of running sentence-transformers on a Spark DataFrame column?

Re: Importing sentence-transformers no longer works on Databricks runtime 17.2 ML

Re: What is the most efficient way of running sentence-transformers on a Spark DataFrame column?