All Lakebase Discussions posts

Re: How to prevent users from creating Lakebase compute?

charl-p-botha — Fri, 26 Jun 2026 14:02:43 GMT

Thank you very much for also looking into this.

I've submitted this as feedback via the new Lakebase Postgres web-ui. It doesn't look like I can link to that submitted feedback here. Let's hope for the best.

Re: How to prevent users from creating Lakebase compute?

Ashwin_DSA — Fri, 19 Jun 2026 09:14:29 GMT

Hi @charl-p-botha,

Based on the current public documentation, I believe your reading is correct. In a Lakebase-enabled workspace, CAN_CREATE is inherited by all workspace users and cannot currently be granted or revoked on a per-project basis. The Azure Databricks docs for granting permissions programmatically state that the grantable permission levels for Lakebase projects are only CAN_USE and CAN_MANAGE, and that CAN_CREATE is inherited automatically from the workspace and cannot be explicitly granted or revoked.

The same position is reflected in the public docs for managing project permissions, which say that the default permissions for a newly created project include CAN_CREATE for all workspace users. The public ACL reference also says that all workspace users automatically inherit CAN_CREATE and that this permission cannot be assigned or removed.

I agree that this feels out of step with the rest of the platform. A workspace-level entitlement, or a privilege analogous to existing compute-creation controls, would be a much more natural fit here. At the moment, however, I have not found any documentation describing a supported way to selectively prevent some workspace users from creating Lakebase projects while still leaving Lakebase enabled for others.

So my understanding is that the only clearly documented hard control is to disable the feature entirely at the workspace or account level through Databricks Support, which, of course, does not help if you want Lakebase enabled only for a controlled subset of users.

If you want to push for this, I think this is a reasonable product feature request to raise with Databricks. You can submit it through the Databricks Ideas Portal, or from within a workspace using Send feedback. I would frame it specifically as a request for a workspace-level entitlement or revocable privilege that lets admins control who can create Lakebase projects, because that would directly address the governance and cost-management gap described above.

If this answer resolves your question, could you mark it as “Accept as Solution”? That helps other users quickly find the correct fix.

How to prevent users from creating Lakebase compute?

charl-p-botha — Thu, 18 Jun 2026 13:35:32 GMT

Dear community,

According to [1] and other sources, all workspace users are assigned `CAN_CREATE` on lakebase projects, and this permission "can't be revoked".

The problem is that such a project comes with by default a 8 - 16 CU lakebase compute instance (Scale-to-zero is enabled, but with a 24-hour idle timeout, any connection or query immediately resumes it, and it has a non-zero minimum (always-on baseline)), which means that anyone of our workspace(s) users is able to rack up a sizeable bill by accident. (the moment you create the project, the compute starts running).

After an in-depth exploration of all documentation and also the latest databricks cli, I have not been able to find any way to disable this regrettable default.

Please suggest a way whereby workspace users can be prevented from creating lakebase projects? We DO want to use lakebase for a number of our products, but we definitely also need to be able to specify who is able to create / use and who is not. (fully disabling the feature via support ticket as suggested in this forum post [2] would not work)

It would be far preferable to have it as an entitlement, or even connected to an existing entitlement (the aptly titled "Allow unrestricted cluster creation" could work), or first prize would be a revokable / assignable privilege. As it stands, there are no usable levers, which is highly uncharacteristic of Databricks products.

Please help.

Kind regards,
Charl Botha, Stone Three

[1] https://learn.microsoft.com/en-us/azure/databricks/oltp/projects/grant-permissions-programmatically
[2] https://community.databricks.com/t5/lakebase-discussions/disable-lakebase-and-model-serving-foundation-models-at-account/m-p/148792

Re: Controlling Agent access to Tools and Tool access to Data Operations

Ashwin_DSA — Thu, 04 Jun 2026 09:50:24 GMT

Hi @venkat-raghavan,

Thanks. Good point, and I agree the distinction should be clearer. What I was trying to separate are two related but different ideas... Runtime tool-level control and End-to-end workflow control for destructive changes

The public documentation on service policies absolutely does imply...and explicitly says... that service policies are there to narrow the action surface at runtime. The blog says the core problem in recent incidents was that agents had delegated authority but lacked restrictions on which tools they could invoke, and there was no trace of what they did. The linked incidents are consistent with that framing. One describes an agent deleting a production volume after finding a token with delete capability, another describes an agent choosing Terraform destroy as the "cleaner and simpler" option during cleanup, and the incident database entry describes an agent reportedly executing unauthorised destructive commands against production data despite repeated instructions not to make changes.

And the Databricks blog is quite direct about the fix. Once MCPs are registered, you get control over what agents are allowed to do. Service policies evaluate every tool call. Admins can allow, deny, or require consent. And, policies can restrict specific tools like delete_database or conditionally allow them only for certain actors.

So I agree with your reading... service policies are not just observability. They are a runtime enforcement mechanism whose primary value is to constrain the tool/action surface.

Where I was drawing a distinction is that this is still slightly narrower than a full workflow pattern like plan → validate/preview → approve → execute. Service policies operate at the individual tool call boundary. Before a tool executes, the policy can block it, allow it, or require consent. That is powerful and important. But it is not automatically the same thing as a multi-step sandboxed change-management workflow with staging, preview state, and commit/abort semantics.

If this answer resolves your question, could you mark it as “Accept as Solution”? That helps other users quickly find the correct fix.

Re: Controlling Agent access to Tools and Tool access to Data Operations

venkat-raghavan — Thu, 04 Jun 2026 02:21:59 GMT

Hi Ashwin

I do want to contrast you stated really well from what is stated or implied in the Service Control policies documentation you shared.

You said things

1) For write or move actions, I would strongly recommend a controlled execution pattern... plan → validate/preview → approve → execute. That gives you a checkpoint before any destructive or irreversible action and is much safer than allowing an agent to directly execute whatever it decides at runtime."

2) Another important design decision is to make the action surface narrower and suitable for policy implementation. For example, instead of giving an agent broad SQL write access, expose a small set of approved operations via UC functions or MCP tools, and then apply runtime policies to those tool calls (via service policies)

But the documentation is misleading. The documentation starts with

" Agents connected to external tools are taking destructive, irreversible actions in production: wiping entire databases in seconds, deleting millions of rows of critical data, and dropping production databases mid-task. In each incident, the agent was acting within the scope of their delegated authority. What it lacked was any restriction on which tools it could invoke, and any record of the actions it took. "

This implies that Service policies actually do "controlled execution" wherein it's primary function is to "limit the action surface". The documentation should make this clear.

Re: Controlling Agent access to Tools and Tool access to Data Operations

venkat-raghavan — Thu, 04 Jun 2026 02:07:32 GMT

This is perfect. Thanks.

I also like your answer on runtime execution - "For write or move actions, I would strongly recommend a controlled execution pattern... plan → validate/preview → approve → execute. That gives you a checkpoint before any destructive or irreversible action and is much safer than allowing an agent to directly execute whatever it decides at runtime."

Re: Lakebase Data API private access with Public Network Access disabled

stbjelcevic — Mon, 01 Jun 2026 23:48:53 GMT

Hi @POCUSER ,

Yes, the Lakebase Data API can be used privately with Public Network Access disabled. Because the Data API is a REST endpoint (Lakebase Data API), it goes through your workspace’s standard inbound (front-end) Private Link, the databricks_ui_api endpoint on port 443, not a dedicated one.

Service Direct Private Link (the port-5432 endpoint for performance-intensive services) is not required for the Data API. The docs state it directly: “If your applications connect only through the Data API, you don’t need this endpoint.” See Private Link for Lakebase Autoscaling and Configure inbound Private Link for performance-intensive services.

So this is a DNS issue, not a missing Private Link. With Public Network Access disabled, your DNS must resolve the Data API hostname to the private IP of your existing inbound private endpoint (the privatelink.azuredatabricks.net zone, databricks_ui_api A record). The 403 is consistent with DNS still resolving the hostname to a public IP instead of your private endpoint. Confirm with nslookup that it returns the private IP. See Configure Inbound Private Link for the DNS verification steps.

Lakebase Data API private access with Public Network Access disabled

POCUSER — Fri, 29 May 2026 07:58:24 GMT

We are testing Azure Databricks Lakebase Autoscaling with Public Network Access disabled and standard inbound Private Link enabled.

The workspace UI works privately through VPN, but the Lakebase Data API hostname still resolves to a public IP and returns:

HTTP 403: Public access is not allowed for workspace

According to the docs, Service Direct Private Link is not required when using only the Data API.

Has anyone successfully used Lakebase Data API privately with Public Network Access disabled?

If yes, what DNS or Private Link configuration is required? Should the Data API hostname resolve through the workspace inbound Private Link, or is another private endpoint/DNS setup needed?

Thanks!

Re: Inquiry regarding Query History and Audit Logs for Databricks Lakebase

Ashwin_DSA — Mon, 25 May 2026 11:40:21 GMT

Hi @POCUSER,

I’m not aware of any public documentation that explicitly confirms that every request sent through the Lakebase Data API is captured in system.query.history or exposed through a dedicated per-request audit log.

A practical way to validate it is to run a simple test query through the Data API and then check system.query.history. The public docs for that table currently describe coverage for SQL warehouses and serverless compute, so testing is the best way to confirm whether Lakebase Data API activity appears there in your environment.

It is also worth checking system.access.audit together with the audit log reference to see whether any Lakebase-related control-plane events are emitted there.

If you need a definitive answer specifically for compliance, such as whether full query text, caller identity, and failed request details are logged for Lakebase Data API requests, I’d recommend opening a Databricks support case. Some of the Lakebase observability surfaces are still evolving, so support is the best route for confirmed guidance on the current supported behaviour.

If this answer resolves your question, could you mark it as “Accept as Solution”? That helps other users quickly find the correct fix.

Re: Inquiry regarding Query History and Audit Logs for Databricks Lakebase

balajij8 — Mon, 25 May 2026 09:43:22 GMT

You can follow below

Query History Tracking - You can use the pg_stat_statements entity for getting the historical SQL query execution details and other runtime metrics in the Lakebase instance. Every successful or failed query sent through the Lakebase is parameterized, normalized and logged alongside its total runtime, execution time and other tracking metrics. This entity serves as the primary ledger for tracking precisely what query texts were run. You can periodically offload these metrics if you require persistent retention beyond the life of the compute cluster as the data lives within the database's shared memory, More details here
Active Query Monitoring - You can use pg_stat_activity to monitor live requests, active transactions and current connections hitting the Lakebase in real time for monitoring. This entity exposes the immediate state of the database showing he currently executing queries, client IP addresses, query, backend type and operational states. It helps in finding which accounts are running active, long running or blocked statements at any given second.
Account Level Audit Logs - You can use system.access.audit for account level governance, infrastructure compliance and broader security perspective. All major activities related to the database - such as provisioning, configuration changes etc are recorded in the system.access.audit table. To isolate these specific events use the service name databaseInstances or postgres for getting the related details.

Inquiry regarding Query History and Audit Logs for Databricks Lakebase

POCUSER — Mon, 25 May 2026 05:13:44 GMT

We are using Lakebase Data API (HTTP Endpoint) to execute queries and need to verify the audit log capabilities for compliance. Could you please clarify:

Query Text Logging: Does Databricks capture the full SQL statement text and the actual user identity for every request sent via the Lakebase Data API?
Log Location: Where are these Data API history logs stored (e.g., in system.query.history, Audit Logs in Storage Bucket, or via Unity Catalog)?
Error Logging: Are failed queries (Query Errors) and their detailed error messages recorded in these logs as well?

Re: Controlling Agent access to Tools and Tool access to Data Operations

Ashwin_DSA — Sun, 24 May 2026 18:00:19 GMT

Hi @smithsonian,

The general best-practice pattern is to separate what the agent plans... from...what the platform will actually allow it to do.

If the data already lives in Unity Catalog, a good approach is to use Unity Catalog as the system of record for data permissions and use Unity AI Gateway as the runtime governance layer for agents, LLM endpoints, and MCP servers.

In practical terms, that usually means keeping planning/reasoning agents read-only where possible and exposing actions through governed tools such as UC functions and MCP servers, rather than letting agents perform arbitrary direct operations. Another recommendation is to run user-facing workflows with on-behalf-of-user authentication so the agent cannot exceed the caller’s permissions. The key is to reserve specific service principals for background or automated jobs that truly require non-user execution.

For write or move actions, I would strongly recommend a controlled execution pattern... plan → validate/preview → approve → execute. That gives you a checkpoint before any destructive or irreversible action and is much safer than allowing an agent to directly execute whatever it decides at runtime.

Another important design decision is to make the action surface narrower and suitable for policy implementation. For example, instead of giving an agent broad SQL write access, expose a small set of approved operations via UC functions or MCP tools, and then apply runtime policies to those tool calls. Databricks has publicly described this direction with service policies and payload logging for MCPs, which is exactly the sort of control plane you want for "this agent can read, this one can write only to dataset X, and this one can move data only under condition Y."

Lastly, make sure every tool invocation is observable. A runtime governance model is only credible if you can answer... who called what tool, with which arguments, on whose behalf, and what happened. AI Gateway’s observability and logging model is designed for that kind of audit trail.. although it is still in beta.

If this answer resolves your question, could you mark it as “Accept as Solution”? That helps other users quickly find the correct fix.

Re: How to access a delta table in UC from lakebase postgres ?

Abhishek_sinha — Tue, 12 May 2026 15:09:21 GMT

Currently, direct write-back from Lakebase PostgreSQL to Unity Catalog Delta tables is not the recommended pattern.

A few points that may help:

• “Sync Tables” are mainly designed for replication/read access scenarios. Databricks currently recommends avoiding direct writes to synced tables because consistency/governance behavior may not be guaranteed.

• If the goal is to move Delta data into PostgreSQL inside Lakebase, the better approach is usually:
- ETL/ELT pipelines using Databricks Jobs
- Structured Streaming
- COPY INTO / JDBC writes
- CDC-based pipelines

In practice, Databricks → PostgreSQL via JDBC or streaming is the safer supported architecture.

• For bidirectional workloads, many teams treat:
- Delta + Unity Catalog = analytical/storage layer
- PostgreSQL/Lakebase = transactional serving layer

Regarding your second question:

• Yes, PostgreSQL can be exposed in Unity Catalog using Lakehouse Federation (Foreign Catalog concept). PostgreSQL is one of the supported federated sources, so you can query PostgreSQL tables from UC without fully ingesting the data.

But federation mainly enables governed query access — it does not make PostgreSQL tables behave like native Delta tables with all UC-managed write semantics.

So today:
- Read/query interoperability → supported
- Native governed write-back from Lakebase PostgreSQL into UC Delta tables → limited/not the primary recommended pattern yet.

Re: How to create a lakebase table ?

balajij8 — Wed, 29 Apr 2026 14:27:57 GMT

You can use Lake base Autoscaling with Synced Tables.

Synced tables empower you to export high quality & governed data from Lakehouse into Lake base. Its specifically designed for downstream applications requiring extreme responsiveness (sub 10ms responses) along with the reliability of ACID compliant transactions for real time cases.

You can sync gold tables from patient_details into a new synced table patient_details_synced. You can use sync modes based on needs

Snapshot - One time copy of all data
Triggered - Scheduled updates that run on demand or at intervals
Continuous - Real time streaming with latency in seconds

You can check the resource Modern Databricks Lake base with Medical Inventory Management Apps here

Re: Lakebase auto start/stop

balajij8 — Wed, 29 Apr 2026 10:56:01 GMT

Lake base supports Autoscaling & Scale to Zero with Automatic Suspension & Reactivation capabilities

Automatic Suspension - Lake base compute automatically suspends after a period of inactivity (default is 5 minutes).
Reactivation - The compute automatically reactivates within a few hundred milliseconds when you run a new query

You can use Autoscaling to adjust resources based on workload demand & scale to zero to suspend compute entirely after a period of inactivity reducing compute costs to zero during idle periods. You pay only for active compute time & not for idle periods

More details here

Re: Lakebase login via REST for a service principal

_Lilith — Fri, 24 Apr 2026 19:42:38 GMT

I created a new Lakebase project to retrace all my steps.

0- I reused my service principal on the workspace

1- installed databricks authentication extension:

CREATE EXTENSION IF NOT EXISTS databricks_auth;

2-Added the lakehouse service principal to the lakebase project

SELECT databricks_create_role('{UUID}', 'SERVICE_PRINCIPAL');

3- Enabled Data API to get authenticator user created

4- Finally granted authenticator role to the service principal

GRANT "{UUID}" TO authenticator;

And this time it worked. I do not understand what the difference is to last time, maybe my authenticator user was somehow corrupted.

Thank you @balajij8 & @szymon_dybczak for your answers and suggestions

Re: Lakebase login via REST for a service principal

balajij8 — Fri, 24 Apr 2026 15:33:05 GMT

It's failing due to insufficient privileges. Can you check that you have 'Can Manage' access under Project Permissions in the Lake base project? If you have 'Can Use' access, you can ask for 'Can Manage' access or you can ask the admin to run the GRANT sequence to initialize the Service Principal

Re: Lakebase login via REST for a service principal

_Lilith — Fri, 24 Apr 2026 14:12:25 GMT

That is what I am using. Taking the example of the documentation what I do first is:

1- Create a service principal in the workspace

2- Add the service principal as a user in lakebase, using service principals Application ID

SELECT databricks_create_role('8c01cfb1-62c9-4a09-88a8-e195f4b01b08', 'SERVICE_PRINCIPAL');

3- I get the mentioned error at the first step of giving permissions in the SQL editor:

-- Allow authenticator to assume the identity of the user GRANT "8c01cfb1-62c9-4a09-88a8-e195f4b01b08" TO authenticator;

ERROR: permission denied to grant role "8c01cfb1-62c9-4a09-88a8-e195f4b01b08" (SQLSTATE 42501)

Re: Lakebase login via REST for a service principal

balajij8 — Fri, 24 Apr 2026 13:06:24 GMT

You can change the code to use the client application ID (UUID) of the service principal as the identity name and run it.

GRANT "UUID" TO authenticator;

Re: Lakebase login via REST for a service principal

_Lilith — Fri, 24 Apr 2026 12:25:55 GMT

Thanks for your reply.

These are the steps that I have also followed and had linked to in my question.

When granting permissions, the first line of the documentation fails when I use a service principal

-- Allow authenticator to assume the identity of the user GRANT "{service principal user ID}" TO authenticator;

ERROR: permission denied to grant role "{service principal user ID}" (SQLSTATE 42501)