Lakebase storage location

[YugandharG]

Hi,

I'm a Solution Architect from a reputed insurance company looking for few key technical information about Lakebase architecture. Being fully managed serverless OLTP offering from Databricks, there is no clear documentation that talks about data storage location, API access and underlying file format. It would appreciate if someone can answer my points. 

1. Storage location: Where does the Lakebase data stored? Will that be in customer cloud storage account or Databricks cloud storage account? Is there an option to choose?

2. File format: What is the data file format of Lakebase tables? Is it a parquet file + delta file + caching ?

3. Asynchronous access: The documentation says about direct connection methodologies like ODBC and Postgres adaptors, is there a direct way to query and update Lakebase table using REST API? 

4. Vector index: Can a vector search index created on a Lakebase table? Imagine it has PK, CDC enabled

#Lakebase #Storage #RESTAPI

[szymon_dybczak]

Hi @YugandharG ,

1. Lakebase data is stored in databricks-managed cloud object storage. There's no option to use customer storage as of now.

2. File format: vanilla postgres pages. The storage format of postgres has nothing to do with parquet/delta. Watch below video where Reynold Xin explains how Lakebase works under the hood and how they achieve separation of compute and storage. Really interesting talk.

Introducing Lakebase - Databricks Co-founder & Chief Architect Reynold Xin

Of course Lakebase support synchronization of delta and iceberg tables but the data itself will be stored in postgres tables (which are supported by postgres pages). 

3. Currently there's no direct way to query Lakebase table using REST API. But you can manage your database instances using REST API. Here's a list of all activities you can perform through rest api:

4. I don't think so. At least I couldn't find any information regarding this in documentation.