We've been digging into serverless interactive compute costs in notebooks, because our internal cost monitoring (cross-referencing system.billing.usage with system.query.history) showed a large share of the spend coming from idle time rather than act...
I have a main notebook where I am setting a python variable to the name of the catalog I want to work in. I then call another notebook, using %run, which runs an insert into using a SQL command where I want to specify the catalog using the catalog v...
Hi @ADB0513 With %run, you dont need to pass anything. %run runs the child inline in the same Python namespace, so the catalog variable set in the main notebook is already visible in the child. Just use it. # main catalog = "dev_catalog" # %run ./chi...
I am using synced tables with Lakebase Postgres and am seeing that all Postgres tables created via synced tables have a partition set based on the primary key I select when making the synced table. This is resulting in our downstream users in Postgre...
Hi @tpde ,This is expected behavior — synced tables in Lakebase use hash partitioning on the primary key internally for sync pipeline performance (parallel upserts). There's currently no option to create a synced table without partitioning.However, y...
Hi Everyone,I am happy to be part of this great community.I just determined to be a Data Engineer by profession and I will need a lot of advice on how I can quickly grab it and become a professional.I have Python Programming knowledge and Web develo...
@DataSax @szymon_dybczak @Rishabh-Pandey Thank you all for the perfect question and answer. I am looking to transition myself into a Data Engineer role from a Data Analyst(10 years). I currently have experience working in DB2 and Hadoop with basic ...
Hi All,I want to know my lakeflow connect pipeline has stopped running because Change Tracking is enabled and table doesn't have a primary key and i want to switch to CDC now. Will it impact my pipeline. What are the steps i need to follow to on inge...
You can trigger a full refresh on this table to establish a clean CDC baseline. Pause the ingestion pipeline (and the gateway if running continuous Standard CDC) to prevent state conflicts while updating the source. Next, on the SQL Server instance, ...
Hi Databricks Community,I am trying to enable Unity Catalog in my Azure Databricks account. As part of the setup, I created an Azure Databricks Access Connector and assigned its managed identity the required permissions on my ADLS Gen2 storage accoun...
Hi @Shubh5073 ,As per my understanding of the issue you are facing, the error indicates that Databricks cannot locate the Access Connector resource in Azure, rather than a problem with your ADLS permissions (since you already correctly assigned Stora...
Calling get_token() on a Unity Catalog service credential fails for any scope using the api:// App ID URI format. Only https://-scheme resource scopes succeed. The same api:// scopes work correctly with a service principal.Reproductioncredential = db...
Hi @Oumeima ,Thank you for the detailed reproduction. As per my understanding of the issue you are facing, your isolation of the problem is entirely correct.This behavior is caused by an undocumented strict URI validation bug within the Databricks db...
I can't find exact documentation on whether VARIANT is supported for Databricks-to-Open sharing.. Any information is welcome
@yit337 Yes, it is supported as there's no such limitation in the docs.https://docs.databricks.com/aws/en/tables/features/variant
Hello Databricks Community,I am using Databricks Free Edition and trying to ingest a CSV file from Google Drive using Data Engineering → Data Ingestion → Ingest data from Google Drive.The pipeline validation is failing with the following error:Pipeli...
Lakeflow Connect for Google Drive requires Serverless Compute and Unity Catalog Account-level feature flags. Databricks Community / Free Edition runs on a restricted, shared classic compute environment where Lakeflow Connect preview connectors and se...
Dear All,As the number of Genie Agents grows, maintaining consistent instructions across multiple Genie Spaces becomes increasingly cumbersome. Updating the same instructions manually in each space requires significant effort and is both time-consumi...
Hi everyone,I'm currently preparing for the Databricks Certified Data Engineer Associate exam and have completed the self-paced training on Databricks Academy. I'm a [student / early-career professional / etc. — fill in your situation] and the exam f...
@Rahulkumar65 Vouchers are distributed to the eligible participants after the Learning Festival. Lookout for the next one in October. If your organisation is a partner of Databricks, then check with your admin.
Hi All,We are trying to create a Databricks agent and were looking for a step by step approach if there is any. We did go through the Author an agent and deploy it on Databricks Apps | Databricks on AWS but weren't very clear on creation of a new age...
@DineshOjha This doc will be useful in proceeding https://docs.databricks.com/aws/en/agents/. Give it a try and let me know.
I wonder if there is any GitHub that specifically focuses on Databricks' Lakehouse monitoring solution with definitions and codes that can do this.I want to onboard it for a customer of mine and don't want to write from scratch if one is already pres...
Hi Surya, You can begin with Data Quality Monitoring solution - already present. It has anomaly detection & Data Profiling (formerly known as Lakehouse Monitoring) solutions helping in Statistical Profiling, data quality, snapshot quality monitors, d...
We run a medallion pipeline on Databricks:Bronze: AutoLoader ingests raw CSV files into Delta tables (append-only, all columns as STRING, schema evolution via addNewColumns)Silver: PySpark jobs clean and transform bronze data into Delta tables using ...
You can skip using traditional schema migration tools like Liquibase, Flyway or Alembic into Delta Lake as they were generally designed for relational databases where state is tracked through sequential DDL scripts.Silver LayerNew Columns - You can l...
Environment: Azure Databricks, Lakebase Autoscaling project. Database created through the Lakebase UI by the project owner (audit log shows createProjectBranchDatabaseUI with a browser user agent). Synced tables created via the databricks_postgres_sy...
Hello @KhaturiabPreet !I like you analysis it made me do some reading this morning to freshen up my memory.I checked the the doc and I can say that databricks_writer_<dbid> being unable to CONNECT does not look like an expected state because it is t...
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 581 | |
| 349 | |
| 287 |
