- 31 Views
- 1 replies
- 0 kudos
Skipping malformed records when reading Avro-files
Using read_files to read Avro-formatted blobs from an external location, I get the error:> Malformed records are detected in record parsing: Event 0. Parse Mode: FAILFAST. To process malformed records as null result, try setting the option 'mode' as ...
- 31 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Malthe ,read_files accepts format-specific options, but the Avro option table has an important scope column. For mode, that scope is only from_avro, not Avro file reads. Spark documents it as:"mode … allows you to specify parse mode for function ...
- 0 kudos
- 140 Views
- 3 replies
- 1 kudos
Unity Catalog service credential get_token rejects api:// scope format — "not a valid URI"
Calling get_token() on a Unity Catalog service credential fails for any scope using the api:// App ID URI format. Only https://-scheme resource scopes succeed. The same api:// scopes work correctly with a service principal.Reproductioncredential = db...
- 140 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi @Oumeima ,Totally understand. Let's hope support comes through with a fix
- 1 kudos
- 346 Views
- 6 replies
- 16 kudos
Why is the default auto-termination for serverless interactive notebook compute 60 minutes?
We've been digging into serverless interactive compute costs in notebooks, because our internal cost monitoring (cross-referencing system.billing.usage with system.query.history) showed a large share of the spend coming from idle time rather than act...
- 346 Views
- 6 replies
- 16 kudos
- 16 kudos
Hi @elizeu_reis , great detective work here. Cross-referencing system.billing.usage with system.query.history is exactly the right way to separate real usage from idle burn. I did some digging, and I can confirm most of what you observed. There are t...
- 16 kudos
- 57 Views
- 1 replies
- 0 kudos
Unexpected behavior of Delta VACUUM – need explanation
HelloI have a question about how VACUUM works.We executed the following VACUUM command from a PySpark job:spark.sql("""VACUUM catalog_name.schema_name.table_name RETAIN 168 HOURS""")The Delta table has a table property retention period of 365 days, b...
- 57 Views
- 1 replies
- 0 kudos
- 0 kudos
Greetings @bricks_2026 , I did some digging and here is what I found. First, good question, and you've already pulled the right evidence from the table history. This looks like retention semantics rather than a PySpark quoting issue or a logging bu...
- 0 kudos
- 68 Views
- 0 replies
- 0 kudos
Error DELTA_CATALOG_MANAGED_TABLE_UPGRADE_WITH_OTHER_PROPERTIES during catalog commit upgrade
Hi everyone,I'm encountering an issue with the Catalog Commit functionality while attempting to upgrade a table, following the official Databricks documentationCatalog commits | Databricks on AWSWhen running the following command on a Managed Iceberg...
- 68 Views
- 0 replies
- 0 kudos
- 137 Views
- 3 replies
- 0 kudos
Disabling Change Tracking and enabling Change Data Capture in SQL Server Lakeflow
Hi All,I want to know my lakeflow connect pipeline has stopped running because Change Tracking is enabled and table doesn't have a primary key and i want to switch to CDC now. Will it impact my pipeline. What are the steps i need to follow to on inge...
- 137 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi Upendra,When both Change Tracking and Change Data Capture are enabled the connector generally prioritizes Change Tracking. Since CT requires the source table to have a primary key and table did not have one, it causes the pipeline to fail. Even af...
- 0 kudos
- 4330 Views
- 3 replies
- 2 kudos
Pass variable from one notebook to another
I have a main notebook where I am setting a python variable to the name of the catalog I want to work in. I then call another notebook, using %run, which runs an insert into using a SQL command where I want to specify the catalog using the catalog v...
- 4330 Views
- 3 replies
- 2 kudos
- 2 kudos
Hi @ADB0513 With %run, you dont need to pass anything. %run runs the child inline in the same Python namespace, so the catalog variable set in the main notebook is already visible in the child. Just use it. # main catalog = "dev_catalog" # %run ./chi...
- 2 kudos
- 131 Views
- 1 replies
- 0 kudos
Synced Tables - Partitioned Tables
I am using synced tables with Lakebase Postgres and am seeing that all Postgres tables created via synced tables have a partition set based on the primary key I select when making the synced table. This is resulting in our downstream users in Postgre...
- 131 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @tpde ,This is expected behavior — synced tables in Lakebase use hash partitioning on the primary key internally for sync pipeline performance (parallel upserts). There's currently no option to create a synced table without partitioning.However, y...
- 0 kudos
- 11608 Views
- 5 replies
- 3 kudos
Resolved! Just a beginner in Data Engineer
Hi Everyone,I am happy to be part of this great community.I just determined to be a Data Engineer by profession and I will need a lot of advice on how I can quickly grab it and become a professional.I have Python Programming knowledge and Web develo...
- 11608 Views
- 5 replies
- 3 kudos
- 3 kudos
@DataSax @szymon_dybczak @Rishabh-Pandey Thank you all for the perfect question and answer. I am looking to transition myself into a Data Engineer role from a Data Analyst(10 years). I currently have experience working in DB2 and Hadoop with basic ...
- 3 kudos
- 111 Views
- 1 replies
- 0 kudos
Unable to Enable Unity Catalog – Azure Managed Identity Credential Not Found
Hi Databricks Community,I am trying to enable Unity Catalog in my Azure Databricks account. As part of the setup, I created an Azure Databricks Access Connector and assigned its managed identity the required permissions on my ADLS Gen2 storage accoun...
- 111 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Shubh5073 ,As per my understanding of the issue you are facing, the error indicates that Databricks cannot locate the Access Connector resource in Azure, rather than a problem with your ADLS permissions (since you already correctly assigned Stora...
- 0 kudos
- 94 Views
- 1 replies
- 0 kudos
Is VARIANT supported in Databricks-to-Open sharing?
I can't find exact documentation on whether VARIANT is supported for Databricks-to-Open sharing.. Any information is welcome
- 94 Views
- 1 replies
- 0 kudos
- 0 kudos
@yit337 Yes, it is supported as there's no such limitation in the docs.https://docs.databricks.com/aws/en/tables/features/variant
- 0 kudos
- 192 Views
- 4 replies
- 4 kudos
Google Drive ingestion pipeline failing – “Google Drive file system is not enabled”
Hello Databricks Community,I am using Databricks Free Edition and trying to ingest a CSV file from Google Drive using Data Engineering → Data Ingestion → Ingest data from Google Drive.The pipeline validation is failing with the following error:Pipeli...
- 192 Views
- 4 replies
- 4 kudos
- 4 kudos
Lakeflow Connect for Google Drive requires Serverless Compute and Unity Catalog Account-level feature flags. Databricks Community / Free Edition runs on a restricted, shared classic compute environment where Lakeflow Connect preview connectors and se...
- 4 kudos
- 89 Views
- 0 replies
- 0 kudos
Solution: Simplify Genie Agent Instruction Updates Across Multiple Spaces
Dear All,As the number of Genie Agents grows, maintaining consistent instructions across multiple Genie Spaces becomes increasingly cumbersome. Updating the same instructions manually in each space requires significant effort and is both time-consumi...
- 89 Views
- 0 replies
- 0 kudos
- 351 Views
- 1 replies
- 0 kudos
Requesting free/discounted voucher for Data Engineer Associate certification exam
Hi everyone,I'm currently preparing for the Databricks Certified Data Engineer Associate exam and have completed the self-paced training on Databricks Academy. I'm a [student / early-career professional / etc. — fill in your situation] and the exam f...
- 351 Views
- 1 replies
- 0 kudos
- 0 kudos
@Rahulkumar65 Vouchers are distributed to the eligible participants after the Learning Festival. Lookout for the next one in October. If your organisation is a partner of Databricks, then check with your admin.
- 0 kudos
- 85 Views
- 1 replies
- 0 kudos
Creating Databricks agent
Hi All,We are trying to create a Databricks agent and were looking for a step by step approach if there is any. We did go through the Author an agent and deploy it on Databricks Apps | Databricks on AWS but weren't very clear on creation of a new age...
- 85 Views
- 1 replies
- 0 kudos
- 0 kudos
@DineshOjha This doc will be useful in proceeding https://docs.databricks.com/aws/en/agents/. Give it a try and let me know.
- 0 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 581 | |
| 349 | |
| 287 |
