- 6548 Views
- 12 replies
- 5 kudos
How to find the source code for the data engineering learning path?
Hi Everyone,I am taking data engineering learning path in customer-academy.databricks.com . I am not able to find any source code attached to the course. Can you please help me to find it so that I can try hands on as well ?ThanksRupesh
- 6548 Views
- 12 replies
- 5 kudos
- 5 kudos
The last module in "Get started with data engineering" learning path is wrong.Please find the screenshot attached. In the video the file used was different that the one present in resource section. Also i think in the video they were using file renam...
- 5 kudos
- 388 Views
- 2 replies
- 0 kudos
lakeflow connect postgres sql
Hi can anyone tell me how to enable tc LakeFlow Connect for Direct Cdc Managed Ingestion Pipeline. it shows that this mode is not available for this workspace for the selected connector.
- 388 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @tasa, Just responded to your other post on the same topic. It looks like your workspace is not currently enabled for Direct CDC/integrated CDC for that connector. In Databricks, this commonly surfaces as the DIRECT_CDC_NOT_ENABLED valid...
- 0 kudos
- 393 Views
- 2 replies
- 0 kudos
lakeflow connect postgres sql in uae north
Hi can anyone tell me how to enable tc LakeFlow Connect for Direct Cdc Managed Ingestion Pipeline. it shows that this mode is not available for this workspace for the selected connector.
- 393 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @tasa, It looks like your workspace is not currently enabled for Direct CDC/integrated CDC for that connector. In Databricks, this commonly surfaces as the DIRECT_CDC_NOT_ENABLED validation error, which means the feature is not enabled for the org...
- 0 kudos
- 300 Views
- 1 replies
- 0 kudos
Header Alignment in databricks dashboard
Hi Expert,when global filter page is added then visual header alignement gets missedThoughts: The user is reporting a UI/layout issue with the Lakeview dashboard — when a global filter page is added, the visual header alignment gets misaligned. This ...
- 300 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Rameshw, As you have pointed out, what you’re seeing is expected behaviour when a filter is added above an existing grid layout. Widgets on the canvas retain their existing positions, so adding a new row at the top does not automatically reflow t...
- 0 kudos
- 422 Views
- 2 replies
- 0 kudos
Urgent: Installing Lakebridge on Databricks
Hi,I want to install Databricks Lakebridge on my Databricks environment and use the analyze and transpile commands through a python script.The usecase is that we need to create an automated pipeline that will migrate the existing scripts from snowfla...
- 422 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi, the error you're seeing is consistent with the execution environment rather than with Lakebridge itself. Lakebridge is a Databricks CLI Labs extension (databricks labs install lakebridge) designed to run wherever the Databricks CLI is installed—y...
- 0 kudos
- 721 Views
- 2 replies
- 2 kudos
Resolved! StatusCode.UNIMPLEMENTED error: DatabricksConnect library using AKS/PySpark to calling Spark cluster
I am running PySpark application in AKS/Pythgon container/pod:Using Databricks 18.2.1 library with Databricks Spark cluster 18.2Once a while I am getting below error: InactiveRpcError of RPC that terminated with: status = StatusCode.UNIMPLEMENTED det...
- 721 Views
- 2 replies
- 2 kudos
- 2 kudos
Short answer: this looks more like an intermittent Spark Connect transport/routing issue than a Spark job logic issue. Databricks Connect uses gRPC over HTTP/2, and the specific InactiveRpcError ... UNIMPLEMENTED ... Received http2 header with status...
- 2 kudos
- 717 Views
- 2 replies
- 0 kudos
Resolved! NOT NULL constraint violated for column during OPTIMIZE
We're running an optimize on a delta table with a VARIANT column that has a NOT NULL constraint.Now, there are no NULL entries in this column. And yet, OPTIMIZE gives this error:[DELTA_NOT_NULL_CONSTRAINT_VIOLATED] NOT NULL constraint violated for co...
- 717 Views
- 2 replies
- 0 kudos
- 0 kudos
So funny story on this: the column in question does not match on "col is null", but it does match on "col::string is null".Perhaps because in the Parquet-file which corresponds to the rows that have the problem, that column is entirely missing.Still,...
- 0 kudos
- 843 Views
- 1 replies
- 0 kudos
Resolved! Lakebase CDF databricks error synced
Hello,I have defined Lakebase Change Data Feed (CDF) in order to sync some lakebase native tables into my catalog.schema, i can see that the tables are created with a naming convention like this: lb_*_history.sometimes the status is persisting in sna...
- 843 Views
- 1 replies
- 0 kudos
- 0 kudos
Thanks for sharing the screenshots. Assuming this is Lakebase Change Data Feed (formerly Lakehouse Sync) from Lakebase Postgres into Unity Catalog lb_*_history tables, this does not look like expected steady-state behavior. A few things stand out: O...
- 0 kudos
- 738 Views
- 1 replies
- 0 kudos
Resolved! Auto Loader duplicate tracking
Hi experts, I read an article about auto loader duplicates handling that got me bit confused. It is checkpoint that tracks what is being processed , and upon confirmation it process only the new incoming records. But, let's say I am reloading whole b...
- 738 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi, Auto Loader's tracking is at the file level, not the record level, and that distinction is exactly what's tripping you up here. The checkpoint keeps a RocksDB-backed record of every file it has already discovered and ingested, keyed by things lik...
- 0 kudos
- 220 Views
- 1 replies
- 0 kudos
Automation for any script changes in databricks and bit bucket
If any code changes are done and moved to bit bucket based on env now i want to fetch all the code changes and move all these changes to a table in adls/unity catalog table.Anyone please help with the approaches we are using bit bucket for repository
- 220 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi, couple ways to do this depending on how tightly you want it wired in. If Bitbucket is just your source repo and you want a row per change, the simplest path is a Bitbucket Pipeline that runs on push. In that pipeline step, pull the commit metada...
- 0 kudos
- 437 Views
- 1 replies
- 1 kudos
Bug? Genie Space benchmark evaluates "Score reason" wrongly as "Empty result" on agent mode
First some context: I'm setting up a Genie Space and have multiple benchmark questions with business validated SQL ground truths defined. Currently no configuration defined, apart from metadata and SQL joins.I haven't found any other way of reporting...
- 437 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi, this looks like a real quirk in agent mode benchmark scoring, not something you're doing wrong on the setup side. Chat mode benchmarks grade by comparing Genie's generated SQL result set against your SQL ground truth, so "Empty result" there mea...
- 1 kudos
- 561 Views
- 5 replies
- 0 kudos
Lakeflow connect Native connectors (tik, meta ads, Google Ads) - one table per account
We want to leverage these connectors to pull in marketing spend data. But the docs seem to say that the destination must be unique based on accounts. For Tik, we have a hundred accounts... each account will have a destination table for each object. ...
- 561 Views
- 5 replies
- 0 kudos
- 0 kudos
You can name a destination table when you create or edit your managed ingestion pipeline using Declarative Automation Bundles, notebooks, or the Databricks CLI, have a look at below doc. This should help. Name a destination table | Databricks on AWS
- 0 kudos
- 211 Views
- 1 replies
- 0 kudos
Getting error hadoop_azure_shaded.com.microsoft.azure.storage.StorageException: The specified blob d
I am exporting parquet files (partitioned by id) in append mode. However, I encounter errors occasionally, while other times the job completes successfully.Apache Spark Exception: Exception thrown in awaitResult: hadoop_azure_shaded.com.microsoft.azu...
- 211 Views
- 1 replies
- 0 kudos
- 0 kudos
Its generally due to race conditions when Spark checks for existing partition files before writing combined with Azure Blob Storage's eventual consistency mode.You can follow below1. Switch to Delta Lake - You can use Delta Lake format instead of Par...
- 0 kudos
- 963 Views
- 4 replies
- 0 kudos
Resolved! DLT pipelines failing out of memory (serverless)
I have a Data Lake Transformation (DLT) pipeline that runs weekly. Normally, it takes 8 minutes to complete, but since last Friday (June 19), it has been running for hours until it encounters an out-of-memory error. This pipeline is responsible for c...
- 963 Views
- 4 replies
- 0 kudos
- 0 kudos
Hey, I found that another engineer added joins to the metric view definitions to a federated query table. I converted it to a streaming table to use CDF and now time reduced. I need to improve joins to reduce latency as much as possible, but I’m not ...
- 0 kudos
- 229 Views
- 1 replies
- 0 kudos
Aspiring Data Engineer offering free project support in exchange for mentorship 🚀
Hi everyone,I am looking to accelerate my growth as a Data Engineer and am seeking hands-on guidance from an experienced professional. To gain real-world experience, I am offering my technical services completely free of charge.What I bring to the ta...
- 229 Views
- 1 replies
- 0 kudos
- 0 kudos
check my github where i make some small project while learning https://github.com/hasansayyed13?tab=repositories
- 0 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 582 | |
| 349 | |
| 287 |
