- 125 Views
- 1 replies
- 0 kudos
How should schema evolution be handled across silver and gold layers in a medallion architecture?
We run a medallion pipeline on Databricks:Bronze: AutoLoader ingests raw CSV files into Delta tables (append-only, all columns as STRING, schema evolution via addNewColumns)Silver: PySpark jobs clean and transform bronze data into Delta tables using ...
- 125 Views
- 1 replies
- 0 kudos
- 0 kudos
You can skip using traditional schema migration tools like Liquibase, Flyway or Alembic into Delta Lake as they were generally designed for relational databases where state is tracked through sequential DDL scripts.Silver LayerNew Columns - You can l...
- 0 kudos
- 163 Views
- 1 replies
- 1 kudos
Synced table pipeline fails with permission denied for database
Environment: Azure Databricks, Lakebase Autoscaling project. Database created through the Lakebase UI by the project owner (audit log shows createProjectBranchDatabaseUI with a browser user agent). Synced tables created via the databricks_postgres_sy...
- 163 Views
- 1 replies
- 1 kudos
- 1 kudos
Hello @KhaturiabPreet !I like you analysis it made me do some reading this morning to freshen up my memory.I checked the the doc and I can say that databricks_writer_<dbid> being unable to CONNECT does not look like an expected state because it is t...
- 1 kudos
- 135 Views
- 2 replies
- 3 kudos
Databricks Support #00984257 - Password Reset Email is not working for me
Hello Everyone,I had raised a support request with the DBX Support team on Wednesday, 5 August 2026, but I have not received any response yet.Could anyone please guide to resolve my login issue? I forgot my password, and when I click the "Forgot Pass...
- 135 Views
- 2 replies
- 3 kudos
- 211 Views
- 1 replies
- 2 kudos
Is it safe to expose JWT in Databricks Job?
There is an usecase where I want to verify/extract the job caller identity in the job runtime of databricks. The job run_as and creator parameter seems to be fixed at the job configuration and doesn't reflect/update who actually triggered a job run.A...
- 211 Views
- 1 replies
- 2 kudos
- 2 kudos
Hello !What I can say is that your conclusion is correct since DBKS does not currently expose the identity of the principal that called run-now to the running job through the Jobs API or notebook context and the audit event is the authoritative sourc...
- 2 kudos
- 410 Views
- 2 replies
- 0 kudos
Resolved! Support Multiple Tasks DAG Inside a `for_each_task` Iteration
Databricks Jobs `for_each_task` accepts exactly one nested task. For workloads that run the *samemulti-step pipeline over many datasets* — a very common metadata-driven pattern — this forces achoice between two architectures, each of which gives up s...
- 410 Views
- 2 replies
- 0 kudos
- 0 kudos
Hey droid,First — your analysis is correct and still current. I re-checked the docs: a ForEach task takes exactly one nested task (you can't even nest another ForEach), and the dependency scope sits at the ForEach level, so you can't fan out a hetero...
- 0 kudos
- 4437 Views
- 4 replies
- 3 kudos
Asset Bundles: Shared libraries and notebooks in monorepo multi-bundle setup
I am part of a small team of Data Engineers which started using Databricks Asset Bundles one year ago. Our code base consists of typical ETL-workloads written primarily in Jupyter notebooks (.ipynb), and jobs (.yaml) with our codebase spanning across...
- 4437 Views
- 4 replies
- 3 kudos
- 3 kudos
Your proposed structure is sensible — it is the standard pattern for large-team monorepo setups with Databricks Asset Bundles. Here is a more complete answer to both questions you raised.1. Shared Python libraries: build a wheel, wire it as a bundle ...
- 3 kudos
- 170 Views
- 2 replies
- 1 kudos
Looking for recommendations
Does anyone know a good tool thats can generate syenthetic time series-data from existing datasets?I am exploring options for data augmentation,testing and experimenting with different models.Open-source or commercial-both workWould love to hear, wha...
- 170 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @ThiamLee , How are you doing today?As per my experience, I’d suggest giving Genie Code a try, especially if you’re already working in Databricks.I’ve been using Genie Code while building a few of my own Data + AI apps and POCs, and what I like mo...
- 1 kudos
- 155 Views
- 1 replies
- 0 kudos
available virtual machines in databricks
My team is investigating the migration of older and now non-adjustable azure skus to newer versions such as v5. I see in azure Usage+Quotas the Standard Dldsv5 and Dldsv6 families with available quota. However, when creating compute in Databricks the...
- 155 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @bill-kotronis ,The behavior you're seeing is expected — not all Azure VM families are exposed in Databricks, even if you have quota for them.Databricks maintains a curated, validated subset of Azure instance types. Having available quota in Azure...
- 0 kudos
- 359 Views
- 2 replies
- 2 kudos
Resolved! Critical Genie Agents Issue – Catalog Rename Causes Tables and Joins Loss
I have identified a serious issue in Genie Agents that requires attention from the Databricks Product Team.When a catalog is renamed, the tables that were previously configured and used within the Genie Agent disappear unexpectedly. After manually re...
- 359 Views
- 2 replies
- 2 kudos
- 198 Views
- 1 replies
- 2 kudos
Google Cloud Lakehouse catalog federation (Beta): queries fail with "Unrecognized...."
Has anyone gotten the new **Google Cloud Lakehouse catalog federation** Beta (GCP workspaces) to actually run queries?Setup (followed the docs end to end):- GCP workspace (us-central1, serverless-only), Unity Catalog- Previews page: Google Cloud Lake...
- 198 Views
- 1 replies
- 2 kudos
- 2 kudos
Hey matko,Short version: your diagnosis looks right, and this reads like a Beta rollout gap rather than anything you misconfigured.The stack trace (toProto failing in loadForeignSecurableMetadataAndUpdateCache on an unrecognized securable kind) plus ...
- 2 kudos
- 11610 Views
- 9 replies
- 2 kudos
Resolved! Trouble Enabling File Events For An External Location
Hello all,I am trying to enable file events on my Azure Workspace for the File Arrival Trigger trigger mode for Databricks Workflows. I'm following this documentation exactly (I think) but I'm not seeing the option to enable them. As you can see here...
- 11610 Views
- 9 replies
- 2 kudos
- 2 kudos
I am also facing some issue with the "file arrival" trigger on one of my environments.Its strange that it kept running fine until today where it broke. I always only had the "Storage Blob Data Contributor" for my Managed Identity ( Access Connector) ...
- 2 kudos
- 139 Views
- 1 replies
- 0 kudos
How to create an image from a cluster so that the compute environment can be replicated
I am working in a regulated industry. It is necessary to document and replicate the compute environment. I guess to do this for a classic cluster vs a serverless one is different. Anyone knows how to do it? Thank you for your assistance.
- 139 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @NW1000 ,For Classic Compute, the best approach is using Databricks Container Services (DCS). This lets you package your exact OS, system libraries, and Python environment into a custom Docker image. You just push that image to a private registry ...
- 0 kudos
- 242 Views
- 2 replies
- 1 kudos
Building a Visitor Data Pipeline for Digital Membership Card
Hi everyone, I'm working on a project where we collect visitor activity from multiple touchpoints such as ticketing, membership sign-ups, event participation, mobile app interactions, and digital membership card usage. The goal is to create a unified...
- 242 Views
- 2 replies
- 1 kudos
- 1 kudos
Concrete answer for the pieces you asked about, from a similar build (loyalty/membership 360 profile at bank scale - different domain, same shape of problem): Data model: model this as classic medallion, but the key decision is at silver/gold - use a...
- 1 kudos
- 2442 Views
- 8 replies
- 1 kudos
Resolved! How do I import a python module when deploying with DAB?
Below is how the folder structure of my project looks like: resources/ |- etl_event/ |- etl_event.job.yml src/ |- pipeline/ |- etl_event/ |- transformers/ |- transformer_1.py |- utils/ |- logger.py databricks.ym...
- 2442 Views
- 8 replies
- 1 kudos
- 1 kudos
You dont need to use wheel files . Use glob as the key instead of file - https://docs.databricks.com/aws/en/dev-tools/bundles/resources#pipelinelibrariesHere is the screenshot .
- 1 kudos
- 400 Views
- 4 replies
- 3 kudos
Resolved! CREATE is not allowed: Cannot CREATE the Streaming Table Serverless Generic Compute
Hello everyone, I am new in Databricks, and currently I am in the 8th lecture of the "Get started with data engineering" module. I came across this problem: [STREAMING_TABLE_OPERATION_NOT_ALLOWED.ST_NOT_ENABLED_ON_SERVERLESS_GENERIC_COMPUTE] The oper...
- 400 Views
- 4 replies
- 3 kudos
- 3 kudos
This usually means your workspace doesn't have the Streaming Tables feature enabled, even if you're using Serverless. If the preview toggle is missing, check with your workspace admin or Databricks support, as it may depend on your workspace or regio...
- 3 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 581 | |
| 349 | |
| 287 |
