There is an usecase where I want to verify/extract the job caller identity in the job runtime of databricks. The job run_as and creator parameter seems to be fixed at the job configuration and doesn't reflect/update who actually triggered a job run.A...
Databricks Jobs `for_each_task` accepts exactly one nested task. For workloads that run the *samemulti-step pipeline over many datasets* — a very common metadata-driven pattern — this forces achoice between two architectures, each of which gives up s...
Hey droid,First — your analysis is correct and still current. I re-checked the docs: a ForEach task takes exactly one nested task (you can't even nest another ForEach), and the dependency scope sits at the ForEach level, so you can't fan out a hetero...
I am part of a small team of Data Engineers which started using Databricks Asset Bundles one year ago. Our code base consists of typical ETL-workloads written primarily in Jupyter notebooks (.ipynb), and jobs (.yaml) with our codebase spanning across...
Your proposed structure is sensible — it is the standard pattern for large-team monorepo setups with Databricks Asset Bundles. Here is a more complete answer to both questions you raised.1. Shared Python libraries: build a wheel, wire it as a bundle ...
Environment: Azure Databricks, Lakebase Autoscaling project. Database created through the Lakebase UI by the project owner (audit log shows createProjectBranchDatabaseUI with a browser user agent). Synced tables created via the databricks_postgres_sy...
Does anyone know a good tool thats can generate syenthetic time series-data from existing datasets?I am exploring options for data augmentation,testing and experimenting with different models.Open-source or commercial-both workWould love to hear, wha...
Hi @ThiamLee , How are you doing today?As per my experience, I’d suggest giving Genie Code a try, especially if you’re already working in Databricks.I’ve been using Genie Code while building a few of my own Data + AI apps and POCs, and what I like mo...
My team is investigating the migration of older and now non-adjustable azure skus to newer versions such as v5. I see in azure Usage+Quotas the Standard Dldsv5 and Dldsv6 families with available quota. However, when creating compute in Databricks the...
Hi @bill-kotronis ,The behavior you're seeing is expected — not all Azure VM families are exposed in Databricks, even if you have quota for them.Databricks maintains a curated, validated subset of Azure instance types. Having available quota in Azure...
I have identified a serious issue in Genie Agents that requires attention from the Databricks Product Team.When a catalog is renamed, the tables that were previously configured and used within the Genie Agent disappear unexpectedly. After manually re...
Has anyone gotten the new **Google Cloud Lakehouse catalog federation** Beta (GCP workspaces) to actually run queries?Setup (followed the docs end to end):- GCP workspace (us-central1, serverless-only), Unity Catalog- Previews page: Google Cloud Lake...
Hey matko,Short version: your diagnosis looks right, and this reads like a Beta rollout gap rather than anything you misconfigured.The stack trace (toProto failing in loadForeignSecurableMetadataAndUpdateCache on an unrecognized securable kind) plus ...
Hello all,I am trying to enable file events on my Azure Workspace for the File Arrival Trigger trigger mode for Databricks Workflows. I'm following this documentation exactly (I think) but I'm not seeing the option to enable them. As you can see here...
I am also facing some issue with the "file arrival" trigger on one of my environments.Its strange that it kept running fine until today where it broke. I always only had the "Storage Blob Data Contributor" for my Managed Identity ( Access Connector) ...
I am working in a regulated industry. It is necessary to document and replicate the compute environment. I guess to do this for a classic cluster vs a serverless one is different. Anyone knows how to do it? Thank you for your assistance.
Hi @NW1000 ,For Classic Compute, the best approach is using Databricks Container Services (DCS). This lets you package your exact OS, system libraries, and Python environment into a custom Docker image. You just push that image to a private registry ...
Hi everyone,I'm currently preparing for the Databricks Certified Data Engineer Associate exam and have completed the self-paced training on Databricks Academy. I'm a [student / early-career professional / etc. — fill in your situation] and the exam f...
Hi everyone, I'm working on a project where we collect visitor activity from multiple touchpoints such as ticketing, membership sign-ups, event participation, mobile app interactions, and digital membership card usage. The goal is to create a unified...
Concrete answer for the pieces you asked about, from a similar build (loyalty/membership 360 profile at bank scale - different domain, same shape of problem): Data model: model this as classic medallion, but the key decision is at silver/gold - use a...
Below is how the folder structure of my project looks like: resources/ |- etl_event/ |- etl_event.job.yml src/ |- pipeline/ |- etl_event/ |- transformers/ |- transformer_1.py |- utils/ |- logger.py databricks.ym...
You dont need to use wheel files . Use glob as the key instead of file - https://docs.databricks.com/aws/en/dev-tools/bundles/resources#pipelinelibrariesHere is the screenshot .
Hello everyone, I am new in Databricks, and currently I am in the 8th lecture of the "Get started with data engineering" module. I came across this problem: [STREAMING_TABLE_OPERATION_NOT_ALLOWED.ST_NOT_ENABLED_ON_SERVERLESS_GENERIC_COMPUTE] The oper...
This usually means your workspace doesn't have the Streaming Tables feature enabled, even if you're using Serverless. If the preview toggle is missing, check with your workspace admin or Databricks support, as it may depend on your workspace or regio...
Hi all,We run a Lakeflow Connect SQL Server ingestion pipeline with the gateway in continuous mode, reading from an Azure SQL Database using an Entra ID service principal (OAuth2 client credentials). The setup works overall, but we see a constant low...
Hi, The troubleshooting advice was actaully the other way round, it said from preview to current. But always worth trying the other channel, it's an easy test. Obviously just checkk it won't cause any other symptoms first (do a test run) THanks,Emm...
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 581 | |
| 349 | |
| 287 |
