- 13 Views
- 1 replies
- 0 kudos
😑Datafactory insists on Cluster compute BUT Databricks defaults to Serverless compute!
Hi all,#Databricks#DatafactoryI hope you are well.I have bullet pointed my situation to allow other community members to easily diagnose my problem (I want to make your lives easy as well!).I have also included screenshot of data pipeline in DataFact...
- 13 Views
- 1 replies
- 0 kudos
- 0 kudos
The reason you are only seeing SQL Warehouses when clicking Compute is likely because the workspace admin has locked down interactive cluster creation. Check if you have the cluster creation entitlement. Your workspace admin can grant you the correct...
- 0 kudos
- 2 Views
- 0 replies
- 0 kudos
Jobs and pipelines UI
Hello, Could you please explain how to enable the same UI interface that my colleague has (shown in the first picture)? Unfortunately, my interface looks like the second picture, where the Error code filter does not highlight or display the number of...
- 2 Views
- 0 replies
- 0 kudos
- 27 Views
- 1 replies
- 0 kudos
How to load PDFs incrementally in volume?
I'm building an Intelligent Document Processing pipeline using Databricks AI Functions (ai_parse_document and ai_extract).I want to ingest PDF files from a Google Drive folder into a Unity Catalog Volume. I can perform a full load successfully using...
- 27 Views
- 1 replies
- 0 kudos
- 0 kudos
You can use the Google Drive Managed connector that does most of it primarily built for syncing structured data directly to Delta Tables or the Standard connector that allows you to use Databricks (SQL, Auto Loader, pipelines) directly against a Goog...
- 0 kudos
- 185 Views
- 4 replies
- 2 kudos
Databricks YAML files
when i change the yam file(not the resources one but can be any config yml) locally and then redeploy that bundle , the changes are not visible in the remote yml but other files(.py, .sql) do reflect these changes. FYI: during dev , i make changes in...
- 185 Views
- 4 replies
- 2 kudos
- 2 kudos
@prakharsachan What you're seeing is designed behavior, not a bug.When you run `bundle deploy`, the CLI does not upload everything in your bundle folder. It builds the upload list by walking the directory and skipping anything your .gitignore matches...
- 2 kudos
- 405 Views
- 3 replies
- 0 kudos
Advanced Learning Festival (15 June - 06 July 2026) - Voucher Eligibility Confirmation
Hello @Jim ,I completed all four modules of Learning Pathway 1 (Associate Data Engineering) for the Advanced Learning Festival on July 6, 2026, the published end date of the event. My Academy account email is [datasci.patel.rupesh@gmail.com]. Screens...
- 405 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @Ashwin_DSA ,Thank you for the confirmation. Following up as it's now July 17 and I haven't yet received the voucher at the email address associated with my Academy account (I've checked spam/promotions folders as well).For reference, I completed ...
- 0 kudos
- 313 Views
- 5 replies
- 6 kudos
DLT pipeline cloning to another workspace.
I know about the clone API:<databricks-instance>/api/2.0/pipelines/<pipeline-id>/clonebut it only works within the same workspace.Is it possible to achieve the same result manually?As I understand:I can't choose where checkpoints are migrated to in U...
- 313 Views
- 5 replies
- 6 kudos
- 6 kudos
@yanchr To the checkpoint part of your question first. There is no supported way to carry DLT checkpoint state over to a different pipeline, and that includes a pipeline in another workspace. This is true even on hive metastore, where the pipeline's ...
- 6 kudos
- 122 Views
- 1 replies
- 0 kudos
Compute Policy for DLT pipeline error
I have a compute policy created using terraform"unified_archive_pipeline_policy" = { policy_name = "Unified Archive Pipeline Policy" policy_definition = { "cluster_type" = { type = "fixed" value = "dlt" } # ...
- 122 Views
- 1 replies
- 0 kudos
- 0 kudos
@iress-nghia From the logs, looks like every failure is a numeric field: autoscale.min_workers, autoscale.max_workers, aws_attributes.first_on_demand, aws_attributes.spot_bid_price_percent.On the contrary, every string field passed: node_type_id, ava...
- 0 kudos
- 72 Views
- 0 replies
- 0 kudos
Stop Translating Alteryx Boxes - A Lakebridge-assisted, test-driven migration to Azure Databricks
Lakebridge can assess an Alteryx estate, but the current support matrix does not list Alteryx for automated conversion or direct reconciliation. A safe migration therefore combines Lakebridge assessment with deliberate redesign, native Databricks eng...
- 72 Views
- 0 replies
- 0 kudos
- 178 Views
- 1 replies
- 1 kudos
DeltaFileStatistics on a nested column (`created date.shipment`) cause filtering issues
Environment- Databricks Runtime version: 17.3 LTS- Cloud: Azure- Catalog: Unity Catalog- Table format: DeltaSummaryWe have a Delta table named `shipment` with a column `created date.shipment` (a column whose name contains a dot). Delta collects delta...
- 178 Views
- 1 replies
- 1 kudos
- 1 kudos
This is a known class of issue, not user error. Delta Lake stores min/max/null-count statistics in JSON using dot notation to represent nested struct paths (e.g. address.city means the city field inside an address struct). When a flat column...
- 1 kudos
- 342 Views
- 6 replies
- 4 kudos
Serverless egress - public internet access issues
I've just setup a new Databricks account on AWS with a workspace with "Serverless compute and default storage" and I am unable to access the public internet from notebooks and jobs for all but a handful of domains.A subset works fine, I've identified...
- 342 Views
- 6 replies
- 4 kudos
- 4 kudos
Hi there, from your note and comments, I couldn't gather if you have tried standard compute. Did you experience the same issues with standard compute? With serverless, you might experience some snags since the infrastructure is owned by Databricks an...
- 4 kudos
- 466 Views
- 5 replies
- 0 kudos
Bundle Validation Error: Volume lifecycle Field Rejected Despite Being in Schema
Environment:Bundle engine: directResource type: Volume (Unity Catalog)Problem:I'm trying to use the lifecycle field with prevent_destroy: true on a Unity Catalog volume resource in my Declarative Automation Bundle (DAB). the validation fails with:Err...
- 466 Views
- 5 replies
- 0 kudos
- 0 kudos
@animeshjain What's happening is that two different components are checking your config, and they are at different versions.databricks bundle schema is generated from the CLI binary itself, and your CLI is latest enough enough to know about lifecycle...
- 0 kudos
- 143 Views
- 1 replies
- 0 kudos
Run failed due to STORAGE_DOWNLOAD_FAILURE_SLOW → BOOTSTRAP_TIMEOUT
Hi there,We're recently being seeing issues with both classic all-purpose compute clusters (multi node specifically) and job clusters (also multi node) failing due to STORAGE_DOWNLOAD_FAILURE_SLOW → BOOTSTRAP_TIMEOUT.Our understanding is that this is...
- 143 Views
- 1 replies
- 0 kudos
- 0 kudos
We hit this exact failure mode a few months ago, so I can share how it played out for us and where I think your situation differs.Observations: Every cluster in one of our environments started failing with BOOTSTRAP_TIMEOUT. The instances themselves ...
- 0 kudos
- 206 Views
- 2 replies
- 2 kudos
AnalysisException: [UNRESOLVED_ROUTINE] Cannot resolve routine `=`
Hi all,We are experiencing an AnalysisException when migrating our Databricks cluster runtime from 13.3 LTS to 16.4 LTS (Scala 2.13), and would appreciate your help in identifying the root cause.**Error:**```AnalysisException: [UNRESOLVED_ROUTINE] Ca...
- 206 Views
- 2 replies
- 2 kudos
- 2 kudos
It doesn't sound like the === operator itself is the issue, since it should compile into a normal Catalyst expression rather than a SQL function call. The fact that Spark is trying to resolve = as a routine makes me suspect a catalog or analyzer beha...
- 2 kudos
- 254 Views
- 0 replies
- 0 kudos
Apache Spark 4.2 is officially here! Key architectural updates for AI-Native & Governed Platforms
Hi community!Matei Zaharia and the Databricks team just announced the release of Apache Spark 4.2. As a Data Architect, seeing how this engine is evolving to bridge the gap between traditional data engineering, governance, and the AI era is incredibl...
- 254 Views
- 0 replies
- 0 kudos
- 292 Views
- 1 replies
- 0 kudos
Genie Space and App Error
I built an App with a chatbot which generates queries and accesses data through a Genie Space. When making questions in the chatbot now, I keep getting the following error in the Genie monitoring page: An error happened. Please try again later.We're ...
- 292 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @emsmarcela , good instinct flagging that you haven't touched your app code. That's the most useful detail in your whole post, and it tells us a lot. An INTERNAL_ERROR like this one means Genie tripped while it was generating or processing the ...
- 0 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 579 | |
| 349 | |
| 287 |
