- 938 Views
- 7 replies
- 0 kudos
Getting 'Unauthorized Access' when Serverless compute is trying to read s3 bucket data
Getting 'Unauthorized Access' when Serverless compute is trying to read s3 bucket data, same works fine with normal compute storage and external location, credentials, IAM policy and serverless egress rules all seems to be correct because normal comp...
- 938 Views
- 7 replies
- 0 kudos
- 0 kudos
One data point that may be useful, we run serverless SQL against several external locations that were created manually through the API rather than auto-created, and they read fine from serverless.So I don't think you've hit a general rule that custom...
- 0 kudos
- 594 Views
- 2 replies
- 1 kudos
Resolved! Lakeflow Connect & Community Connectors - 403 Errors + What is the compute?
Hi there,I am regularly getting 403 errors with Lakeflow Connect or Community Connectors accessing our UC catalog/schema. As far as I'm seeing, I believe that we have the proper role on our Azure storage account (Storage Blob Data Contributor), and t...
- 594 Views
- 2 replies
- 1 kudos
- 1 kudos
@ChristianRRL Most likely the compute behind these connectors is serverless, and that would explain both parts of your question. SharePoint and the community connectors run on serverless compute only as far as I know.The docs list serverless as a pre...
- 1 kudos
- 906 Views
- 3 replies
- 1 kudos
Lakeflow Connect SharePoint - 403 Error
Hi, I'd like some help clarifying how lakeflow connect pipelines work (or should work). I went through the steps to (1) set up a connection, and (2) setup the ingestion, but I keep running into the following error:Operation failed: "This request is n...
- 906 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi! The 403 error you're seeing is almost certainly not coming from SharePoint — it's coming from Azure Storage.The key clue is in the error detail: access denied to /cas cloud path. The /cas path is Unity Catalog's managed storage location (Catalog ...
- 1 kudos
- 3256 Views
- 2 replies
- 3 kudos
Resolved! Cost-Effective Databricks Pipeline for API Ingestion - Best Practices?
Hi Community,As a senior data engineer migrating ETL workloads to Databricks (with Unity Catalog and Delta Lake), I'm building a cost-effective pipeline to ingest data from a REST API. Goals: minimize DBU costs, handle incremental loads, ensure scala...
- 3256 Views
- 2 replies
- 3 kudos
- 3 kudos
HI @TheDataMaverick ,The most efficient approach for your REST API ingestion pipeline on Databricks is to use an external service like Azure Functions (or AWS Lambda) to handle API calls, then land raw JSON/Parquet in ADLS/S3 for Auto Loader ingestio...
- 3 kudos
- 455 Views
- 1 replies
- 0 kudos
Jobs and pipelines UI
Hello, Could you please explain how to enable the same UI interface that my colleague has (shown in the first picture)? Unfortunately, my interface looks like the second picture, where the Error code filter does not highlight or display the number of...
- 455 Views
- 1 replies
- 0 kudos
- 0 kudos
What you are seeing is likely a classic staged UI rollout. Databricks frequently pushes updates such as the aggregated error code metrics on the Job Runs page using user-level feature flags or testing cohorts. Because these flags can be evaluated at ...
- 0 kudos
- 537 Views
- 1 replies
- 1 kudos
Resolved! How to load PDFs incrementally in volume?
I'm building an Intelligent Document Processing pipeline using Databricks AI Functions (ai_parse_document and ai_extract).I want to ingest PDF files from a Google Drive folder into a Unity Catalog Volume. I can perform a full load successfully using...
- 537 Views
- 1 replies
- 1 kudos
- 1 kudos
You can use the Google Drive Managed connector that does most of it primarily built for syncing structured data directly to Delta Tables or the Standard connector that allows you to use Databricks (SQL, Auto Loader, pipelines) directly against a Goog...
- 1 kudos
- 726 Views
- 4 replies
- 2 kudos
Databricks YAML files
when i change the yam file(not the resources one but can be any config yml) locally and then redeploy that bundle , the changes are not visible in the remote yml but other files(.py, .sql) do reflect these changes. FYI: during dev , i make changes in...
- 726 Views
- 4 replies
- 2 kudos
- 2 kudos
@prakharsachan What you're seeing is designed behavior, not a bug.When you run `bundle deploy`, the CLI does not upload everything in your bundle folder. It builds the upload list by walking the directory and skipping anything your .gitignore matches...
- 2 kudos
- 980 Views
- 3 replies
- 0 kudos
Advanced Learning Festival (15 June - 06 July 2026) - Voucher Eligibility Confirmation
Hello @Jim ,I completed all four modules of Learning Pathway 1 (Associate Data Engineering) for the Advanced Learning Festival on July 6, 2026, the published end date of the event. My Academy account email is [datasci.patel.rupesh@gmail.com]. Screens...
- 980 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @Ashwin_DSA ,Thank you for the confirmation. Following up as it's now July 17 and I haven't yet received the voucher at the email address associated with my Academy account (I've checked spam/promotions folders as well).For reference, I completed ...
- 0 kudos
- 796 Views
- 5 replies
- 6 kudos
DLT pipeline cloning to another workspace.
I know about the clone API:<databricks-instance>/api/2.0/pipelines/<pipeline-id>/clonebut it only works within the same workspace.Is it possible to achieve the same result manually?As I understand:I can't choose where checkpoints are migrated to in U...
- 796 Views
- 5 replies
- 6 kudos
- 6 kudos
@yanchr To the checkpoint part of your question first. There is no supported way to carry DLT checkpoint state over to a different pipeline, and that includes a pipeline in another workspace. This is true even on hive metastore, where the pipeline's ...
- 6 kudos
- 932 Views
- 1 replies
- 0 kudos
Compute Policy for DLT pipeline error
I have a compute policy created using terraform"unified_archive_pipeline_policy" = { policy_name = "Unified Archive Pipeline Policy" policy_definition = { "cluster_type" = { type = "fixed" value = "dlt" } # ...
- 932 Views
- 1 replies
- 0 kudos
- 0 kudos
@iress-nghia From the logs, looks like every failure is a numeric field: autoscale.min_workers, autoscale.max_workers, aws_attributes.first_on_demand, aws_attributes.spot_bid_price_percent.On the contrary, every string field passed: node_type_id, ava...
- 0 kudos
- 319 Views
- 1 replies
- 1 kudos
DeltaFileStatistics on a nested column (`created date.shipment`) cause filtering issues
Environment- Databricks Runtime version: 17.3 LTS- Cloud: Azure- Catalog: Unity Catalog- Table format: DeltaSummaryWe have a Delta table named `shipment` with a column `created date.shipment` (a column whose name contains a dot). Delta collects delta...
- 319 Views
- 1 replies
- 1 kudos
- 1 kudos
This is a known class of issue, not user error. Delta Lake stores min/max/null-count statistics in JSON using dot notation to represent nested struct paths (e.g. address.city means the city field inside an address struct). When a flat column...
- 1 kudos
- 1154 Views
- 6 replies
- 4 kudos
Serverless egress - public internet access issues
I've just setup a new Databricks account on AWS with a workspace with "Serverless compute and default storage" and I am unable to access the public internet from notebooks and jobs for all but a handful of domains.A subset works fine, I've identified...
- 1154 Views
- 6 replies
- 4 kudos
- 4 kudos
Hi there, from your note and comments, I couldn't gather if you have tried standard compute. Did you experience the same issues with standard compute? With serverless, you might experience some snags since the infrastructure is owned by Databricks an...
- 4 kudos
- 779 Views
- 5 replies
- 0 kudos
Bundle Validation Error: Volume lifecycle Field Rejected Despite Being in Schema
Environment:Bundle engine: directResource type: Volume (Unity Catalog)Problem:I'm trying to use the lifecycle field with prevent_destroy: true on a Unity Catalog volume resource in my Declarative Automation Bundle (DAB). the validation fails with:Err...
- 779 Views
- 5 replies
- 0 kudos
- 0 kudos
@animeshjain What's happening is that two different components are checking your config, and they are at different versions.databricks bundle schema is generated from the CLI binary itself, and your CLI is latest enough enough to know about lifecycle...
- 0 kudos
- 363 Views
- 1 replies
- 1 kudos
Run failed due to STORAGE_DOWNLOAD_FAILURE_SLOW → BOOTSTRAP_TIMEOUT
Hi there,We're recently being seeing issues with both classic all-purpose compute clusters (multi node specifically) and job clusters (also multi node) failing due to STORAGE_DOWNLOAD_FAILURE_SLOW → BOOTSTRAP_TIMEOUT.Our understanding is that this is...
- 363 Views
- 1 replies
- 1 kudos
- 1 kudos
We hit this exact failure mode a few months ago, so I can share how it played out for us and where I think your situation differs.Observations: Every cluster in one of our environments started failing with BOOTSTRAP_TIMEOUT. The instances themselves ...
- 1 kudos
- 847 Views
- 1 replies
- 1 kudos
Resolved! Genie Space and App Error
I built an App with a chatbot which generates queries and accesses data through a Genie Space. When making questions in the chatbot now, I keep getting the following error in the Genie monitoring page: An error happened. Please try again later.We're ...
- 847 Views
- 1 replies
- 1 kudos
- 1 kudos
Hello @emsmarcela , good instinct flagging that you haven't touched your app code. That's the most useful detail in your whole post, and it tells us a lot. An INTERNAL_ERROR like this one means Genie tripped while it was generating or processing the ...
- 1 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 582 | |
| 349 | |
| 287 |
