- 441 Views
- 2 replies
- 2 kudos
Add resource to app space
Hi,I recently started trying out the Genie App Builder first presented at DAIS. For this I created an app space and I wanted to connect my app to a lakebase project. But I can't seem to find any options to do this like it works with apps which are no...
- 441 Views
- 2 replies
- 2 kudos
- 2 kudos
Hello @JanW Following up on the answer above from @saurabh18cs and the results of a quick API test. Thank you for setting out what you tried and how you compared it with an app outside a space. That made the case easy to reproduce Short answer: You...
- 2 kudos
- 5746 Views
- 8 replies
- 1 kudos
Py4JJavaError: An error occurred while calling o465.coun
The following error occured when running .count() on a big sparkDF. Py4JJavaError: An error occurred while calling o465.count. : org.apache.spark.SparkException: Job aborted due to stage failure: Task 6 in stage 3.0 failed 4 times, most recent failur...
- 5746 Views
- 8 replies
- 1 kudos
- 1 kudos
Hi @sinclair Seems you have so many upstream transformations before doing this count operation on running dataframe??for me your nested json is suspicious , you might be trying to retrieve something nullable there causing this issue, can you check ru...
- 1 kudos
- 298 Views
- 2 replies
- 0 kudos
Databricks Free Edition – AWS S3 Storage Credential Validation Fails at Assume Role
Hi Databricks Team,I'm trying to connect an AWS S3 bucket to Databricks Unity Catalog using an IAM role in Databricks Free Edition.EnvironmentDatabricks Edition: Free EditionCloud: AWSWorkspace region: us-east-2Metastore region: us-east-2S3 bucket re...
- 298 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @Visheshd_21 Based on the validation results, the failure is occurring at the initial AssumeRole step. Since Self Assume Role and External ID checks are skipped, Databricks is not successfully assuming the IAM role, so I would troubleshoot the AWS...
- 0 kudos
- 322 Views
- 2 replies
- 0 kudos
Resolved! Unable to create metastore in account console with my Azure Account
Azure Managed Identity Credential with Access Connector Id /subscriptions/c03723bd-79a8-401a-8392-a6aef2b0d7b9/resourceGroups/RG_databricks_masterclass/providers/Microsoft.Databricks/accessConnectors/ac_meta_new and Managed Identity ID: for Account I...
- 322 Views
- 2 replies
- 0 kudos
- 0 kudos
Resourceid: /subscriptions/c03723bd-79a8-401a-8392-a6aef2b0d7b9/resourceGroups/RG_databricks_masterclass/providers/Microsoft.Databricks/accessConnectors/ac_meta_new Databricks Accountid:https://accounts.azuredatabricks.net/?account_id=79762b8f-7...
- 0 kudos
- 505 Views
- 4 replies
- 0 kudos
Cluster environment variables not visible to Java Dataset.foreachPartition / mapPartitions
We run a Spark JAR task (Java) on a standard access mode cluster. Credentials come from cluster environmentvariables set as secret references in the cluster's Environment variables field, e.g.MY_PASSWORD={{secrets/<scope>/<key>}}.The variables are vi...
- 505 Views
- 4 replies
- 0 kudos
- 0 kudos
Hi @capoorhimanshu,Your worry about the closure is fair. A secret that rides inside the serialized task is harder to control than one that never leaves a secret store.As far as I can tell, there's no supported setting on standard access mode that pus...
- 0 kudos
- 374 Views
- 3 replies
- 0 kudos
Free Edition — S3 storage credential validation fails at "Assume Role" though IAM config is verified
Workspace ID 7474651448614968 (Free Edition, AWS). Credential databricks-dataengineer-v2 (IAM role arn:aws:iam::278137443935:role/DATABRICKS-DATAENGINEER) fails validation at the Assume Role step; CREATE EXTERNAL LOCATION returns PERMISSION_DENIED .....
- 374 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @Arisk, I agree with @DoTA that your AWS setup looks right. The master role ARN you're using matches the one in the AWS setup docs (https://docs.databricks.com/aws/en/connect/unity-catalog/cloud-storage/s3/s3-external-location-manual), and I coul...
- 0 kudos
- 720 Views
- 2 replies
- 2 kudos
Resolved! Streaming read fails with "Detected a data update" after a restatement job touches the source
Hi everyone,Looking for the right pattern here rather than a workaround.Setup. DBR 15.4 LTS, Unity Catalog. A silver streaming table reads from a bronze Delta table with a normal streaming read. Bronze is append only in the ordinary course of busines...
- 720 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi @Islam_hoti, good instinct not to just slap skipChangeCommits on and walk away. Both of your ideas work. Which one I'd pick depends on how often corrections happen.Option A: rare, batched corrections (what you have). The Delta streaming docs descr...
- 2 kudos
- 138 Views
- 1 replies
- 0 kudos
Managed table conversion and vacuum
Hello everyone,I am looking for clarification about the 14-day rollback period after converting an external table to a managed table using: ALTER TABLE <table_name> SET MANAGEDDatabricks document...
- 138 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Davide, this thread never got a reply, but you asked the same thing in SET MANAGED and rollback in relation to VACUUM and posted your test result there, so I'm linking it for anyone who lands here first.Short version: the 14-day rollback window i...
- 0 kudos
- 449 Views
- 3 replies
- 1 kudos
Question regarding Databricks Apps deployment
Hello Databricks Community,I am currently working on setting up a Databricks App and ran into a few questions.Here are the details of what I am working with:Target Framework: [e.g., Streamlit / Dash]Goal: [Briefly describe what you are trying to achi...
- 449 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi @Khasim_1,It looks like the template placeholders made it into the post, so I'm not sure exactly where you're stuck. In case it helps, here's a minimal Streamlit app that reads a Unity Catalog table through a SQL warehouse. That covers the three t...
- 1 kudos
- 726 Views
- 5 replies
- 1 kudos
Resolved! Databricks Unity Catalog Foreign Catalog – Timezone Mismatch with AWS RDS PostgreSQL Affecting Query
Hi Databricks Community,I'm facing a timezone-related issue with a Unity Catalog Foreign Catalog connected to AWS RDS PostgreSQL and would appreciate some guidance on the best way to handle it centrally.Current Setup:Source: AWS RDS PostgreSQL (confi...
- 726 Views
- 5 replies
- 1 kudos
- 1 kudos
Hi @vkondepati,Quick follow-up on my view above, because data_pulse's tests made me double-check one line and it needs a fix.CAST(created_at AS TIMESTAMP_NTZ) is evaluated in the session time zone of whoever queries the view. The Databricks docs say ...
- 1 kudos
- 404 Views
- 2 replies
- 1 kudos
floating precision error in databricks
Hey team,I ran into this issue back in 2019, but I don't recall how we resolved it. I'm looking for guidance from experienced Spark/Databricks engineers.Our source system contains values like: source in-databricks source in-databricks 24.563824.56000...
- 404 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @Aviral-Bhardwaj - From your sample it looks like the values are losing everything past the 2nd decimal place (e.g. 24.5638 → 24.56 / 24.57), which usually points to the scale of the target column or a cast in the pipeline, not a Databricks displ...
- 1 kudos
- 1169 Views
- 5 replies
- 0 kudos
How to extract data from SAP to Databricks?
Hey everyone, I'm working through a project where I need to get data out of SAP S4Hana and into Databricks, and I'd love to pick the brains of anyone who's done this before.A few things I'm trying to figure out:How are you actually pulling the data o...
- 1169 Views
- 5 replies
- 0 kudos
- 536 Views
- 4 replies
- 0 kudos
Resolved! LEFT JOIN WITH FILTER vs LEFT JOIN and WHERE FILTER
Hi,I may need to investigate further but this is my observation,I had a SQL query working fine until today,Query-A SELECT ta.id,tb.id FROM TableA ta left join TableB tb on ta.id = tb.id and ta.dtx = to_date(<date string>) where ...
- 536 Views
- 4 replies
- 0 kudos
- 0 kudos
@RGSLCA Nice that you observed this in oracle too. Good to see the behaviour is consistent across platforms.
- 0 kudos
- 804 Views
- 3 replies
- 1 kudos
Resolved! Dark mode setting ignored when embedding BI dashboard inside Databricks App
When using the '@databricks/aibi-client' library to embed a BI dashboard inside a databricks app, the " colorScheme: 'dark' " setting is ignored when the app's 'prefer-color-scheme' is 'light' by default.As a result, I am unable to set my dashboards ...
- 804 Views
- 3 replies
- 1 kudos
- 1 kudos
A fix to this issue was included in the v1.2.1 release of the '@databricks/aibi-client' library.
- 1 kudos
- 539 Views
- 2 replies
- 3 kudos
Resolved! Issue Reading DLT-Generated Materialized Views on DBR 18.x
We are experiencing an intermittent issue when reading Unity Catalog materialized views from downstream transformation notebooks.EnvironmentAzure DatabricksUnity Catalog enabledMaterialized views and streaming tables are created by a Lakeflow (former...
- 539 Views
- 2 replies
- 3 kudos
- 3 kudos
Adding a small practical note to Nina's answer, since your policy uses auto:latest-lts.The quickest way to apply the 17.3 LTS workaround without hardcoding a version key is to switch the policy to auto:prev-lts. It maps to the second-latest LTS runti...
- 3 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 586 | |
| 349 | |
| 287 |
