- 2581 Views
- 2 replies
- 0 kudos
Resolved! databricks asset bundles on azure devops
@herewe are able to work on DAB locally or Github actions, but looks from azure devops end lot of missing pieces in terms of pipeline and examples. can any one point me out github repo that is good to test azure devops pipeline for DAB.
- 2581 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @karthik_p , Certainly! When it comes to setting up Azure DevOps pipelines for Data API Builder (DAB), it’s essential to find a reliable example repository. Here are a few GitHub repositories that you might find helpful for testing Azure DevOps...
- 0 kudos
- 1530 Views
- 3 replies
- 2 kudos
DLT Compute: "Ephemeral" Job Compute vs. All-purpose compute
Hi there, I would like to understand for DLT jobs if there is any way to get DLT jobs running in an *existing* (and currently running) All-purpose compute rather than spinning up an "ephemeral" (not yet initialized) Job Compute?
- 1530 Views
- 3 replies
- 2 kudos
- 2 kudos
Hi @ChristianRRL , you cannot run a DLT pipeline in an All-purpose compute cluster. DLT Clusters are managed resources. Here are some details on how to configure a DLT cluster: https://docs.databricks.com/en/delta-live-tables/settings.html#configure-...
- 2 kudos
- 1628 Views
- 4 replies
- 1 kudos
Materialized Views Without DLT?
I'm curious, is DLT *required* to use Materialized Views in Databricks? Is it not possible to create and refresh a Materialized view via a standard Databricks Workflow?
- 1628 Views
- 4 replies
- 1 kudos
- 1 kudos
For additional clarity, I would like to show that when I attempt to run this in the SQL Editor with the SQL Starter Warehouse, I get an error because we do not have UC enabled. But I'm wanting to get the Materialized View functionality without fully ...
- 1 kudos
- 7984 Views
- 4 replies
- 4 kudos
org.apache.spark.SparkException: Job aborted due to stage failure:
HiI have around 20 million records in my DF, and want to save it in HORIZINTAL SQL DB.This is error:org.apache.spark.SparkException: Job aborted due to stage failure: A shuffle map stage with indeterminate output was failed and retried. However, Spar...
- 7984 Views
- 4 replies
- 4 kudos
- 4 kudos
If there are any failures which may lead to a stage retry, but retrying the stage translates into potentially having an inconsistent result (indeterminacy) then this exception is raised. The exception is raised in newer version where the validation i...
- 4 kudos
- 351 Views
- 0 replies
- 0 kudos
Identifying the Dependency Automatically on the SQL Statements
Hi All,We have a SQL script file with lots of create, insert, and SQL statements. We need to figure out how to identify the dependencies between the statements dynamically(by using either spark/sql or using any program).For example - First, we need t...
- 351 Views
- 0 replies
- 0 kudos
- 1342 Views
- 2 replies
- 1 kudos
Issue with Creating and Running Databricks Jobs with new databricks cli v0.214.0
Hi Databricks Support,I'm encountering an issue with creating and running jobs on Databricks. Here are the details:Problem Description:When attempting to create and run a job using the old JSON (which was successfully used to create and run jobs usin...
- 1342 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @Kaniz_Fatma ,Thanks for the reference links for the solution.I found the solution mentioned in this https://github.com/databricks/databricks-sdk-go/discussions/384 GitHub. By using the get-run API, I was able to retrieve the running status of my...
- 1 kudos
- 979 Views
- 1 replies
- 1 kudos
dbdemo LLAM CHATBOT RAG
i have an issue when running the below code using the default dbdemos in the advanced preparation , i have reduced the chunk_size and max_batch_size and running the code in a proper compute resources , could anyone help on that please :(spark.readStr...
- 979 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @hossam, Enable detailed logging to get more information about the failure. Look for additional error messages or stack traces.If possible, run the code in a local development environment (outside Databricks) to isolate the issue.
- 1 kudos
- 1644 Views
- 4 replies
- 1 kudos
"AWS S3 resource has been disabled" error on job, not appearing on notebook
I am getting an "INTERNAL_ERROR" on a databricks job submitted through the API. Which says:"Run result unavailable: run failed with error message All access to AWS S3 resource has been disabled"However, when I click on the notebook created by the job...
- 1644 Views
- 4 replies
- 1 kudos
- 1 kudos
@Kaniz_Fatma In the s3 logs of the run, I am seeing this:24/05/29 06:39:30 WARN FileSystem: Failed to initialize fileystem dbfs:///: java.io.FileNotFoundException: Bucket user-workspace-s3-bucket does not exist24/05/29 06:39:30 ERROR DbfsHadoop3: Fai...
- 1 kudos
- 1377 Views
- 1 replies
- 2 kudos
DAIS 2023
I am very excited to learn more about DLT and spark performance tuning at the summit. LakehouseIQ from the summit is very exciting and looking forward to implementing.
- 1377 Views
- 1 replies
- 2 kudos
- 2 kudos
Hi @Harrison, We're thrilled to hear that you had a great experience at DAIS 2023! Your feedback is valuable to us, and we appreciate you taking the time to share it on the community platform. We wanted to let you know that the Databricks Community T...
- 2 kudos
- 863 Views
- 2 replies
- 0 kudos
- 863 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @DAIs2023 and @Ag726 , We're thrilled to hear that you had a great experience at DAIS 2023! Your feedback is valuable to us, and we appreciate you taking the time to share it on the community platform. We wanted to let you know that the Databricks...
- 0 kudos
- 476 Views
- 1 replies
- 1 kudos
Loving Databricks Summit
Loving Databricks SQL! Looking to learn more how to integrate into our current infrastructure as we built in Cloud. Currently part of Data Modernization team building new pipelines to pull in data from multiple raw sources.
- 476 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @JackieD, We're thrilled to hear that you had a great experience at DAIS 2023! Your feedback is valuable to us, and we appreciate you taking the time to share it on the community platform. We wanted to let you know that the Databricks Community Te...
- 1 kudos
- 574 Views
- 1 replies
- 1 kudos
- 574 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @weihey2017, We're thrilled to hear that you had a great experience at DAIS 2023! Your feedback is valuable to us, and we appreciate you taking the time to share it on the community platform. We wanted to let you know that the Databricks Communi...
- 1 kudos
- 553 Views
- 1 replies
- 0 kudos
At Data+AI summit
At the summit- it’s a blast
- 553 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @RobertrheaUK, We're thrilled to hear that you had a great experience at DAIS 2023! Your feedback is valuable to us, and we appreciate you taking the time to share it on the community platform. We wanted to let you know that the Databricks Commu...
- 0 kudos
- 721 Views
- 1 replies
- 0 kudos
Unity Catalog - Quality - Monitor error
Monitor errorAn error occurred while configuring your monitor for this table:Error while creating dashboard for unity-catalog-xxx: com.databricks.api.base.DatabricksServiceException: INTERNAL_ERROR: An internal error occurredPlease delete and recreat...
- 721 Views
- 1 replies
- 0 kudos
- 0 kudos
If you have too many dashboards, there's a chance that the workspace reached the quota. I recommend you contacting Databricks Support for a more in-depth analysis.
- 0 kudos
- 1388 Views
- 2 replies
- 0 kudos
count or toPandas taking too long
Hi,I am fetching data from unity catalog from notebooks using spark.sql(). The query takes just a few seconds - I am actually trying to retrieving 2 rows - but some operations like count() or toPandas() take forever. I wonder why does it take so long...
- 1388 Views
- 2 replies
- 0 kudos
- 0 kudos
Hey @jimcast how are you? You can check the internals and have a good hint of what's happening using the SparkUI. Filter and select the jobs that are taking the longest and check what is being requested on the SQL/Data Frame tab, as well as their pla...
- 0 kudos
