- 599 Views
- 1 replies
- 0 kudos
Handling Over-Usage of Capacity in Databricks Jobs/Processes
Hi all,Is there a tool or method in Databricks to ensure data integrity and stability when a job or process exceeds the allocated capacity? Specifically, I’m looking for ways to:Prevent failures or data loss due to resource overuse.Automatically scal...
- 599 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @smanda88 - For point 1, please see; https://docs.databricks.com/en/lakehouse-architecture/reliability/best-practices.html For 2, you can use auto-scaling, please refer to: https://docs.databricks.com/en/lakehouse-architecture/cost-optimization...
- 0 kudos
- 579 Views
- 1 replies
- 0 kudos
Where to find Jupyter Notebook course materials for Get Started with Databricks for Generative AI
Hello, I can't seem to find any way to gain access to the Jupyter Notebook demo source of "Get Started with Databricks for Generative AI" course. Please help. Thank you kindly in advance.
- 579 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @nathanmle! We are sorry to inform you that we are no longer offering notebooks or the DBC files for the self-paced courses due to recent changes.If you’re interested in working on labs in a provided Databricks environment, you can purchase the...
- 0 kudos
- 3071 Views
- 0 replies
- 0 kudos
Databricks CE - Where is the quickstart tutorial?
Hello! I was looking through Databricks tutorials online, but my interface looks different from many of the videos I'm seeing. What happened to the Quickstart tutorials on the home page? Are they no longer available on the dashboard?
- 3071 Views
- 0 replies
- 0 kudos
- 1977 Views
- 1 replies
- 2 kudos
Databricks asset bundles dependencies
Is anyone aware of a way to include a requirements.txt within the job definition of a databricks asset bundle? Documentation mentions how to have dependencies in workspace files, or Unity Catalog volumes, but I wanted to ask if it is possible to decl...
- 1977 Views
- 1 replies
- 2 kudos
- 2 kudos
I have the same question.
- 2 kudos
- 3378 Views
- 1 replies
- 0 kudos
Error uploading files to a Unity Catalog volume in Databricks
Hi everyone,I'm developing an API in Flask that interacts with Databricks to upload files to a Unity Catalog volume, but I'm encountering the following error:{"error_code": "ENDPOINT_NOT_FOUND", "message": "No se encontró API para 'POST /unity-catalo...
- 3378 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @EngHol, This endpoint: /api/2.0/unity-catalog/volumes/upload is not a valid one, hence the issue. Looking at the API for volumes, unfortunately there is no way to upload to a volume: https://docs.databricks.com/api/workspace/volumes
- 0 kudos
- 847 Views
- 2 replies
- 0 kudos
Hide function definition in Unity catalog
Hi ,I have created a function to anonymize user id using secret.I want to give access of this function to other users so they can execute it without giving access to the secret .Is it possible in databricks? I have tested it and see user is not able ...
- 847 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @NehaR, I am afraid it might not be possible without giving secret access to the users. Another approach would be to use a Service Principal.
- 0 kudos
- 3389 Views
- 0 replies
- 0 kudos
CloudFormation Stack Failure: Custom::CreateWorkspace in CREATE_FAILED State
I am trying to create a workspace using AWS CloudFormation, but the stack fails with the following error:"The resource CreateWorkspace is in a CREATE_FAILED state. This Custom::CreateWorkspace resource is in a CREATE_FAILED state. Received response s...
- 3389 Views
- 0 replies
- 0 kudos
- 969 Views
- 1 replies
- 1 kudos
Resolved! Container lifetime?
When launching a job via "Create and trigger a one-time run" (docs), when using a custom image (docs), what's the lifetime of the container? Does it create the cluster, start the container, run the job, then terminate the container? Or does the runni...
- 969 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @mrstevegross Cluster Creation: When you submit a job using the "Create and trigger a one-time run" API, a new cluster is created if one is not specified.Container Start: The custom Docker image specified in the cluster configuration is us...
- 1 kudos
- 3336 Views
- 0 replies
- 0 kudos
DLT detecting changes but not applying them
We have three source tables used for a streaming dimension table in silver. Around 50K records are changed in one of the source tables, and the DLT pipeline shows that it has updated those 50K records, but they remain unchanged. The only way to pick ...
- 3336 Views
- 0 replies
- 0 kudos
- 887 Views
- 3 replies
- 0 kudos
No confirmation mail received after scheduling the exam
Hi team, I have scheduled my Databricks Data Engineer Associate exam for 12th Feb 2025 using the below mail id, but I still have not received any confirmation mail there. I have checked spam folder too.Could you please resend it to barnitac@kpmg.com ...
- 887 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi team, I have cleared my exam today. Unfortunately I have not received a single mail either to confirm my exam or to confirm test completion and result. @Cert-Team
- 0 kudos
- 967 Views
- 2 replies
- 1 kudos
Partnership with Databricks
Hello,What are the pre-requisites to become Databricks partner?
- 967 Views
- 2 replies
- 1 kudos
- 1 kudos
Hello @Bala_K! For information on becoming a Databricks partner, please email partnerops@Databricks.com. They can guide you through the prerequisites and next steps.
- 1 kudos
- 3167 Views
- 0 replies
- 0 kudos
How to Define Constants at Bundle Level in Databricks Asset Bundles for Use in Notebooks?
I'm working with Databricks Asset Bundles and need to define constants at the bundle level based on the target environment. These constants will be used inside Databricks notebooks.For example, I want a constant gold_catalog to take different values ...
- 3167 Views
- 0 replies
- 0 kudos
- 1781 Views
- 3 replies
- 0 kudos
Getting access token error when connecting from azure databricks to GCS bucket
I am creating a data frame by reading a table's data residing in Azure backed unity catalog. I need to write the dataframe or file to GCS bucket. I have configured the spark cluster config using the GCP service account json values. Also tried uploadi...
- 1781 Views
- 3 replies
- 0 kudos
- 0 kudos
@kiko_roy unfortunately that didnt' work. the error is stating its trying to get the access token from metadata server, I wonder why from the metadata server?
- 0 kudos
- 806 Views
- 2 replies
- 0 kudos
How to fetch nested data structures in Databricks using JDBC
I've asked the question also here on stack overflow When using nested data structures in Databricks (e.g. `ARRAY` or `ROW`) using JDBC, it appears that the results can be fetched as JSON `String` values, e.g.: try (Statement s = connection.createStat...
- 806 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @lukaseder, This looks to be a bug, but will get more details about it internally. The driver is relatively new. Have you tried with another version, for instance JDBC: 2.6.40
- 0 kudos
- 6405 Views
- 2 replies
- 2 kudos
Model Serving Endpoint keeps failing with SIGKILL error
I am trying to deploy a model in the serving endpoints section, but it keeps failing after attempting to create for an hour. Here are the service logs:Container failed with: 9 +0000] [115] [INFO] Booting worker with pid: 115[2023-09-15 19:15:35 +0000...
- 6405 Views
- 2 replies
- 2 kudos
- 2 kudos
Hello @AChang, This is a common issue when the memory requirements of your model exceed the available memory on your current compute resources. Moving to a larger compute instance with more memory can help accommodate the memory requirements of yo...
- 2 kudos
