- 6111 Views
- 2 replies
- 1 kudos
Delta Lake S3 multi-cluster writes - DynamoDB
Hi there!I'm trying to figure out how the multi-writers architecture for Delta Lake tables is implemented under the hood.I understand that a DynamoDB table is used to provide mutual exclusion, but the question is: where is the table located? Is it in...
- 6111 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi, could you please help me here? How can i use this configuration in DataBricks? So I will maintain my transcription logs there, and with Parallel, I can use the Delta-RS job.spark.conf.set("spark.delta.logStore.s3a.impl", "io.delta.storage.S3Dynam...
- 1 kudos
- 6195 Views
- 1 replies
- 0 kudos
Technical Architecture - Feedback
Hello MembersI have designed a Technical Architecture (image attached). I would like some feedback on the current design (especially from 5.1 and onwards) and maybe some more ideas or anything else I can use instead of Azure Service Bus and Cosmos DB...
- 6195 Views
- 1 replies
- 0 kudos
- 0 kudos
In the step 3 you will want to consider using Databricks Workflows for orchestration. The ADF databricks notebook activity is not actively developed by microsoft and the API it uses is legacy by Databricks So neither vendor is actively supporting t...
- 0 kudos
- 895 Views
- 0 replies
- 0 kudos
- 895 Views
- 0 replies
- 0 kudos
- 24423 Views
- 4 replies
- 0 kudos
Resolved! Help Needed: Errors with df.display() and df.show() in Databricks
Dear Databricks Community,I am reaching out to you for assistance with some issues I'm encountering in my Databricks environment. I'm hoping the community can provide some guidance to help me resolve these problems.1. Error with df.display(): When I ...
- 24423 Views
- 4 replies
- 0 kudos
- 0 kudos
Dear Databricks Community,I wanted to share some updates regarding the issues I've been encountering in my Databricks environment.After raising a ticket with Microsoft and collaborating with their team for approximately a week, we undertook several t...
- 0 kudos
- 4260 Views
- 2 replies
- 0 kudos
Asset bundle yml factorization
Hello,I have a project using asset bundle in which I have several jobs using roughly the same job definition (tags and job clusters definitions are always the same) Is there a way to put everything in common in a yml file and reuse that in each indiv...
- 4260 Views
- 2 replies
- 0 kudos
- 0 kudos
@erigaud What might work, I actually never tried it by myself so far, is this:Define your complex variables in a separate yaml file (complex variables are supported since v0.222.0), import this file using include, and reference these variables accord...
- 0 kudos
- 1318 Views
- 1 replies
- 1 kudos
Capture error for databricks job
Greetings ! We have created a Databricks job using Notebook. This notebook has 6 cells . Can we capture the Success and failure (along with error) and Store it into for monitoring and analysis . Ex if we want to capture the below error
- 1318 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @sukanya09 ,You can use jobs API, each run will have information about status of teach task in the jobhttps://docs.databricks.com/api/workspace/jobs/getrunoutput
- 1 kudos
- 3617 Views
- 2 replies
- 0 kudos
Account Verification Code
There's a recent change in the way users in my company workspace now log in to Databricks.When logging into an instance I get the prompt to check my email with a OTP that can be used to login after entering the password.Email body as follows:Account ...
- 3617 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @Retired_mod Is there a way to disable signing in with email OTPs and continue using only passwords? If so, please provide the steps. Thanks!
- 0 kudos
- 1949 Views
- 2 replies
- 0 kudos
why is recommended default setting is delta deleted file duration to be 7 days?
Due to frequent updates on table our backend storage size is growing a lot, even though we have vacuum and optimize scheduled unable to clean up files 7 days or less.Current settings is: delta.logRetentionDuration="interval 7 days" and deleted files ...
- 1949 Views
- 2 replies
- 0 kudos
- 0 kudos
It's not really a recommendation per se, it's basically a default, which you simply need. And yes, it's supposed to be adapted to your specific needs.In this case:I don't need delta log more than one day.If you're fine that you won't be able to rollb...
- 0 kudos
- 6382 Views
- 1 replies
- 0 kudos
Cannot manage permissions for Databricks workspace disconnected from metastore
So in the Databricks Account console, under "Workspaces", I am trying to configure permissions for a specific workspace that was formerly attached to a metastore, but the metastore is now deleted (on purpose).No errors show up in the Account console ...
- 6382 Views
- 1 replies
- 0 kudos
- 1397 Views
- 2 replies
- 0 kudos
workspace creation with public IP enabled ( "is_no_public_ip_enabled": false )
Hi All,Am trying to create a workspace on my account which has premium subscription managing through aws market place with is_no_public_ip_enabled": false and am getting below mentioned error.MALFORMED_REQUEST: workspace creation with public IP enabl...
- 1397 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @itsakhil , Thank you for reaching out to our community! We're here to help you. To ensure we provide you with the best support, could you please take a moment to review the response and choose the one that best answers your question? Your feedba...
- 0 kudos
- 3245 Views
- 2 replies
- 0 kudos
Disable Vector Search
Hello all,Is there a way to disable Vector Search? It's not vetted in my organization and I don't want the users to be able to create it.Best,Pawel
- 3245 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @Pawel_Pilkowski , Thank you for reaching out to our community! We're here to help you. To ensure we provide you with the best support, could you please take a moment to review the response and choose the one that best answers your question? Your...
- 0 kudos
- 7694 Views
- 2 replies
- 1 kudos
Databricks platform setup best practices in Azure
Could you please help me on the prerequisites, configuring steps and best practices for the data bricks platform and workspace setup in Azure(include subscription, subnet /network configuration ,storage configuration best practices etc).
- 7694 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @Phani1 , Thank you for reaching out to our community! We're here to help you. To ensure we provide you with the best support, could you please take a moment to review the response and choose the one that best answers your question? Your feedback...
- 1 kudos
- 13129 Views
- 6 replies
- 4 kudos
Different settings per target with Asset bundles
When generating the standard setup with databricks bundle init we will get databricks.yml that references resources/*. The targets are set in the databricks.yml and the resources (pipelines and jobs) are set in different files.I have dlt pipelines th...
- 13129 Views
- 6 replies
- 4 kudos
- 3310 Views
- 2 replies
- 1 kudos
Resolved! Using the API, get the list of the schemas and tables a group or user has permissions for
I am attempting to use the Databricks API to get a list of the schemas and tables a group or user has permissions for. Is this possible? Is there another method I should be using instead?I see the Unity Catalog > Grants > Get permissions endpoint c...
- 3310 Views
- 2 replies
- 1 kudos
- 1 kudos
Thanks, @Walter_C. In my case, I was able to get the data I needed by using the Databricks SQL Driver for Node.js , querying the information_schema.table_privileges table.
- 1 kudos
- 8549 Views
- 5 replies
- 0 kudos
Azure Entra SSO Error: Your user has not been registered
I have set up SSO within databricks and automatic user provisioning with Azure Entra and confirmed it is working for all users. However 1 user is presented with this when signing in. The user is in the enterprise app within Azure Entra and the user i...
- 8549 Views
- 5 replies
- 0 kudos
