cancel
Showing results for 
Search instead for 
Did you mean: 
Administration & Architecture
Explore discussions on Databricks administration, deployment strategies, and architectural best practices. Connect with administrators and architects to optimize your Databricks environment for performance, scalability, and security.
cancel
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Lakehouse Architecture


Forum Posts

gabo2023
by New Contributor III
  • 2460 Views
  • 2 replies
  • 3 kudos

Unable to read resources - Unsupported Protocol Scheme (Terraform AWS)

Hello everyone!Over the last few weeks my company has been trying to deploy a Databricks workspace on AWS adapted to the customer's needs, using Terraform. To do this, we started from a base code on Databricks own github (https://github.com/databrick...

image.png image (1).png
  • 2460 Views
  • 2 replies
  • 3 kudos
Latest Reply
meeran007
New Contributor II
  • 3 kudos

Whats the solution for this? facing same issue.

  • 3 kudos
1 More Replies
pawelzak
by New Contributor III
  • 2538 Views
  • 2 replies
  • 0 kudos

Databricks dashboard programatically

Hi,How can I create a databricks dashboard, filters and visuals programatically (api, terraform, sdk, cli...)?Thanks,Pawel

  • 2538 Views
  • 2 replies
  • 0 kudos
Latest Reply
marcin-sg
New Contributor III
  • 0 kudos

Maybe slightly late (maybe because development was late :P), but hopefully it will also help other.1. There seems to be support added to the newest terraform databricks provider - 1.49.0 - here2. Other solution would be to use databricks cli (e.g. `d...

  • 0 kudos
1 More Replies
Schofield
by New Contributor III
  • 559 Views
  • 1 replies
  • 2 kudos

List deployed Databricks asset bundles (DABs)?

Is there a databricks cli command or REST API to list all the DABs that have been deployed to a workspace?

  • 559 Views
  • 1 replies
  • 2 kudos
Latest Reply
szymon_dybczak
Contributor III
  • 2 kudos

Hi @Schofield ,Unfortunately, I don't think there is out of the box command that will provide you this information yet. As a workaround, you can try write some code that will extract this information from REST API. For example, you can use /api/2.1/j...

  • 2 kudos
lizou8
by New Contributor III
  • 791 Views
  • 2 replies
  • 2 kudos

Resolved! delta sharing issue after enable predictive optimization

Some of our delta sharing tables are not working May be related to this, or maybe not, we enabled predictive optimization on all tables a few days agoes not working any morebut any new tables created works fine after setting thisSET TBLPROPERTIES (de...

  • 791 Views
  • 2 replies
  • 2 kudos
Latest Reply
lizou8
New Contributor III
  • 2 kudos

after some debugging, I find out a very unique cause if we used a JSON string in column comment, and it make sense that a JSON string in column comment breaks delta sharingexample: column COMMENT  {"key": "primary_key", "is_identity": "true"}The erro...

  • 2 kudos
1 More Replies
JonLaRose
by New Contributor III
  • 3037 Views
  • 2 replies
  • 1 kudos

Delta Lake S3 multi-cluster writes - DynamoDB

Hi there!I'm trying to figure out how the multi-writers architecture for Delta Lake tables is implemented under the hood.I understand that a DynamoDB table is used to provide mutual exclusion, but the question is: where is the table located? Is it in...

  • 3037 Views
  • 2 replies
  • 1 kudos
Latest Reply
prem14f
New Contributor II
  • 1 kudos

Hi, could you please help me here? How can i use this configuration in DataBricks? So I will maintain my transcription logs there, and with Parallel, I can use the Delta-RS job.spark.conf.set("spark.delta.logStore.s3a.impl", "io.delta.storage.S3Dynam...

  • 1 kudos
1 More Replies
SumedhPuri
by New Contributor
  • 821 Views
  • 1 replies
  • 0 kudos

Technical Architecture - Feedback

Hello MembersI have designed a Technical Architecture (image attached). I would like some feedback on the current design (especially from 5.1 and onwards) and maybe some more ideas or anything else I can use instead of Azure Service Bus and Cosmos DB...

  • 821 Views
  • 1 replies
  • 0 kudos
Latest Reply
Schofield
New Contributor III
  • 0 kudos

In the step 3 you will want to consider using Databricks Workflows for orchestration.  The ADF databricks notebook activity is not actively developed by microsoft and the API it uses is legacy by Databricks So neither vendor  is actively supporting t...

  • 0 kudos
yvishal519
by Contributor
  • 15263 Views
  • 4 replies
  • 0 kudos

Resolved! Help Needed: Errors with df.display() and df.show() in Databricks

Dear Databricks Community,I am reaching out to you for assistance with some issues I'm encountering in my Databricks environment. I'm hoping the community can provide some guidance to help me resolve these problems.1. Error with df.display(): When I ...

  • 15263 Views
  • 4 replies
  • 0 kudos
Latest Reply
yvishal519
Contributor
  • 0 kudos

Dear Databricks Community,I wanted to share some updates regarding the issues I've been encountering in my Databricks environment.After raising a ticket with Microsoft and collaborating with their team for approximately a week, we undertook several t...

  • 0 kudos
3 More Replies
erigaud
by Honored Contributor
  • 3084 Views
  • 2 replies
  • 0 kudos

Asset bundle yml factorization

Hello,I have a project using asset bundle in which I have several jobs using roughly the same job definition (tags and job clusters definitions are always the same) Is there a way to put everything in common in a yml file and reuse that in each indiv...

  • 3084 Views
  • 2 replies
  • 0 kudos
Latest Reply
Witold
Honored Contributor
  • 0 kudos

@erigaud What might work, I actually never tried it by myself so far, is this:Define your complex variables in a separate yaml file (complex variables are supported since v0.222.0), import this file using include, and reference these variables accord...

  • 0 kudos
1 More Replies
sukanya09
by New Contributor II
  • 409 Views
  • 1 replies
  • 1 kudos

Capture error for databricks job

Greetings ! We have created a Databricks job using Notebook. This notebook has 6 cells . Can we capture the Success and failure (along with error)  and Store it into for monitoring and analysis . Ex if we want to capture the below error   

sukanya09_0-1721831111079.png
  • 409 Views
  • 1 replies
  • 1 kudos
Latest Reply
szymon_dybczak
Contributor III
  • 1 kudos

Hi @sukanya09 ,You can use jobs API, each run will have information about status of teach task in the jobhttps://docs.databricks.com/api/workspace/jobs/getrunoutput

  • 1 kudos
anishr
by New Contributor II
  • 1026 Views
  • 2 replies
  • 0 kudos

Account Verification Code

There's a recent change in the way users in my company workspace now log in to Databricks.When logging into an instance I get the prompt to check my email with a OTP that can be used to login after entering the password.Email body as follows:Account ...

  • 1026 Views
  • 2 replies
  • 0 kudos
Latest Reply
121558
New Contributor II
  • 0 kudos

Hi @Retired_mod Is there a way to disable signing in with email OTPs and continue using only passwords? If so, please provide the steps. Thanks!

  • 0 kudos
1 More Replies
Avvar2022
by Contributor
  • 469 Views
  • 2 replies
  • 0 kudos

why is recommended default setting is delta deleted file duration to be 7 days?

Due to frequent updates on table our backend storage size is growing a lot, even though we have vacuum and optimize scheduled unable to clean up files 7 days or less.Current settings is: delta.logRetentionDuration="interval 7 days" and deleted files ...

  • 469 Views
  • 2 replies
  • 0 kudos
Latest Reply
Witold
Honored Contributor
  • 0 kudos

It's not really a recommendation per se, it's basically a default, which you simply need. And yes, it's supposed to be adapted to your specific needs.In this case:I don't need delta log more than one day.If you're fine that you won't be able to rollb...

  • 0 kudos
1 More Replies
ziad
by New Contributor III
  • 1650 Views
  • 4 replies
  • 1 kudos

Resolved! Creating Azure Databricks Workspace Without NAT Gateway

Hello,Recently, when I create a new Databricks workspace on Azure, it automatically create a NAT Gateway which incurs additional cost !When creating the workspace, I don't choose secure cluster connectivity, so I'm expecting not to have a NAT Gateway...

nat 1.jpg nat 2.jpg
  • 1650 Views
  • 4 replies
  • 1 kudos
Latest Reply
szymon_dybczak
Contributor III
  • 1 kudos

Hi @ziad ,Do you create workspace with secure cluster connectivity? According to the documentation: If you use secure cluster connectivity with the default VNet that Azure Databricks creates, Azure Databricks automatically creates a NAT gateway for o...

  • 1 kudos
3 More Replies
m997al
by Contributor III
  • 3622 Views
  • 1 replies
  • 0 kudos

Cannot manage permissions for Databricks workspace disconnected from metastore

So in the Databricks Account console, under "Workspaces", I am trying to configure permissions for a specific workspace that was formerly attached to a metastore, but the metastore is now deleted (on purpose).No errors show up in the Account console ...

  • 3622 Views
  • 1 replies
  • 0 kudos
Latest Reply
" src="" />
This widget could not be displayed.
This widget could not be displayed.
This widget could not be displayed.
  • 0 kudos

This widget could not be displayed.
So in the Databricks Account console, under "Workspaces", I am trying to configure permissions for a specific workspace that was formerly attached to a metastore, but the metastore is now deleted (on purpose).No errors show up in the Account console ...

This widget could not be displayed.
  • 0 kudos
This widget could not be displayed.

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels