cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Administration & Architecture
Explore discussions on Databricks administration, deployment strategies, and architectural best practices. Connect with administrators and architects to optimize your Databricks environment for performance, scalability, and security.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Lakehouse Architecture


Forum Posts

Start a conversation
lizou8
by • New Contributor II
  • 20 Views
  • 1 replies
  • 0 kudos

delta sharing issue after enable predictive optimization

Some of our delta sharing tables are not working May be related to this, or maybe not, we enabled predictive optimization on all tables a few days agoes not working any morebut any new tables created works fine after setting thisSET TBLPROPERTIES (de...

Administration & Architecture
Reply
  • 20 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
lizou8
New Contributor II
  • 0 kudos

errorJsonParseException: Unexpected character ('k' (code 107)): was expecting comma to separate Object entries at [Source: (String)"{"type":"struct","fields":[{"name":"UserKey","type":"long","nullable":false,"metadata":{"comment":"{"key":"primary_key...

  • 0 kudos
JonLaRose
by • New Contributor III
  • 2277 Views
  • 4 replies
  • 1 kudos

Delta Lake S3 multi-cluster writes - DynamoDB

Hi there!I'm trying to figure out how the multi-writers architecture for Delta Lake tables is implemented under the hood.I understand that a DynamoDB table is used to provide mutual exclusion, but the question is: where is the table located? Is it in...

Administration & Architecture
Reply
  • 2277 Views
  • 4 replies
  • 1 kudos
View Replies
Latest Reply
prem14f
Visitor
  • 1 kudos

Hi, could you please help me here? How can i use this configuration in DataBricks? So I will maintain my transcription logs there, and with Parallel, I can use the Delta-RS job.spark.conf.set("spark.delta.logStore.s3a.impl", "io.delta.storage.S3Dynam...

  • 1 kudos
3 More Replies
SumedhPuri
by • New Contributor
  • 309 Views
  • 2 replies
  • 0 kudos

Technical Architecture - Feedback

Hello MembersI have designed a Technical Architecture (image attached). I would like some feedback on the current design (especially from 5.1 and onwards) and maybe some more ideas or anything else I can use instead of Azure Service Bus and Cosmos DB...

Administration & Architecture
Reply
  • 309 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
Schofield
Visitor
  • 0 kudos

In the step 3 you will want to consider using Databricks Workflows for orchestration.  The ADF databricks notebook activity is not actively developed by microsoft and the API it uses is legacy by Databricks So neither vendor  is actively supporting t...

  • 0 kudos
1 More Replies
sparrap
by • New Contributor
  • 58 Views
  • 1 replies
  • 0 kudos

Error when Connecting Databricks Cluster to RStudio Desktop App

Hi! I am trying to connect RStudio to my Databricks Cluster, I already change the permissions to CAN MANAGE and CAN ATTACH to the cluster. Also I have verified to have the correct python version and Databricks version in my computer.This is the error...

Administration & Architecture
Reply
  • 58 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @sparrap,  Databricks Connect for R currently supports Databricks personal access tokens for authentication. If you don’t have one, create one in your Databricks workspace.In RStudio Desktop, install the sparklyr and pysparklyr packages from CRAN....

  • 0 kudos
Witold
by • New Contributor III
  • 85 Views
  • 1 replies
  • 1 kudos

Resolved! Databricks runtime and Java Runtime

The Databricks runtime is shipped with two Java Runtimes: JRE 8 and JRE 17. While the first one is used by default, you can use the environment variable JNAME to specify the other JRE: JNAME: zulu17-ca-amd64.FWIW, AFAIK JNAME is available since DBR 1...

Administration & Architecture
Reply
  • 85 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
Kaniz_Fatma
Community Manager
  • 1 kudos

Hi @Witold,  Databricks Strategy on Java Runtimes: Databricks provides flexibility by shipping both JRE 8 and JRE 17 with the Databricks runtime. While JRE 8 is the default, you can specify the other JRE using the environment variable JNAME. Reason...

  • 1 kudos
yvishal519
by • New Contributor III
  • 1356 Views
  • 4 replies
  • 0 kudos

Resolved! Help Needed: Errors with df.display() and df.show() in Databricks

Dear Databricks Community,I am reaching out to you for assistance with some issues I'm encountering in my Databricks environment. I'm hoping the community can provide some guidance to help me resolve these problems.1. Error with df.display(): When I ...

Administration & Architecture
Reply
  • 1356 Views
  • 4 replies
  • 0 kudos
View Replies
Latest Reply
yvishal519
New Contributor III
  • 0 kudos

Dear Databricks Community,I wanted to share some updates regarding the issues I've been encountering in my Databricks environment.After raising a ticket with Microsoft and collaborating with their team for approximately a week, we undertook several t...

  • 0 kudos
3 More Replies
sukanya09
by • New Contributor II
  • 81 Views
  • 2 replies
  • 1 kudos

Capture error for databricks job

Greetings ! We have created a Databricks job using Notebook. This notebook has 6 cells . Can we capture the Success and failure (along with error)  and Store it into for monitoring and analysis . Ex if we want to capture the below error   

sukanya09_0-1721831111079.png
Administration & Architecture
Reply
  • 81 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
Kaniz_Fatma
Community Manager
  • 1 kudos

Hi @sukanya09,  Thank you for reaching out to our community! We're here to help you. To ensure we provide you with the best support, could you please take a moment to review the response and choose the one that best answers your question? Your feedba...

  • 1 kudos
1 More Replies
erigaud
by • Honored Contributor
  • 184 Views
  • 3 replies
  • 1 kudos

Asset bundle yml factorization

Hello,I have a project using asset bundle in which I have several jobs using roughly the same job definition (tags and job clusters definitions are always the same) Is there a way to put everything in common in a yml file and reuse that in each indiv...

Administration & Architecture
Reply
  • 184 Views
  • 3 replies
  • 1 kudos
View Replies
Latest Reply
Witold
New Contributor III
  • 1 kudos

@erigaud What might work, I actually never tried it by myself so far, is this:Define your complex variables in a separate yaml file (complex variables are supported since v0.222.0), import this file using include, and reference these variables accord...

  • 1 kudos
2 More Replies
anishr
by • New Contributor II
  • 149 Views
  • 3 replies
  • 1 kudos

Account Verification Code

There's a recent change in the way users in my company workspace now log in to Databricks.When logging into an instance I get the prompt to check my email with a OTP that can be used to login after entering the password.Email body as follows:Account ...

Administration & Architecture
Reply
  • 149 Views
  • 3 replies
  • 1 kudos
View Replies
Latest Reply
121558
New Contributor II
  • 1 kudos

Hi @Kaniz_Fatma Is there a way to disable signing in with email OTPs and continue using only passwords? If so, please provide the steps. Thanks!

  • 1 kudos
2 More Replies
Avvar2022
by • Contributor
  • 79 Views
  • 2 replies
  • 0 kudos

why is recommended default setting is delta deleted file duration to be 7 days?

Due to frequent updates on table our backend storage size is growing a lot, even though we have vacuum and optimize scheduled unable to clean up files 7 days or less.Current settings is: delta.logRetentionDuration="interval 7 days" and deleted files ...

Administration & Architecture
Reply
  • 79 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
Witold
New Contributor III
  • 0 kudos

It's not really a recommendation per se, it's basically a default, which you simply need. And yes, it's supposed to be adapted to your specific needs.In this case:I don't need delta log more than one day.If you're fine that you won't be able to rollb...

  • 0 kudos
1 More Replies
ziad
by • New Contributor
  • 389 Views
  • 4 replies
  • 1 kudos

Resolved! Creating Azure Databricks Workspace Without NAT Gateway

Hello,Recently, when I create a new Databricks workspace on Azure, it automatically create a NAT Gateway which incurs additional cost !When creating the workspace, I don't choose secure cluster connectivity, so I'm expecting not to have a NAT Gateway...

nat 1.jpg nat 2.jpg
Administration & Architecture
Reply
  • 389 Views
  • 4 replies
  • 1 kudos
View Replies
Latest Reply
Slash
New Contributor II
  • 1 kudos

Hi @ziad ,Do you create workspace with secure cluster connectivity? According to the documentation: If you use secure cluster connectivity with the default VNet that Azure Databricks creates, Azure Databricks automatically creates a NAT gateway for o...

  • 1 kudos
3 More Replies
Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!

Labels