cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
isultano
by • New Contributor II
  • 4361 Views
  • 4 replies
  • 1 kudos

Where did the settings button move to - where is the admin console?

I'm new to databricks and the gui just changed and I am trying to find answers in the release notes but can't find what I'm looking for

Data Engineering
Reply
  • 4361 Views
  • 4 replies
  • 1 kudos
View Replies
Latest Reply
isultano
New Contributor II
  • 1 kudos

I have found the Administrator Console. It is in the top right under my name. My ui is different.Thanks community for your support. I am a junior on this product.

  • 1 kudos
3 More Replies
HariharaSam
by • Contributor
  • 9667 Views
  • 3 replies
  • 4 kudos

Performance Tuning of Databricks Notebook

Hi Everyone ,I am trying to run a databricks notebook in parallel using ThreadPoolExecutor .Can anyone suggest how to reduce the time taken based on the below findings so far.Current Performance:Time taken - 25 minutes ThreadPoolExecutor max_workers ...

Data Engineering
Reply
  • 9667 Views
  • 3 replies
  • 4 kudos
View Replies
Latest Reply
Hubert-Dudek
Esteemed Contributor III
  • 4 kudos

ThreadPoolExecutor will not help as Databricks/Spark will process job by job.So please analyze in Spark UI what is consuming the most time.There are a lot of tips on how to optimize they depend on the dataset (size etc. transformations)Look for data ...

  • 4 kudos
2 More Replies
noimeta
by • Contributor III
  • 3503 Views
  • 2 replies
  • 4 kudos

Resolved! Adjust label size in SQL visualization

Hi,Anyone knows if there's a way to adjust the font size of label in the SQL visualization? Our dashboard users complain that it's too small.

Data Engineering
Reply
  • 3503 Views
  • 2 replies
  • 4 kudos
View Replies
Latest Reply
susodapop
Contributor
  • 4 kudos

It's on our radar to make this customisable in the future. For now, browser zoom as suggested by @debayan is the suggested workaround.

  • 4 kudos
1 More Replies
rahul08yadav
by • New Contributor III
  • 4333 Views
  • 2 replies
  • 4 kudos

Error while writing Data Frame from Azure Databricks to Azure synapse Dedicated Pool

I am reading delta tables from my Databricks workspace as DF and then I am trying to write this DF into Azure Synapse Dedicated SQL Pool, but I am getting error like:-Py4JJavaError: An error occurred while calling o1509.save. : com.databricks.spark.s...

Data Engineering
Reply
  • 4333 Views
  • 2 replies
  • 4 kudos
View Replies
Latest Reply
rahul08yadav
New Contributor III
  • 4 kudos

The issue was getting created because I was passing SAS Token for setting the configuration for Storage Account instead of Storage Access Key. It's working now without any issue..spark.conf.set("fs.azure.account.key.datalakestorage.dfs.core.windows.n...

  • 4 kudos
1 More Replies
newmetrocity
by • New Contributor
  • 661 Views
  • 0 replies
  • 0 kudos

The Nova City Islamabad, is an upcoming and emerging housing project in the federal capital. This project aims to target all budget genres of people o...

The Nova City Islamabad, is an upcoming and emerging housing project in the federal capital. This project aims to target all budget genres of people of Pakistan. In the future, this project is going to compete with the mega projects of Islamabad, Pak...

Data Engineering
Reply
  • 661 Views
  • 0 replies
  • 0 kudos
newmetrocity
by • New Contributor
  • 687 Views
  • 0 replies
  • 0 kudos

 The New Metro City Gujar khan Payment Plan has not yet been made public, but the developers are anticipated to do so shortly. All of the plots will b...

The New Metro City Gujar khan Payment Plan has not yet been made public, but the developers are anticipated to do so shortly. All of the plots will be within the budget and at reasonable pricing. The investment in this project will be simple and hass...

Data Engineering
Reply
  • 687 Views
  • 0 replies
  • 0 kudos
109005
by • New Contributor III
  • 3062 Views
  • 6 replies
  • 0 kudos

Does Databricks support proxy for BigQuery?

Hi team, we tried to use the proxy options for BigQuery Spark connector as mentioned in this documentation. However, we keep getting "connect timed out" error. The proxy host is working on our end. This made us wonder if by chance Databricks does not...

Data Engineering
Reply
  • 3062 Views
  • 6 replies
  • 0 kudos
View Replies
Latest Reply
Vidula
Honored Contributor
  • 0 kudos

Hi @Ayushi Pandey​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Than...

  • 0 kudos
5 More Replies
Laure_Decaudin
by • New Contributor
  • 2026 Views
  • 2 replies
  • 0 kudos

Order By disabled in sql dashboard

Hello, I have an issue, when I created my query, with an Order By, and my vizualization, everything was in the right order. Then, when I put it in my dashboard the order changed. As you can see with the pictures, when I clik on "edit visualization", ...

Data Engineering
Reply
  • 2026 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
Vidula
Honored Contributor
  • 0 kudos

Hi @Laure Decaudin​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Tha...

  • 0 kudos
1 More Replies
ossinova
by • Contributor II
  • 7403 Views
  • 4 replies
  • 1 kudos

ADF - Failed To Send Request to Databricks Cluster

I have noticed that my orchestrated pipelines (in ADF) sometimes fail due to this error:ErrorCode=FailedToReadFromAzureDatabricksDeltaLake,Failed to read from azure databricks delta lake. Error message : Failed to send request to Azure Databricks Clu...

Data Engineering
Reply
  • 7403 Views
  • 4 replies
  • 1 kudos
View Replies
Latest Reply
Vidula
Honored Contributor
  • 1 kudos

Hi @Oscar Dyremyhr​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Tha...

  • 1 kudos
3 More Replies
fritzwijaya
by • New Contributor II
  • 2766 Views
  • 5 replies
  • 2 kudos

Resolved! Databricks Premium for Personal

Hi Databricks Community,I want to setup premium cluster with multiple nodes to run some Spark workload for my personal project. Do Databricks offer​s premium upgrade for personal? How to do upgrade?​Thanks.​

Data Engineering
Reply
  • 2766 Views
  • 5 replies
  • 2 kudos
View Replies
Latest Reply
Vidula
Honored Contributor
  • 2 kudos

Hi @fritzwijaya​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks...

  • 2 kudos
4 More Replies
tanjil
by • New Contributor III
  • 8113 Views
  • 6 replies
  • 6 kudos

Resolved! Read and transform CSVs in parallel.

I need to read and transform several CSV files and then append them to a single data frame. I am able to do this in databricks using simple for loops, but I would like to speed this up.Below is the rough structure of my code: for filepath in all_file...

Data Engineering
Reply
  • 8113 Views
  • 6 replies
  • 6 kudos
View Replies
Latest Reply
Vidula
Honored Contributor
  • 6 kudos

Hi @tanjil​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks!

  • 6 kudos
5 More Replies
nemesis
by • New Contributor
  • 1476 Views
  • 2 replies
  • 0 kudos

Same Public ip of two machines in workspace.

Is there any way to change public ip of driver node? when I try to launch two different instance (single node), I see those two machine have the same public ip.I'm writing this command into notebook of each instance. I got the same ip.from requests i...

Data Engineering
Reply
  • 1476 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
Vidula
Honored Contributor
  • 0 kudos

Hi @Hakan Tekbas​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thank...

  • 0 kudos
1 More Replies
deisou
by • New Contributor
  • 2713 Views
  • 1 replies
  • 0 kudos

How to change Databricks Academy username(Email)?

Is there a way to change my Databricks Academy username(email)? It is greyed out in my profile and I cannot update it. How do I go about getting it updated?

Data Engineering
Reply
  • 2713 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Debayan
Databricks Employee
  • 0 kudos

Hi, Please go through the Databricks Academy FAQs here: https://files.training.databricks.com/lms/docebo/databricks-academy-faq.pdfAlso, please go through the post here: https://community.databricks.com/s/feed/0D53f00001dq6W6CAI.

  • 0 kudos
Gim
by • Contributor
  • 1732 Views
  • 1 replies
  • 1 kudos

Incremental loading from Delta tables to MySQL database

I was thinking of using Azure Data Factory to orchestrate this. But is it possible to do incremental loads from tables in Delta format going to MySQL database? My Delta table sources will be coming from ADLS storage. Would I need to convert them firs...

Data Engineering
Reply
  • 1732 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
Debayan
Databricks Employee
  • 1 kudos

Hi, You can refer: https://community.databricks.com/s/question/0D53f00001UCQJ7CAP/delta-tables-incremental-backup-method. Please let us know if you need any further clarification on the same.

  • 1 kudos
RantoB
by • Valued Contributor
  • 8844 Views
  • 10 replies
  • 8 kudos

Resolved! Import notebook with python script using API

Hi,I would like to import a python notebook to my databricks workspace from my local machine using a python script.I manages to create the folder but then I have a status code 400 when I try to import a file :create_folder = requests.post( '{}/api/...

Data Engineering
Reply
  • 8844 Views
  • 10 replies
  • 8 kudos
View Replies
Latest Reply
RantoB
Valued Contributor
  • 8 kudos

Hi, Thanks for your answer.Actually both your code and mine are working. However, I cannot write in the directory Repos which is reserved (but I can create subdirectories...)Thanks to your code I got an error message which helped me to understand. Wi...

  • 8 kudos
9 More Replies

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels