cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

FK
by New Contributor
  • 1765 Views
  • 1 replies
  • 1 kudos
  • 1765 Views
  • 1 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @Faizan Khan​ Thank you for reaching out, and we’re sorry to hear about this log-in issue! We have this Community Edition login troubleshooting post on Community. Please take a look, and follow the troubleshooting steps. If the steps do not resolv...

  • 1 kudos
SG_
by New Contributor II
  • 4816 Views
  • 1 replies
  • 2 kudos

How to display Sparklyr table in a clean readable format similar to the output of display()?

There exist a Databricks’s built-in display() function (see documentation here) which allow users to display R or SparkR dataframe in a clean and human readable manner where user can scroll to see all the columns and perform sorting on the columns. S...

ewFTT.jpg h3hff
  • 4816 Views
  • 1 replies
  • 2 kudos
Latest Reply
rich_goldberg
New Contributor II
  • 2 kudos

I found that the display() function returned this issue when it came across date-type fields that were NULL. The following function seemed to fix the problem:library(tidyverse) library(lubridate)   display_fixed = function(df) { df %>% ...

  • 2 kudos
KVNARK
by Honored Contributor II
  • 7670 Views
  • 2 replies
  • 7 kudos

need to fetch secrets from key vault in my local

Could you please look into this if I'm missing something. Getting the below error:azure.core.exceptions.ServiceRequestError: Bearer token authentication is not permitted for non-TLS protected (non-https) URLs.Using below function for that.def get_aut...

  • 7670 Views
  • 2 replies
  • 7 kudos
Latest Reply
Anonymous
Not applicable
  • 7 kudos

Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us so we can help you. C...

  • 7 kudos
1 More Replies
agagrins
by New Contributor III
  • 4408 Views
  • 3 replies
  • 2 kudos

How to speed up `dbx launch --from-assets`

Hiya,I'm trying to follow the testing workflow of```$ dbx deploy test --assets-only$ dbx launch test --from-assets --trace --include-output stdout```But I find the turnaround time is quite long, even with an instance pool.The `deployment.yaml` looks ...

  • 4408 Views
  • 3 replies
  • 2 kudos
Latest Reply
tonkol
New Contributor II
  • 2 kudos

Hi, I have no solution, actually I've just registered to open a very similar ticket, when saw yours.According to my experiments getting an already running VM from the pool (times between events: CREATING - INIT_SCRIPTS_STARTED) can take anything betw...

  • 2 kudos
2 More Replies
joshberry
by New Contributor II
  • 4738 Views
  • 2 replies
  • 0 kudos

Resolved! Unable to add password to a user (with SSO enabled)

I am trying to add a non-SSO admin user to my account (not to a workspace). I have SSO backed off to Google for the majority of users.I can create the account OK, then go in and reset the password to something, but when I try and log in I get the err...

  • 4738 Views
  • 2 replies
  • 0 kudos
Latest Reply
joshberry
New Contributor II
  • 0 kudos

Ah, missed that bit of the docs. Thanks

  • 0 kudos
1 More Replies
vinaykumar
by Databricks Partner
  • 4052 Views
  • 3 replies
  • 0 kudos

File optimization for delta table (versioning and snapshot ) in storage S3

Delta table generates new file for every insert or update on table and keep the old version files also for versioning  and time travel history . I have 1tb data as delta table and every 30 minutes , 90 percent data getting updated so file size will b...

  • 4052 Views
  • 3 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @vinay kumar​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks...

  • 0 kudos
2 More Replies
dispersion
by New Contributor
  • 2779 Views
  • 2 replies
  • 1 kudos

Running large volume of SQL queries in Python notebooks. How to minimise overheads/maintenance.

I have around 200 SQL queries id like to run in databricks python notebooks. Id like to avoid creating an ETL process for each of the 200 SQL processes.Any suggestions on how to run the queries in a way that it loops through them so i have minimum am...

  • 2779 Views
  • 2 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @Chris French​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thank...

  • 1 kudos
1 More Replies
jairomonassa
by New Contributor
  • 3934 Views
  • 4 replies
  • 2 kudos
  • 3934 Views
  • 4 replies
  • 2 kudos
Latest Reply
Anonymous
Not applicable
  • 2 kudos

Hi @jairo neder monassa moreira​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear...

  • 2 kudos
3 More Replies
JordGray_57117
by Databricks Partner
  • 3048 Views
  • 3 replies
  • 0 kudos

Resolved! Is it possible to reset user passwords outside of the Admin Console UI?

There is a business requirement for some of our accounts to have their passwords rotated. This currently requires an admin to go in and manually reset the password for the account via UI. I wanted to know if there's a more automated way to handle thi...

  • 3048 Views
  • 3 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Jordan Gray​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks...

  • 0 kudos
2 More Replies
zUnkn0wn990
by New Contributor II
  • 1729 Views
  • 1 replies
  • 2 kudos

Resolved! Lakehouse Fundamentals Accreditation badge not received

I have passed the test today for Lakehouse Fundamentals Accreditation, but have not yet received the badge yet.Please let me know how and when I can receive the badge for this passed test.Thank you.

  • 1729 Views
  • 1 replies
  • 2 kudos
Latest Reply
Anonymous
Not applicable
  • 2 kudos

Hi @Chang Su Lee​ Thank you for reaching out! Please submit a ticket to our Training Team here: https://help.databricks.com/s/contact-us?ReqType=training  and our team will get back to you shortly. 

  • 2 kudos
Mado
by Valued Contributor II
  • 4873 Views
  • 2 replies
  • 2 kudos

Should I create Azure storage & Metastore in the same region?

I am going to create a Metasore following the documentation. Regarding the storage account, I don't understand if it should be in the same region as Metastore. From documentation:You can create no more than one metastore per region. It is recommended...

image
  • 4873 Views
  • 2 replies
  • 2 kudos
Latest Reply
Mado
Valued Contributor II
  • 2 kudos

Thanks. But the question is about the region for Storage Account & Metastore.

  • 2 kudos
1 More Replies
Mado
by Valued Contributor II
  • 7030 Views
  • 3 replies
  • 2 kudos

Databricks Audit Logs, Where the log files are stored? How to read them?

Hi,I want to access the Databricks Audit Logs to check user activity. For example, the number of times that a table was viewed by a user.I have a few questions in this regard. 1) Where the log files are stored? Are they stored on DBFS?2) Can I read l...

  • 7030 Views
  • 3 replies
  • 2 kudos
Latest Reply
Anonymous
Not applicable
  • 2 kudos

Hi @Mohammad Saber​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Tha...

  • 2 kudos
2 More Replies
zeta_load
by Databricks Partner
  • 2785 Views
  • 2 replies
  • 0 kudos

Resolved! When does delta lake actually compute a table?

Maybe I'm completely wrong, but from my understanding delta lake only calculates a table at certain points, for instance when you display your data. Before that point, operations are only written to the log file and are not executed (meaning no chang...

  • 2785 Views
  • 2 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Lukas Goldschmied​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you....

  • 0 kudos
1 More Replies
abhishek_dutta3
by Databricks Partner
  • 17286 Views
  • 5 replies
  • 0 kudos

Merge upsert in Delta is throwing error "concurrent merge in delta lake tables in Azure Databricks .ConcurrentAppendException"

This is the error which is coming while processing concurrent merge in delta lake tables in Azure Databricks .ConcurrentAppendException: Files were added to the root of the table by a concurrent update. Please try the operation again.. What are the o...

  • 17286 Views
  • 5 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Abhishek Dutta​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Tha...

  • 0 kudos
4 More Replies
vinaykumar
by Databricks Partner
  • 3726 Views
  • 3 replies
  • 0 kudos

Resolved! Time travel and version control- can create custom version control for each day data load when multiple updates happening in a day.

Time travel and version control- can create custom version control for each day data load when multiple updates happening in a day. For example , let’s assume we are doing multiple operation on table in a day every minute and want to keep time travel...

  • 3726 Views
  • 3 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @vinay kumar​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks...

  • 0 kudos
2 More Replies
Labels