cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

FK
by New Contributor
  • 1655 Views
  • 1 replies
  • 1 kudos
  • 1655 Views
  • 1 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @Faizan Khan​ Thank you for reaching out, and we’re sorry to hear about this log-in issue! We have this Community Edition login troubleshooting post on Community. Please take a look, and follow the troubleshooting steps. If the steps do not resolv...

  • 1 kudos
SG_
by New Contributor II
  • 4588 Views
  • 1 replies
  • 2 kudos

How to display Sparklyr table in a clean readable format similar to the output of display()?

There exist a Databricks’s built-in display() function (see documentation here) which allow users to display R or SparkR dataframe in a clean and human readable manner where user can scroll to see all the columns and perform sorting on the columns. S...

ewFTT.jpg h3hff
  • 4588 Views
  • 1 replies
  • 2 kudos
Latest Reply
rich_goldberg
New Contributor II
  • 2 kudos

I found that the display() function returned this issue when it came across date-type fields that were NULL. The following function seemed to fix the problem:library(tidyverse) library(lubridate)   display_fixed = function(df) { df %>% ...

  • 2 kudos
KVNARK
by Honored Contributor II
  • 7448 Views
  • 2 replies
  • 7 kudos

need to fetch secrets from key vault in my local

Could you please look into this if I'm missing something. Getting the below error:azure.core.exceptions.ServiceRequestError: Bearer token authentication is not permitted for non-TLS protected (non-https) URLs.Using below function for that.def get_aut...

  • 7448 Views
  • 2 replies
  • 7 kudos
Latest Reply
Anonymous
Not applicable
  • 7 kudos

Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us so we can help you. C...

  • 7 kudos
1 More Replies
agagrins
by New Contributor III
  • 4082 Views
  • 3 replies
  • 2 kudos

How to speed up `dbx launch --from-assets`

Hiya,I'm trying to follow the testing workflow of```$ dbx deploy test --assets-only$ dbx launch test --from-assets --trace --include-output stdout```But I find the turnaround time is quite long, even with an instance pool.The `deployment.yaml` looks ...

  • 4082 Views
  • 3 replies
  • 2 kudos
Latest Reply
tonkol
New Contributor II
  • 2 kudos

Hi, I have no solution, actually I've just registered to open a very similar ticket, when saw yours.According to my experiments getting an already running VM from the pool (times between events: CREATING - INIT_SCRIPTS_STARTED) can take anything betw...

  • 2 kudos
2 More Replies
joshberry
by New Contributor II
  • 4455 Views
  • 2 replies
  • 0 kudos

Resolved! Unable to add password to a user (with SSO enabled)

I am trying to add a non-SSO admin user to my account (not to a workspace). I have SSO backed off to Google for the majority of users.I can create the account OK, then go in and reset the password to something, but when I try and log in I get the err...

  • 4455 Views
  • 2 replies
  • 0 kudos
Latest Reply
joshberry
New Contributor II
  • 0 kudos

Ah, missed that bit of the docs. Thanks

  • 0 kudos
1 More Replies
vinaykumar
by Databricks Partner
  • 3717 Views
  • 3 replies
  • 0 kudos

File optimization for delta table (versioning and snapshot ) in storage S3

Delta table generates new file for every insert or update on table and keep the old version files also for versioning  and time travel history . I have 1tb data as delta table and every 30 minutes , 90 percent data getting updated so file size will b...

  • 3717 Views
  • 3 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @vinay kumar​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks...

  • 0 kudos
2 More Replies
dispersion
by New Contributor
  • 2624 Views
  • 2 replies
  • 1 kudos

Running large volume of SQL queries in Python notebooks. How to minimise overheads/maintenance.

I have around 200 SQL queries id like to run in databricks python notebooks. Id like to avoid creating an ETL process for each of the 200 SQL processes.Any suggestions on how to run the queries in a way that it loops through them so i have minimum am...

  • 2624 Views
  • 2 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @Chris French​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thank...

  • 1 kudos
1 More Replies
jairomonassa
by New Contributor
  • 3674 Views
  • 4 replies
  • 2 kudos
  • 3674 Views
  • 4 replies
  • 2 kudos
Latest Reply
Anonymous
Not applicable
  • 2 kudos

Hi @jairo neder monassa moreira​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear...

  • 2 kudos
3 More Replies
JordGray_57117
by Databricks Partner
  • 2789 Views
  • 3 replies
  • 0 kudos

Resolved! Is it possible to reset user passwords outside of the Admin Console UI?

There is a business requirement for some of our accounts to have their passwords rotated. This currently requires an admin to go in and manually reset the password for the account via UI. I wanted to know if there's a more automated way to handle thi...

  • 2789 Views
  • 3 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Jordan Gray​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks...

  • 0 kudos
2 More Replies
zUnkn0wn990
by New Contributor II
  • 1547 Views
  • 1 replies
  • 2 kudos

Resolved! Lakehouse Fundamentals Accreditation badge not received

I have passed the test today for Lakehouse Fundamentals Accreditation, but have not yet received the badge yet.Please let me know how and when I can receive the badge for this passed test.Thank you.

  • 1547 Views
  • 1 replies
  • 2 kudos
Latest Reply
Anonymous
Not applicable
  • 2 kudos

Hi @Chang Su Lee​ Thank you for reaching out! Please submit a ticket to our Training Team here: https://help.databricks.com/s/contact-us?ReqType=training  and our team will get back to you shortly. 

  • 2 kudos
Mado
by Valued Contributor II
  • 4616 Views
  • 2 replies
  • 2 kudos

Should I create Azure storage & Metastore in the same region?

I am going to create a Metasore following the documentation. Regarding the storage account, I don't understand if it should be in the same region as Metastore. From documentation:You can create no more than one metastore per region. It is recommended...

image
  • 4616 Views
  • 2 replies
  • 2 kudos
Latest Reply
Mado
Valued Contributor II
  • 2 kudos

Thanks. But the question is about the region for Storage Account & Metastore.

  • 2 kudos
1 More Replies
Mado
by Valued Contributor II
  • 6653 Views
  • 3 replies
  • 2 kudos

Databricks Audit Logs, Where the log files are stored? How to read them?

Hi,I want to access the Databricks Audit Logs to check user activity. For example, the number of times that a table was viewed by a user.I have a few questions in this regard. 1) Where the log files are stored? Are they stored on DBFS?2) Can I read l...

  • 6653 Views
  • 3 replies
  • 2 kudos
Latest Reply
Anonymous
Not applicable
  • 2 kudos

Hi @Mohammad Saber​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Tha...

  • 2 kudos
2 More Replies
zeta_load
by Databricks Partner
  • 2496 Views
  • 2 replies
  • 0 kudos

Resolved! When does delta lake actually compute a table?

Maybe I'm completely wrong, but from my understanding delta lake only calculates a table at certain points, for instance when you display your data. Before that point, operations are only written to the log file and are not executed (meaning no chang...

  • 2496 Views
  • 2 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Lukas Goldschmied​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you....

  • 0 kudos
1 More Replies
abhishek_dutta3
by Databricks Partner
  • 16900 Views
  • 5 replies
  • 0 kudos

Merge upsert in Delta is throwing error "concurrent merge in delta lake tables in Azure Databricks .ConcurrentAppendException"

This is the error which is coming while processing concurrent merge in delta lake tables in Azure Databricks .ConcurrentAppendException: Files were added to the root of the table by a concurrent update. Please try the operation again.. What are the o...

  • 16900 Views
  • 5 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Abhishek Dutta​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Tha...

  • 0 kudos
4 More Replies
vinaykumar
by Databricks Partner
  • 3469 Views
  • 3 replies
  • 0 kudos

Resolved! Time travel and version control- can create custom version control for each day data load when multiple updates happening in a day.

Time travel and version control- can create custom version control for each day data load when multiple updates happening in a day. For example , let’s assume we are doing multiple operation on table in a day every minute and want to keep time travel...

  • 3469 Views
  • 3 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @vinay kumar​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks...

  • 0 kudos
2 More Replies
Labels