cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Engineering & Streaming

Forum Posts

jairomonassa
by New Contributor
  • 1649 Views
  • 4 replies
  • 2 kudos
  • 1649 Views
  • 4 replies
  • 2 kudos
Latest Reply
Anonymous
Not applicable
  • 2 kudos

Hi @jairo neder monassa moreira​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear...

  • 2 kudos
3 More Replies
JordGray_57117
by New Contributor II
  • 1238 Views
  • 3 replies
  • 0 kudos

Resolved! Is it possible to reset user passwords outside of the Admin Console UI?

There is a business requirement for some of our accounts to have their passwords rotated. This currently requires an admin to go in and manually reset the password for the account via UI. I wanted to know if there's a more automated way to handle thi...

  • 1238 Views
  • 3 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Jordan Gray​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks...

  • 0 kudos
2 More Replies
zUnkn0wn990
by New Contributor II
  • 614 Views
  • 1 replies
  • 2 kudos

Resolved! Lakehouse Fundamentals Accreditation badge not received

I have passed the test today for Lakehouse Fundamentals Accreditation, but have not yet received the badge yet.Please let me know how and when I can receive the badge for this passed test.Thank you.

  • 614 Views
  • 1 replies
  • 2 kudos
Latest Reply
Anonymous
Not applicable
  • 2 kudos

Hi @Chang Su Lee​ Thank you for reaching out! Please submit a ticket to our Training Team here: https://help.databricks.com/s/contact-us?ReqType=training  and our team will get back to you shortly. 

  • 2 kudos
Mado
by Valued Contributor II
  • 3051 Views
  • 3 replies
  • 2 kudos

Databricks Audit Logs, Where the log files are stored? How to read them?

Hi,I want to access the Databricks Audit Logs to check user activity. For example, the number of times that a table was viewed by a user.I have a few questions in this regard. 1) Where the log files are stored? Are they stored on DBFS?2) Can I read l...

  • 3051 Views
  • 3 replies
  • 2 kudos
Latest Reply
Anonymous
Not applicable
  • 2 kudos

Hi @Mohammad Saber​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Tha...

  • 2 kudos
2 More Replies
zeta_load
by New Contributor II
  • 1061 Views
  • 2 replies
  • 0 kudos

Resolved! When does delta lake actually compute a table?

Maybe I'm completely wrong, but from my understanding delta lake only calculates a table at certain points, for instance when you display your data. Before that point, operations are only written to the log file and are not executed (meaning no chang...

  • 1061 Views
  • 2 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Lukas Goldschmied​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you....

  • 0 kudos
1 More Replies
abhishek_dutta3
by New Contributor II
  • 9219 Views
  • 5 replies
  • 0 kudos

Merge upsert in Delta is throwing error "concurrent merge in delta lake tables in Azure Databricks .ConcurrentAppendException"

This is the error which is coming while processing concurrent merge in delta lake tables in Azure Databricks .ConcurrentAppendException: Files were added to the root of the table by a concurrent update. Please try the operation again.. What are the o...

  • 9219 Views
  • 5 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Abhishek Dutta​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Tha...

  • 0 kudos
4 More Replies
Smu_Tan
by New Contributor
  • 1803 Views
  • 3 replies
  • 1 kudos

Resolved! Does Databricks supports the Pytorch Distributed Training for multiple devices?

Hi, Im trying to use the databricks platform to do the pytorch distributed training, but I didnt find any info about this. What I expected is using multiple clusters to run a common job using pytorch distributed data parallel (DDP) with the code belo...

  • 1803 Views
  • 3 replies
  • 1 kudos
Latest Reply
axb0
New Contributor III
  • 1 kudos

With Databricks MLR, HorovodRunner is provided which supports distributed training and inference with PyTorch. Here's an example notebook for your reference: PyTorchDistributedDeepLearningTraining - Databricks.

  • 1 kudos
2 More Replies
vinaykumar
by New Contributor III
  • 1282 Views
  • 3 replies
  • 0 kudos

Resolved! Time travel and version control- can create custom version control for each day data load when multiple updates happening in a day.

Time travel and version control- can create custom version control for each day data load when multiple updates happening in a day. For example , let’s assume we are doing multiple operation on table in a day every minute and want to keep time travel...

  • 1282 Views
  • 3 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @vinay kumar​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks...

  • 0 kudos
2 More Replies
srDataEngineer
by New Contributor II
  • 2648 Views
  • 4 replies
  • 2 kudos

Resolved! how does databricks time travel work

Hi, Since it is not very well explained, I want to know if the table history is a snapshot of the whole table at that point of time containing all the data or it tracks only some metadata of the table changes.To be more precise : if I have a table in...

  • 2648 Views
  • 4 replies
  • 2 kudos
Latest Reply
Anonymous
Not applicable
  • 2 kudos

Hi @data engineer​ Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us so...

  • 2 kudos
3 More Replies
aline_alvarez
by New Contributor III
  • 2976 Views
  • 6 replies
  • 7 kudos

Resolved! How can I delete a file in DBFS with Illegal character?

How can I delete a file in DBFS with Illegal character?Someone put the file named "planejamento_[4098.]___SHORT_SAIA_JEANS__.xlsx" inside the folder /FileStore and I can delete it, because of this error: java.net.URISyntaxException: Illegal character...

  • 2976 Views
  • 6 replies
  • 7 kudos
Latest Reply
-werners-
Esteemed Contributor III
  • 7 kudos

try this %sh ls -li /dbfsif the file is located in a subdirectory you can change the path mentioned above.the %sh magic command gives you access to linux shell commands.

  • 7 kudos
5 More Replies
EDDatabricks
by Contributor
  • 2471 Views
  • 2 replies
  • 0 kudos

Resolved! Pool Max Capacity vs Cluster Max Workers

Hi all, we have a databricks instance on Azure with a Compute Cluster version 7.3 LTS. Currently the cluster has 4 max workers (min workers: 1) of type: Standard_D13_v2 and 1 driver of the same type. There are several jobs that are running on this cl...

  • 2471 Views
  • 2 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @EDDatabricks EDDatabricks​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear f...

  • 0 kudos
1 More Replies
tinendra
by New Contributor III
  • 2778 Views
  • 5 replies
  • 5 kudos

How to reduce time while loading data into the azure synapse table?

Hi All,I just wanted to know if is there any option to reduce time while loading Pyspark Dataframe into the Azure synapse table using Databricks.like..I have a pyspark dataframe that has around 40k records and I am trying to load data into the azure ...

  • 2778 Views
  • 5 replies
  • 5 kudos
Latest Reply
Anonymous
Not applicable
  • 5 kudos

Hi @Tinendra Kumar​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Tha...

  • 5 kudos
4 More Replies
Murthy1
by Contributor II
  • 4810 Views
  • 3 replies
  • 3 kudos

Resolved! Impacts of running multiple jobs in parallel that refers the same notebook

Can I run multiple jobs(for example: 100+) in parallel that refers the same notebook? I supply each job with a different parameter. If we can do this, what would be the impact? (for example: reliability, performance, troubleshooting etc. )Example: N...

  • 4810 Views
  • 3 replies
  • 3 kudos
Latest Reply
Anonymous
Not applicable
  • 3 kudos

Hi @Murthy Ramalingam​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you....

  • 3 kudos
2 More Replies
venkat-bodempud
by New Contributor III
  • 2086 Views
  • 4 replies
  • 3 kudos

Resolved! Databricks Design Help

Hello Community,I am currently working on populating gold layer tables. Source for these gold layer tables are silver layer tables. A query is going to run on silver layer tables, spark sql query contains joins between multiple tables.ex:select colum...

  • 2086 Views
  • 4 replies
  • 3 kudos
Latest Reply
Anonymous
Not applicable
  • 3 kudos

Hi @bodempudi venkat​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.T...

  • 3 kudos
3 More Replies
MelMazz
by New Contributor
  • 1863 Views
  • 2 replies
  • 0 kudos

Data in transit security when using jdbc connections.

When using spark.read.jdbc() for a direct connection, how is data in transit secured?

  • 1863 Views
  • 2 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Melody Mazaiwana​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.T...

  • 0 kudos
1 More Replies
Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!

Labels