cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Orianh
by Valued Contributor II
  • 59058 Views
  • 25 replies
  • 37 kudos

Fatal error: Python kernel is unresponsive

Hey guys, I'm using petastorm to train DNN, First i convert spark df with make_spark_convertor and then open a reader on the materialized dataset.While i start training session only on subset of the data every thing works fine but when I'm using all...

  • 59058 Views
  • 25 replies
  • 37 kudos
Latest Reply
Anonymous
Not applicable
  • 37 kudos

Same error. This started a few days ago on notebooks that used to run fine in the past. Now, I cannot finish a notebook.I have already disabled almost all output being streamed to the result buffer, but the problem persists. I am left with <50 lines ...

  • 37 kudos
24 More Replies
pawelmitrus
by Contributor
  • 3273 Views
  • 3 replies
  • 3 kudos

log4j vulnerability - action plan for clients

I'm looking for some information regarding log4j vulnerability - if any databricks runtime should be changed manually by the client or when specific update will be applied.I know I can go through the docs by myself, finding out which log4j library is...

  • 3273 Views
  • 3 replies
  • 3 kudos
Latest Reply
Mr_Srinivasa
New Contributor II
  • 3 kudos

Thanks for sharing such important facts. I got the best security service provider website on the internet. They are excellence in the field of security. 

  • 3 kudos
2 More Replies
Cosimo_F_
by Contributor
  • 14551 Views
  • 7 replies
  • 9 kudos

Incorrectly truncated long numbers in DS&E workspaces notebooks

Hello,I am getting inconsistent representation of long types.1661817599972 is the unix timestamp in milliseconds for Monday, August 29, 2022 11:59:59.972 PM GMTwhen I execute:`select 1661817599972 as t`the result is:166181759997 (last digit truncated...

  • 14551 Views
  • 7 replies
  • 9 kudos
Latest Reply
Anonymous
Not applicable
  • 9 kudos

Hi @Cosimo Felline​ Hope all is well!Does @Hubert Dudek (Customer)​ response answer your question?If you could resolve your issue, would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd...

  • 9 kudos
6 More Replies
sage5616
by Valued Contributor
  • 15761 Views
  • 11 replies
  • 10 kudos

Error in SQL statement: AnalysisException: Cannot up cast documents from array

Hi Everyone,I am getting the following error when running a SQL query and do not understand what it means or what can be done to resolve it. Any recommendations?View DDL:CREATE VIEW myschema.table ( accountId, agreementType, capture_file_name, ...

  • 15761 Views
  • 11 replies
  • 10 kudos
Latest Reply
Anonymous
Not applicable
  • 10 kudos

Hi @Michael Okulik​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Tha...

  • 10 kudos
10 More Replies
Jijo_George
by New Contributor II
  • 3694 Views
  • 3 replies
  • 4 kudos

Lakehouse Fundamentals Accreditation badge not received

Lakehouse Fundamentals Accreditation badge not received. @Kaniz Fatma​ @Lindsay Olson​ 

  • 3694 Views
  • 3 replies
  • 4 kudos
Latest Reply
Anonymous
Not applicable
  • 4 kudos

Hi @Haobo Zhu​  and @Srihari P​ Just a friendly- follow-up that if you were able to resolve your certification issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell u...

  • 4 kudos
2 More Replies
HZ
by New Contributor III
  • 6389 Views
  • 5 replies
  • 4 kudos

Serverless SQL Datawarehouse cluster failing to launch

Hope someone may able to help, got a serverless SQL datawarehouse in SQL workspace, it was created by another member in the team and it had been working all good over past month.The person who created this serverless SQL datawarehouse left the team a...

  • 6389 Views
  • 5 replies
  • 4 kudos
Latest Reply
Anonymous
Not applicable
  • 4 kudos

Hi @Helen Zhang​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks...

  • 4 kudos
4 More Replies
StuartParker188
by New Contributor III
  • 9474 Views
  • 5 replies
  • 5 kudos

How to handle java.io.Exception in python notebook

I'm attempting to mount a volume using dbutils.fs.mount in a python workbookin the exception handling for this statement, I have found an exception that doesn't get caught using the standard try/except handlingfor example, if passing through a contai...

  • 9474 Views
  • 5 replies
  • 5 kudos
Latest Reply
Anonymous
Not applicable
  • 5 kudos

Hi @Stuart Parker​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Than...

  • 5 kudos
4 More Replies
salmacgregor
by New Contributor III
  • 7089 Views
  • 7 replies
  • 7 kudos

Forgot password email is not sent

I couldn't login to my Databricks premium 14 day free trial workspace so I tried to reset my password but the email was never sent. Please help - I've already lost a day of my trial because I couldn't login. Thank you

  • 7089 Views
  • 7 replies
  • 7 kudos
Latest Reply
Anonymous
Not applicable
  • 7 kudos

Hi @Sarah Rudston​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Than...

  • 7 kudos
6 More Replies
146041
by New Contributor II
  • 3214 Views
  • 2 replies
  • 2 kudos

Resolved! Can someone give me a hand with Azure quotas and Databricks?

I'm trying to use databricks learning and go through the various lessons using my Azure pay-as-you-go account. IT's been super frustrating because I'm constantly hitting quota limits. First it was Regional vCPUs, then dedicated, now 'low priority' ...

  • 3214 Views
  • 2 replies
  • 2 kudos
Latest Reply
Anonymous
Not applicable
  • 2 kudos

Hi @user_bfe431b2 erithtotl​ Does @Hubert Dudek​  and @Kaniz Fatma​ 's response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?We'd love to hear from you.Thanks!

  • 2 kudos
1 More Replies
Mohit_Kumar_Sut
by Databricks Partner
  • 7742 Views
  • 5 replies
  • 1 kudos

Write in Single CSV file

We are reading 520GB partitions files from CSV and when we write in a Single CSV using repartition(1) it is taking 25+ hours. please let us know an optimized way to create a single CSV file so that our process could complete within 5 hours.

  • 7742 Views
  • 5 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @mohit kumar suthar​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you...

  • 1 kudos
4 More Replies
saulrc
by New Contributor III
  • 2981 Views
  • 4 replies
  • 1 kudos

Hi,On the 27 of October, I did and pass my exam for data bricks data engineer associate but I didn&#39;t get my badge. could you help me with that? Al...

Hi,On the 27 of October, I did and pass my exam for data bricks data engineer associate but I didn't get my badge. could you help me with that? Also, I raised the ticker number: #00232560 but also I don't get any updates about it.cc:@Jose Gonzalez​ 

  • 2981 Views
  • 4 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @SAUL Ramirez​ Hope everything is going great.Just wanted to check in if you were able to resolve your issue.If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please visit the cr...

  • 1 kudos
3 More Replies
geertvanhove
by New Contributor III
  • 8510 Views
  • 6 replies
  • 5 kudos
  • 8510 Views
  • 6 replies
  • 5 kudos
Latest Reply
Anonymous
Not applicable
  • 5 kudos

Could you please confirm if you are using E2 architecture (https://docs.databricks.com/getting-started/overview.html#e2-architecture) or is it a single tenant shard?

  • 5 kudos
5 More Replies
JoeWMP
by New Contributor III
  • 2635 Views
  • 1 replies
  • 7 kudos

All-purpose compute clusters that are attached to a pool are no longer able to switch to a different pool/change to a non-pool worker/driver.

Would like to know if anyone else is experiencing this - we're seeing this across 5+ different Databricks workspaces in both AWS and Azure.Reproduction: Create all purpose compute cluster, attach it to existing pool, save and start cluster. Edit clus...

image
  • 2635 Views
  • 1 replies
  • 7 kudos
Latest Reply
JoeWMP
New Contributor III
  • 7 kudos

We're also seeing the same behavior when trying to change the pool on an all-purpose cluster using Terraform and Databricks Labs Terraform provider as well. The Terraform apply will go through and say the cluster was updated to the new pool id, but t...

  • 7 kudos
Labels