- 9880 Views
- 5 replies
- 8 kudos
python multiprocessing hangs at map on one cluster but works fine on another
I have a simple python script which have been running fine on my cluster but recently the same script gets stuck at map. So I tried creating a new cluster with less resources and tried to run the same script over that and it ran just fine.Here are th...
- 9880 Views
- 5 replies
- 8 kudos
- 8 kudos
I agree with @raphaelblg. Most likely you're running out of memory. Multiprocessing or threadpools unfortunately do not benefit from extra workers as they only run on your driver node. This is very annoying and not a very known fact. Spark driver als...
- 8 kudos
- 1683 Views
- 1 replies
- 1 kudos
Streaming Query
How to remove duplicates in streaming query on the basis of some id?
- 1683 Views
- 1 replies
- 1 kudos
- 1 kudos
@nileshtiwaari Are you refering to Strucutred Streaming or DLT?In case of Structured Streaming: https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#streaming-deduplicationAbout DLT, here's a thread from a couple of months...
- 1 kudos
- 1061 Views
- 0 replies
- 0 kudos
Business Loans NZ
Boost Your Business with Unsecured Business Loans in NZ - Apply Now Looking for quick and easy unsecured business loans in New Zealand?For Business Loans NZ, it's crucial to understand the pivotal role they play in supporting businesses of all sizes ...
- 1061 Views
- 0 replies
- 0 kudos
- 1292 Views
- 0 replies
- 0 kudos
Clarification on overwriting in Unity Catalog
Hello, While reviewing Unity Catalog to better understand its limitations, I came across the following statement:Overwrite mode for DataFrame write operations into Unity Catalog is supported only for Delta tables, not for other file formats. The user...
- 1292 Views
- 0 replies
- 0 kudos
- 571 Views
- 0 replies
- 0 kudos
Amazing experience at the DAIS 24 so far
Looking forward to the amazing updates with Gen AI during tomorrow's keynote
- 571 Views
- 0 replies
- 0 kudos
- 767 Views
- 1 replies
- 0 kudos
What have you learned today?
Learned about Managed tables
- 767 Views
- 1 replies
- 0 kudos
- 0 kudos
Delta Sharing is pretty sweet!
- 0 kudos
- 2639 Views
- 1 replies
- 0 kudos
Facing CORS issue while calling Databricks REST API from a react application
I am trying to call the REST API of Databricks from a React(v17) application, using the following documentation https://docs.databricks.com/api/workspace/introduction . I am getting CORS issue on the APIs and not able to call them. But, when I try t...
- 2639 Views
- 1 replies
- 0 kudos
- 0 kudos
There was a similar issue resolved regarding cors seen here: https://community.databricks.com/t5/administration-architecture/databricks-rest-apis-cors-issue/td-p/63936 If this is just for your own development and testing you might consider allowing a...
- 0 kudos
- 551 Views
- 0 replies
- 0 kudos
Summit 2024
I've loved getting to meet everyone this week!
- 551 Views
- 0 replies
- 0 kudos
- 978 Views
- 1 replies
- 0 kudos
Need help calculating the cost benefits of switching from interactive to job cluster
I need help calculating the cost benefits of switching from interactive to job cluster. Can you help me get some formulas on how to calculate the cost differences in Databricks?
- 978 Views
- 1 replies
- 0 kudos
- 0 kudos
Assuming you're on Azure (otherwise use the AWS/GCP equivalent), did you try the Azure cost calculator? https://azure.microsoft.com/en-us/pricing/details/databricks/Question to ask yourself to get more specific: Do you have an idea how much DBU's you...
- 0 kudos
- 5823 Views
- 0 replies
- 0 kudos
Unable to import ml.dmlc.xgboost4j.scala.spark.XGBoostClassifier
Hi I am trying to use the Scala XGBoostClassifier on Databricks, inspiring by this https://www.youtube.com/watch?v=vD3k-moYgmU video shared by Dbx official account. But I couldn't import the library.I followed the official tutorial https://docs.datab...
- 5823 Views
- 0 replies
- 0 kudos
- 6069 Views
- 1 replies
- 2 kudos
Resolved! get_json_object and json path filtering
I have following string [{"key":"abc","value":{"string_value":"abc123"}},{"key":"def","value":{"int_value":123}},{"key":"ghi","value":{"string_value":"ghi456"}}] and from that string i need to extract key.value.string_value for key with the value equ...
- 6069 Views
- 1 replies
- 2 kudos
- 2 kudos
Can you try with the below code snippet WITH exploded_json AS ( SELECT explode(from_json( '[{"key":"abc","value":{"string_value":"abc123"}},{"key":"def","value":{"int_value":123}},{"key":"ghi","value":{"string_value":"ghi456"}}]', 'array<s...
- 2 kudos
- 1098 Views
- 0 replies
- 0 kudos
UCX installation error
I am getting the below error while Installing UCX. But Installation is done in the workspace.I have admin privileges on the workspace. The below error suggests to check token or URL of workspace.They are provided correctly.Then why below error is sho...
- 1098 Views
- 0 replies
- 0 kudos
- 2254 Views
- 4 replies
- 0 kudos
Error when cloning repository
Hi all,cloning a devops repo worked like a charm every time, but now I get a weird error, that I do not understandWhat container is meant here and why would I need one?Thanks and regards,Raphael
- 2254 Views
- 4 replies
- 0 kudos
- 0 kudos
@Yeshwanth I am sorry, I thought I replied to you on friday. I cannot upload a .har file. I get the error message:The file type (.har) is not supported. Valid file types are: jpg, gif, png, pdf.
- 0 kudos
- 4185 Views
- 3 replies
- 1 kudos
Certification Coupon Voucher Not Received
Hi Team I have attended the webinar long back and completed the survey as well still i have not receive any coupon voucher and because of that not able to give exam . Please look into this isue asap and provide me with the voucher .
- 4185 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi Guilherme, We can see that, whereas you used the email address "guilherme.santos@bridgeconsulting.com.br" to attend the webinar, but you did not complete the survey. So you have not received the voucher code.
- 1 kudos
- 896 Views
- 0 replies
- 0 kudos
Disaster Recovery for Databricks Unity catalog enab;led workspace
Hi I have a unity catalog enabled workspaces in us-east-1 and would like to setup DR Databricks workspace in us-east-2. How can we achieve as I don't see a clear path for syncing metadata and delta tables across regions.
- 896 Views
- 0 replies
- 0 kudos
