- 8131 Views
- 8 replies
- 8 kudos
Using .repartition(100000) causes the unit test to be extremely slow (>20 mins). Is there a way to speed it up?
Here's the code:val result = spark .createDataset(List("test")) .rdd .repartition(100000) .map { _ => "test" } .collect() .toList println(result)I write tests to test for correctness, so I wonde...
- 8131 Views
- 8 replies
- 8 kudos
- 8 kudos
Hey there @tanin​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thank...
- 8 kudos
- 39995 Views
- 2 replies
- 2 kudos
Resolved! Connect to an on-prem SQL server database
Need to connect to an on-prem SQL database to extract data, we are using the Apache Spark SQL connector. The problem is can't able to connect to connection failure SQLServerException: The TCP/IP connection to the host ***.***.X.XX, port 1433 has fail...
- 39995 Views
- 2 replies
- 2 kudos
- 2 kudos
Maybe you can check the below docs and see if something is missing in the set uphttps://docs.microsoft.com/en-us/azure/databricks/administration-guide/cloud-configurations/azure/on-prem-network
- 2 kudos
- 3155 Views
- 2 replies
- 1 kudos
Resolved! Noteboook-scoped env variables?
Is it possible to set environment variables at the notebook level instead of the cluster level? Will they be available in the workers in addition to the driver? Can they override the env variables set at the cluster level?
- 3155 Views
- 2 replies
- 1 kudos
- 6474 Views
- 5 replies
- 5 kudos
Resolved! How much of this tutorial or blog post can I run before starting a cloud instance of databricks?
I'm new to python and databricks so I'm still running tests on features, and not sure how much of this can be run without databricks which I guess requires an AWS or Google cloud account? Can I do all three stages without the AWS databricks or how fa...
- 6474 Views
- 5 replies
- 5 kudos
- 5 kudos
Hi, to run it, you need databricks. You can try to open a free community account. Here is explained how: https://community.databricks.com/s/feed/0D53f00001ebEasCAE
- 5 kudos
- 4301 Views
- 2 replies
- 3 kudos
Resolved! Maximum job execution per hour
Hi, what is the maximum number of jobs we can execute in an hour for a given workspace?This page mentions 5000https://docs.microsoft.com/en-us/azure/databricks/data-engineering/jobs/jobsThe number of jobs a workspace can create in an hour is limited ...
- 4301 Views
- 2 replies
- 3 kudos
- 3 kudos
Up to 5000 jobs (both normal and ephemeral) may be created per hour in a single workspace
- 3 kudos
- 21742 Views
- 20 replies
- 3 kudos
Resolved! Cloudformation error when launching Databricks in AWS
I've seen many posts here in the Community as potential solutions to this error, but none seem to be a solution for us. We are trying to launch the 14 day free trial of Databricks from the AWS Marketplace and are getting the error below. Moreover, ...
- 21742 Views
- 20 replies
- 3 kudos
- 3 kudos
Here are some answers:copyObject error - we were using a Databricks provided cloudformation template but this error goes away when we use the AWS provided templatecreateWorkspace error - we had subscribed>unsubscribed>resubscribed to Databricks via t...
- 3 kudos
- 8520 Views
- 4 replies
- 1 kudos
Apply change data with delete and schema evolution
Hi,Currently, I'm using structure streaming to insert/update/delete to a table. A row will be deleted if value in 'Operation' column is 'deleted'. Everything seems to work fine until there's a new column.Since I don't need 'Operation' column in the t...
- 8520 Views
- 4 replies
- 1 kudos
- 1 kudos
please go through this documentation https://docs.delta.io/latest/api/python/index.html
- 1 kudos
- 3345 Views
- 1 replies
- 1 kudos
Spark Error when running python script on databricks
I have the following basic script that works fine using pycharm on my machine.from pyspark.sql import SparkSessionprint("START")spark = SparkSession \ .Builder() \ .appName("myapp") \ .master('local[*, 4]') \ .getOrCreate()print(spark)dat...
- 3345 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @Patricia Mayer​ Just wanted to check in if you were able to resolve your issue or do you need more help? We'd love to hear from you.Thanks!
- 1 kudos
- 1489 Views
- 1 replies
- 2 kudos
Advanced ML Virtual Training Video from 2022 Summit (not currently accessible)
There does not seem to be a way to log into and view the recent "paid" training sessions from the 2022 Data/AI Summit. I was able to log in and view the videos yesterday, but the website currently posted has no option for logging in/access. Is the...
- 1489 Views
- 1 replies
- 2 kudos
- 2 kudos
Hey there @Christopher Warner​ Just wanted to check in if you were able to resolve your issue or do you need more help? We'd love to hear from you.Thanks!
- 2 kudos
- 7860 Views
- 4 replies
- 1 kudos
Why Databricks spawns multiple jobs
I have a Delta table spark101.airlines (sourced from `/databricks-datasets/airlines/`) partitioned by `Year`. My `spark.sql.shuffle.partitions` is set to default 200. I run a simple query:select Origin, count(*) from spark101.airlines group by Origi...
- 7860 Views
- 4 replies
- 1 kudos
- 3997 Views
- 1 replies
- 1 kudos
Snowflake/GCP error: Premature end of chunk coded message body: closing chunk expected
Hello all,I've been experiencing the error described below, where I try to query a table from Snowflake which is about ~5.5B rows and ~30columns, and it fails almost systematically; specifically, either the Spark Job doesn't even start or I get the ...
- 3997 Views
- 1 replies
- 1 kudos
- 1 kudos
Hey there @hamzatazib96​ Does @Kaniz Fatma​ response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?We'd love to hear from you.Thanks!
- 1 kudos
- 6496 Views
- 5 replies
- 3 kudos
Encryption/Decryption options in ADB
Hello all,We are working on one of the client requirements to implement suitable data encryption in Azure Databricks.We should be able to encrypt and decrypt the data based on the access, we explored fernet library but client denied it saying it degr...
- 6496 Views
- 5 replies
- 3 kudos
- 3 kudos
Hi @purushotham Chanda​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you...
- 3 kudos
- 9010 Views
- 3 replies
- 28 kudos
Some key notes from today's Databricks Azure Roadmap Q4 event:Â - Faster merge and updates with Photon + Deletion vectors in Q3,- Unity Catalog - s...
Some key notes from today's Databricks Azure Roadmap Q4 event:- Faster merge and updates with Photon + Deletion vectors in Q3,- Unity Catalog - system tables, including system table for lineage,- ML features and models under Unity Catalog governance,...
- 9010 Views
- 3 replies
- 28 kudos
- 28 kudos
Thanks, for sharing this.Nothing about Delta Live Tables availability in Unity Catalog ?
- 28 kudos
- 2691 Views
- 1 replies
- 1 kudos
Resolved! Attended 27th July 2022 webinar but have not recieved voucher,even uploaded Lakehouse certificate
@Kaniz Fatma (Databricks)​ @Samantha (Databricks)​ This is Diego Navarro .Actually I attended Databricks webinar on 27th July for (Databricks Certification Exam Overview Training: Databricks Certified Data Analyst Associate).I was expecting vouchers ...
- 2691 Views
- 1 replies
- 1 kudos
- 1 kudos
Hello,you received your voucher on 8/4. It might have gone to spam. Here you go: APP82P22diwBJHT4Thank you
- 1 kudos
- 14219 Views
- 4 replies
- 10 kudos
Resolved! UI Improvements / Personalization?
I have a few suggestions for UI improvement on Databricks console -- Or maybe if anyone has figured out a way (using greasemonkey or similar scripts) to make some changes to Databricks UI -- i would like to know. # 1 - Workspace NavigationCan we have...
- 14219 Views
- 4 replies
- 10 kudos
- 10 kudos
Great ideas.I know that regarding #1 new file manager is in development.#3 I also proposed when we discussed possible improvements.@Lindsay Olson​ @Jose Gonzalez​ @Prabakar Ammeappin​ maybe we can push it as user feedback as that are great ideas with...
- 10 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 582 | |
| 349 | |
| 287 |
