- 19240 Views
- 2 replies
- 1 kudos
ModuleNotFoundError / SerializationError when executing over databricks-connect
I am running into the following error when I run a model fitting process over databricks-connect.It looks like worker nodes are unable to access modules from the project's parent directory. Note that the program runs successfully up to this point; n...
- 19240 Views
- 2 replies
- 1 kudos
- 1 kudos
@Sarosh Ahmad​ , Could you try adding the zip of the module to the addPyFile like belowspark.sparkContext.addPyFile("src.zip")
- 1 kudos
- 802 Views
- 0 replies
- 0 kudos
Hi juliet Wu I have completed my databricks apache spark associate developer exam on 7/10/2021 after subsequent completion of my exam I got my badge t...
Hi juliet Wu I have completed my databricks apache spark associate developer exam on 7/10/2021 after subsequent completion of my exam I got my badge to my Webaccesor Mail immediately after 1 day of exam which is on 8/10/2021​.but I didn't received my...
- 802 Views
- 0 replies
- 0 kudos
- 3091 Views
- 1 replies
- 2 kudos
Resolved! How to source control a Dashboard?
Is it possible to source control the dashboard along with a notebook code? When source controlling a python notebook it gets converted to *.py. It looks like the resulting *.py file loses the information about the dashboard cells. Thus, if this *.py ...
- 3091 Views
- 1 replies
- 2 kudos
- 2 kudos
No, you will need to save as another source, like DBC Archive, to replicate the Notebook features.
- 2 kudos
- 4610 Views
- 1 replies
- 1 kudos
How to provide custom class extending SparkPlugin/ExecutorPlugin in Databricks 7.3?
How to properly configure the jar containing the class and spark plugin in Databricks?During DBR 7.3 cluster creation, I tried by setting the spark.plugins, spark.driver.extraClassPath and spark.executor.extraClassPath Spark configs by copying the ja...
- 4610 Views
- 1 replies
- 1 kudos
- 1 kudos
Hello @Krishna Kashiv​ - I don't know if we've met yet. My name is Piper and I'm a community moderator here. Thank you for your new question. It looks thorough! Let's give it a while to see what our members have to say. Otherwise, we will circle back...
- 1 kudos
- 3405 Views
- 1 replies
- 0 kudos
Create gpg file and save to AWS s3 storage in scala
Hi - Could you please help me on how can I create a scala notebook to perform the below tasksEncrypt a text file using the gpgUpload the file to amazon s3 storageverify the file exists in amazon s3decrypt the encrypted file to verify no issuesApprec...
- 3405 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello! My name is Piper and I'm a community moderator for Databricks. Thanks for your question. Let's give it a bit more to see what our members have to say. If not, we'll circle back around.
- 0 kudos
- 8443 Views
- 11 replies
- 7 kudos
Resolved! What is the proper way to import the new pyspark.pandas library?
I am moving an existing, working pandas program into Databricks. I want to use the new pyspark.pandas library, and change my code as little as possible. It appears that I should do the following:1) Add from pyspark import pandas as ps at the top2) Ch...
- 8443 Views
- 11 replies
- 7 kudos
- 12869 Views
- 6 replies
- 2 kudos
CLI Command <databricks fs cp> Not Uploading Files to DBFS
Hi all, So far I have been successfully using the CLI interface to upload files from my local machine to DBFS/FileStore/tables. Specifically, I have been using my terminal and the following command: databricks fs cp -r <MyLocalDataset> dbfs:/FileStor...
- 12869 Views
- 6 replies
- 2 kudos
- 2 kudos
hi @Ignacio Castineiras​ ,If Arjun.kr's fully answered your question, would you be happy to mark their answer as best so that others can quickly find the solution?Please let us know if you still are having this issue.
- 2 kudos
- 699 Views
- 0 replies
- 0 kudos
1.   DIFFERENT TYPES OF TACTICAL GEAR 1. HARDWAREOptical hardware, for instance, cuffs, laser sights, optics, and night vision goggles accompany a hug...
1. DIFFERENT TYPES OF TACTICAL GEAR1. HARDWAREOptical hardware, for instance, cuffs, laser sights, optics, and night vision goggles accompany a huge group of features and capacities. Packs and pockets are made of climate-safe material planned to ke...
- 699 Views
- 0 replies
- 0 kudos
- 3079 Views
- 1 replies
- 0 kudos
Creating a table like in SQL with Spark
Hi !I'm working on a project at my company on Databricks using Scala and Spark. I'm new to Spark and Databricks and so I would like to know how to create a table on specific location (on the Delta Lake of my company). In SQL + some Delta features, I ...
- 3079 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Adrien MERAT​ ,I would like to share the following documentation that will provide examples on how to create Delta tables:Create Delta table linkDelta data types link
- 0 kudos
- 4163 Views
- 8 replies
- 0 kudos
Spark adding NUL
Hi I have a DF which contains Json string so the value is like {"key": Value, "anotherKey": anotherValue}, so when I am trying to write the DF containing this string to the CSV, spark is ​adding NUL character af the front of this line and at the end,...
- 4163 Views
- 8 replies
- 0 kudos
- 0 kudos
Hard to tell without having the code, but it might be the separator for the csv? You do have comma's in the string, and comma is the default separator for csv.
- 0 kudos
- 5277 Views
- 4 replies
- 3 kudos
Hi, I like to create a web form with displayHTML in a notebook cell and when the users presses the post button, i like to write the content of my text...
Hi,I like to create a web form with displayHTML in a notebook cell and when the users presses the post button, i like to write the content of my text area of my form back in to the code cell of the notebook.Example:displayHTML ("""<form><textarea> u...
- 5277 Views
- 4 replies
- 3 kudos
- 3 kudos
Hi @afshin riahi​ ,Did Dan's response helped you to solve your question? if it did, can you mark it as best answer? I will help to move the post to the top so other can quickly find the solution.
- 3 kudos
- 6234 Views
- 5 replies
- 2 kudos
Resolved! AWS VPC peering connection: can't make Databricks VPC reach our services on the accepter VPC
Hi,We followed this document (https://docs.databricks.com/administration-guide/cloud-configurations/aws/vpc-peering.html) describing how to establish a connection between two (or more) VPC in AWS, but so far we haven't been able to communicate with t...
- 6234 Views
- 5 replies
- 2 kudos
- 2 kudos
Hi @Martin Cigorraga​ ,If Huaming's fully answered your question, would you be happy to mark their answer as best so that others can quickly find the solution?
- 2 kudos
- 6241 Views
- 3 replies
- 0 kudos
- 6241 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @Ayman Alneser​ ,Did Huaming.lu's response worked for you? if it did, could you marked as the best solution so that other can quickly find it in the future.
- 0 kudos
- 18373 Views
- 6 replies
- 5 kudos
Resolved! Can you help with this error please? Issue when using a new high concurrency cluster
Hello, I am trying to use MLFlow on a new high concurrency cluster but I get the error below. Does anyone have any suggestions? It was working before on a standard cluster. Thanks.py4j.security.Py4JSecurityException: Method public int org.apache.spar...
- 18373 Views
- 6 replies
- 5 kudos
- 5 kudos
@Tom Soto​ We have a workaround for this. This cluster spark configuration setting will disable py4jSecurity while still enabling passthrough spark.databricks.pyspark.enablePy4JSecurity false
- 5 kudos
- 15210 Views
- 8 replies
- 2 kudos
Resolved! How many hours I can estimate to trainning in a Databricks Academy Self-Placed Trainning platform ?
I done the Data Engineering Profissional and others training in a Self-Placed Trainning (https://www.linkedin.com/posts/wscardua_data-engineering-professional-activity-6851487238774108160-IsTE) . How many hours can I estimate for this training (and o...
- 15210 Views
- 8 replies
- 2 kudos
| User | Count |
|---|---|
| 1619 | |
| 790 | |
| 485 | |
| 349 | |
| 287 |
