- 8835 Views
- 2 replies
- 15 kudos
Resolved! Pyspark dataframe column comparison
I have a string column which is a concatenation of elements with a hyphen as follows. Let 3 values from that column looks like below, Row 1 - A-B-C-D-E-FRow 2 - A-B-G-C-D-E-FRow 3 - A-B-G-D-E-FI want to compare 2 consecutive rows and create a column ...
- 8835 Views
- 2 replies
- 15 kudos
- 15 kudos
Hi,I think you can follow these steps:1. Use window function to create a new column by shifting, then your df will look like thisid value lag1 A-B-C-D-E-F null2 A-B-G-C-D-E-F A-B-C-D-E-F3 A-B-G-D-E-F ...
- 15 kudos
- 8326 Views
- 5 replies
- 5 kudos
What does "ScalaDriverLocal: User Code Compile error" mean?
22/11/30 01:45:31 WARN ScalaDriverLocal: loadLibraries: Libraries failed to be installed: Set() 22/11/30 01:50:14 INFO Utils: resolved command to be run: WrappedArray(getconf, PAGESIZE) 22/11/30 01:50:15 WARN ScalaDriverLocal: User Code Compile err...
- 8326 Views
- 5 replies
- 5 kudos
- 5 kudos
Hi @Werner Stinckens​ thanks for the help. Unfortunately I don't think its so simple - I do have a JAR that I submitted as a Databricks JAR task, and the JAR does have the org.apache.beam class: I guess what I'm trying to understand is what does Scal...
- 5 kudos
- 20569 Views
- 11 replies
- 9 kudos
Why is execution too fast?
I have a table, full scan of which takes ~20 minutes on my cluster. The table has "Time" TIMESTAMP column and "day" DATE column. The latter is computed (manually) as "Time" truncated to day and used for partitioning.I query the table using predicate ...
- 20569 Views
- 11 replies
- 9 kudos
- 9 kudos
Hi @Vladimir Ryabtsev​ ,Because you are creating a delta table, I think that you are seeing a performance improvement because of Dynamic Partition pruning, According to the documentation, "Partition pruning can take place at query compilation time wh...
- 9 kudos
- 2118 Views
- 1 replies
- 3 kudos
Hello, When working in a python notebook and using tab-complete to navigate the file system, I find that pressing enter on a partially completed path ...
Hello,When working in a python notebook and using tab-complete to navigate the file system, I find that pressing enter on a partially completed path will add the full path to the cell in the notebook. This is annoying behaviour, since you end up with...
- 2118 Views
- 1 replies
- 3 kudos
- 3 kudos
Someone heard you In the experimental Monaco editor, I found this particular issue not appearing.
- 3 kudos
- 6985 Views
- 8 replies
- 19 kudos
Python job run error messages are unreadable
This has been going on for some time now; all errors look like this (note the weird `[0;34m` marks everywhere). How can we fix this?We're not doing anything crazy, this is just the latest runtime with pretty much the simplest possible hello world pro...
- 6985 Views
- 8 replies
- 19 kudos
- 19 kudos
Have you tried detaching and reattaching the notebook? Or Cluster restart? Did you check you are not importing any specific library someone else with the right access might have installed some library with install to all clusters as checked.
- 19 kudos
- 11048 Views
- 2 replies
- 6 kudos
Resolved! How to go up two folders using relative path in %run?
I'm wanting to store a notebook with functions two folders up from the current notebook. I know that I can start the path with ../ to go up one folder but when I've tried .../ it won't go up two folders. Is there a way to do this?
- 11048 Views
- 2 replies
- 6 kudos
- 6 kudos
In order to access a notebook in the current folder use ../notebook_2to go 2 folders up and access (say notebook "secret") use ../../secret
- 6 kudos
- 1601 Views
- 1 replies
- 6 kudos
Just a shout out to Databricks Support team and customers!@Joseph Kambourakis​ @Nadia Elsayed​ @Vidula Khanna​ @Jose Gonzalez​ @Harshjot Singh​ you al...
Just a shout out to Databricks Support team and customers!@Joseph Kambourakis​ @Nadia Elsayed​ @Vidula Khanna​ @Jose Gonzalez​ @Harshjot Singh​ you all are fabulous bunch of teams and very helpful.Thanks very much for your responses when asked. Happy...
- 1601 Views
- 1 replies
- 6 kudos
- 6 kudos
@Smitha Nelapati​ so happy to see that the issue is resolved
- 6 kudos
- 9694 Views
- 5 replies
- 6 kudos
Resolved! How to avoid trimming in EXPLAIN?
I am looking on EXPLAIN EXTENDED plan for a statement.In == Physical Plan == section, I go down to FileScan node and see a lot of ellipsis, like +- FileScan parquet schema.table[Time#8459,TagName#8460,Value#8461,Quality#8462,day#8...
- 9694 Views
- 5 replies
- 6 kudos
- 26291 Views
- 5 replies
- 8 kudos
Databricks notebook sometime takes too long to run query (even on empty table)
Hi,sometime I notice that running a query takes too long - even simple queries - and next time when I run same query it runs much faster. I have cluster running (DBR 10.4 LTS • 5 workers) and it has constantly several workers.An Example of query is s...
- 26291 Views
- 5 replies
- 8 kudos
- 8 kudos
Probably the cluster is always in use and the query always falls into the processing query, or the cluster auto stops every time that you use it.
- 8 kudos
- 4208 Views
- 4 replies
- 26 kudos
First post on databricks community
Hello Guys!​This my first databricks community post. Looking forward to contribute from my end
- 4208 Views
- 4 replies
- 26 kudos
- 7575 Views
- 5 replies
- 5 kudos
Mount an uncrypted AWS EFS in AWS Databricks
Hi,I want to mount an uncrypted AWS EFS in AWS Databricks. When I do:mount -t nfs4 -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport fs-abcdef.efs.region.amazonaws.com:/ /mnt/efs-uncryptedI get this error:mount.nfs4: moun...
- 7575 Views
- 5 replies
- 5 kudos
- 5 kudos
"To support NFS under LXC, some of the apparmor protections need to be lifted." (see https://theorangeone.net/posts/mount-nfs-inside-lxc/)
- 5 kudos
- 6334 Views
- 3 replies
- 1 kudos
- 6334 Views
- 3 replies
- 1 kudos
- 1 kudos
Its broken again, i am seeing this several times a week, and it is offline for hours at a time.
- 1 kudos
- 10924 Views
- 1 replies
- 5 kudos
java.util.NoSuchElementException: key not found
Hello,We are using a Azure Databricks with Standard DS14_V2 Cluster with Runtime 9.1 LTS, Spark 3.1.2 and Scala 2.12 and facing the below issue frequently when running our ETL pipeline. As part of the operation that is failing there are several joins...
- 10924 Views
- 1 replies
- 5 kudos
- 5 kudos
Hey man,Please use these configuration in your cluster and it will work,spark.sql.storeAssignmentPolicy LEGACYspark.sql.parquet.binaryAsString truespark.speculation falsespark.sql.legacy.timeParserPolicy LEGACYif it wont work let me know what problem...
- 5 kudos
- 10203 Views
- 1 replies
- 1 kudos
Python: Generate new dfs from a list of dataframes using for loop
I have a list of dataframes (for this example 2) and want to apply a for-loop to the list of frames to generate 2 new dataframes. To start, here is my starting dataframe called df_final:First, I create 2 dataframes: df2_b2c_fast, df2_b2b_fast:for x i...
- 10203 Views
- 1 replies
- 1 kudos
- 1 kudos
thanks
- 1 kudos
- 3218 Views
- 1 replies
- 6 kudos
Databricks Security Review
Conducting a security review or vendor assessment of Databricks and looking to learn more about our security features, compliance information, and privacy policies?You can find the latest on Databricks security features, architecture, compliance and ...
- 3218 Views
- 1 replies
- 6 kudos
- 6 kudos
thanks man
- 6 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 572 | |
| 349 | |
| 287 |
