- 7161 Views
- 4 replies
- 1 kudos
ANALYZE TABLE showing NULLs for all statistics in Spark
var df2 = spark.read .format("csv") .option("sep", ",") .option("header", "true") .option("inferSchema", "true") .load("src/main/resources/datasets/titanic.csv") df2.createOrReplaceTempView("titanic") spark.table("titanic").cach...
- 7161 Views
- 4 replies
- 1 kudos
- 1 kudos
can you share what the *newtitanic* is I think that you would have done something similarspark.sql("create table newtitanic as select * from titanic")something like this works for me, but the issue is i first make a temp view then again create a tab...
- 1 kudos
- 3249 Views
- 1 replies
- 0 kudos
How to install GDAL on Databricks Cluster ?
I am currently using Runtime 10.4 LTS.The options available on Maven Central does not work as well as on PyPi.I am running:try: from osgeo import gdal except ImportError: import gdalto validate but it throws ModuleNotFoundError: No module n...
- 3249 Views
- 1 replies
- 0 kudos
- 0 kudos
@Abhishek Jain​ I can understand your issue it happens to me also multiple times so solving this issue I used to install the init script in my clusterMajor reason is that your 10X version does not support your current library so you have to find rig...
- 0 kudos
- 14221 Views
- 1 replies
- 1 kudos
AttributeError: 'SparkSession' object has no attribute '_wrapped' when attempting CoNLL.readDataset()
I'm getting the error...AttributeError: 'SparkSession' object has no attribute '_wrapped'---------------------------------------------------------------------------AttributeError Traceback (most recent call last)<command-2311820097584616> in <cell li...
- 14221 Views
- 1 replies
- 1 kudos
- 1 kudos
this can happen in 10X version try to use 7.3 LTS and share your observationand if it not working there try to create init script and load it to your databricks cluster so whenever your machine go up you can get advantage of that library because some...
- 1 kudos
- 3203 Views
- 1 replies
- 5 kudos
Not able to parse .doc extension file using scala in databricks notebook?
I could able to parse .doc extension files using Java programming with the help of POI libraries but when trying to convert Java code into Scala i expect it has to work with same java libraries with Scala programming but it is showing with below erro...
- 3203 Views
- 1 replies
- 5 kudos
- 5 kudos
Hi @Ramesh Bathini​ In pyspark, we have a docx module. I found that to be working perfectly fine. Can you try using that ?Documentation and stuff could be found online. Cheers...
- 5 kudos
- 2969 Views
- 4 replies
- 3 kudos
- 2969 Views
- 4 replies
- 3 kudos
- 3 kudos
I used to get these kind of error from databricks partner page, try to manually search the course that you are looking for. For example, when i used the link to navigate to datalakehouse foundational course page it showed the same error to me. i manu...
- 3 kudos
- 10378 Views
- 2 replies
- 0 kudos
Resolved! delete files from the directory
Is there a way to delete recursively files using a command in notebookssince in the below directory i have many combination of files like .txt,,png,.jpg but i only want to delete files with .csv example dbfs:/FileStore/.csv*
- 10378 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @Rakesh Reddy Gopidi​ You can use the os module to iterate over a directory.By using a loop over the directory, you can check what the file ends with using .endsWith(".csv).After fetching all the files, you can remove it. Hope this helps..Cheers.
- 0 kudos
- 8957 Views
- 2 replies
- 15 kudos
Resolved! Pyspark dataframe column comparison
I have a string column which is a concatenation of elements with a hyphen as follows. Let 3 values from that column looks like below, Row 1 - A-B-C-D-E-FRow 2 - A-B-G-C-D-E-FRow 3 - A-B-G-D-E-FI want to compare 2 consecutive rows and create a column ...
- 8957 Views
- 2 replies
- 15 kudos
- 15 kudos
Hi,I think you can follow these steps:1. Use window function to create a new column by shifting, then your df will look like thisid value lag1 A-B-C-D-E-F null2 A-B-G-C-D-E-F A-B-C-D-E-F3 A-B-G-D-E-F ...
- 15 kudos
- 8419 Views
- 5 replies
- 5 kudos
What does "ScalaDriverLocal: User Code Compile error" mean?
22/11/30 01:45:31 WARN ScalaDriverLocal: loadLibraries: Libraries failed to be installed: Set() 22/11/30 01:50:14 INFO Utils: resolved command to be run: WrappedArray(getconf, PAGESIZE) 22/11/30 01:50:15 WARN ScalaDriverLocal: User Code Compile err...
- 8419 Views
- 5 replies
- 5 kudos
- 5 kudos
Hi @Werner Stinckens​ thanks for the help. Unfortunately I don't think its so simple - I do have a JAR that I submitted as a Databricks JAR task, and the JAR does have the org.apache.beam class: I guess what I'm trying to understand is what does Scal...
- 5 kudos
- 21015 Views
- 11 replies
- 9 kudos
Why is execution too fast?
I have a table, full scan of which takes ~20 minutes on my cluster. The table has "Time" TIMESTAMP column and "day" DATE column. The latter is computed (manually) as "Time" truncated to day and used for partitioning.I query the table using predicate ...
- 21015 Views
- 11 replies
- 9 kudos
- 9 kudos
Hi @Vladimir Ryabtsev​ ,Because you are creating a delta table, I think that you are seeing a performance improvement because of Dynamic Partition pruning, According to the documentation, "Partition pruning can take place at query compilation time wh...
- 9 kudos
- 2149 Views
- 1 replies
- 3 kudos
Hello, When working in a python notebook and using tab-complete to navigate the file system, I find that pressing enter on a partially completed path ...
Hello,When working in a python notebook and using tab-complete to navigate the file system, I find that pressing enter on a partially completed path will add the full path to the cell in the notebook. This is annoying behaviour, since you end up with...
- 2149 Views
- 1 replies
- 3 kudos
- 3 kudos
Someone heard you In the experimental Monaco editor, I found this particular issue not appearing.
- 3 kudos
- 7140 Views
- 8 replies
- 19 kudos
Python job run error messages are unreadable
This has been going on for some time now; all errors look like this (note the weird `[0;34m` marks everywhere). How can we fix this?We're not doing anything crazy, this is just the latest runtime with pretty much the simplest possible hello world pro...
- 7140 Views
- 8 replies
- 19 kudos
- 19 kudos
Have you tried detaching and reattaching the notebook? Or Cluster restart? Did you check you are not importing any specific library someone else with the right access might have installed some library with install to all clusters as checked.
- 19 kudos
- 11153 Views
- 2 replies
- 6 kudos
Resolved! How to go up two folders using relative path in %run?
I'm wanting to store a notebook with functions two folders up from the current notebook. I know that I can start the path with ../ to go up one folder but when I've tried .../ it won't go up two folders. Is there a way to do this?
- 11153 Views
- 2 replies
- 6 kudos
- 6 kudos
In order to access a notebook in the current folder use ../notebook_2to go 2 folders up and access (say notebook "secret") use ../../secret
- 6 kudos
- 1633 Views
- 1 replies
- 6 kudos
Just a shout out to Databricks Support team and customers!@Joseph Kambourakis​ @Nadia Elsayed​ @Vidula Khanna​ @Jose Gonzalez​ @Harshjot Singh​ you al...
Just a shout out to Databricks Support team and customers!@Joseph Kambourakis​ @Nadia Elsayed​ @Vidula Khanna​ @Jose Gonzalez​ @Harshjot Singh​ you all are fabulous bunch of teams and very helpful.Thanks very much for your responses when asked. Happy...
- 1633 Views
- 1 replies
- 6 kudos
- 6 kudos
@Smitha Nelapati​ so happy to see that the issue is resolved
- 6 kudos
- 10031 Views
- 5 replies
- 6 kudos
Resolved! How to avoid trimming in EXPLAIN?
I am looking on EXPLAIN EXTENDED plan for a statement.In == Physical Plan == section, I go down to FileScan node and see a lot of ellipsis, like +- FileScan parquet schema.table[Time#8459,TagName#8460,Value#8461,Quality#8462,day#8...
- 10031 Views
- 5 replies
- 6 kudos
- 26488 Views
- 5 replies
- 8 kudos
Databricks notebook sometime takes too long to run query (even on empty table)
Hi,sometime I notice that running a query takes too long - even simple queries - and next time when I run same query it runs much faster. I have cluster running (DBR 10.4 LTS • 5 workers) and it has constantly several workers.An Example of query is s...
- 26488 Views
- 5 replies
- 8 kudos
- 8 kudos
Probably the cluster is always in use and the query always falls into the processing query, or the cluster auto stops every time that you use it.
- 8 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 576 | |
| 349 | |
| 287 |
