- 8146 Views
- 10 replies
- 6 kudos
Older Spark Version loaded into the spark notebook
I have databricks runtime for a job set to latest 10.0 Beta (includes Apache Spark 3.2.0, Scala 2.12) .In the notebook when I check for the spark version, I see version 3.1.0 instead of version 3.2.0I need the Spark version 3.2 to process workloads a...
- 8146 Views
- 10 replies
- 6 kudos
- 6 kudos
hi @Dhaivat Upadhyay​ ,Good news, DBR 10 was release yesterday October 20th. You can find more details in the release notes website
- 6 kudos
- 2093 Views
- 2 replies
- 6 kudos
Where can I enable notification for comments made on my notebook?
I'm not being communicated when a comment is made on a notebook, is it possible to enable?
- 2093 Views
- 2 replies
- 6 kudos
- 21803 Views
- 15 replies
- 6 kudos
Resolved! BufferHolder Exceeded on Json flattening
On Databricks, we use the following code to flatten JSON in Python. The data is from a REST API:```df = spark.read.format("json").option("header", "true").option("multiline", "true").load(SourceFileFolder + sourcetable + "*.json")df2 = df.select(psf....
- 21803 Views
- 15 replies
- 6 kudos
- 6 kudos
@Dennis D​ , what's happening here is that more than 2 GB (2147483648 bytes) is being loaded into a single column value. This is a hard-limit for serialization. This KB article addresses it. The solution would be to find some way to have this loaded ...
- 6 kudos
- 2079 Views
- 4 replies
- 3 kudos
Feature request: It is possible to add comments to both databricks sql databases and tables. It would be really usefull if these comments could show u...
Feature request: It is possible to add comments to both databricks sql databases and tables. It would be really usefull if these comments could show up (if they are provided) in PowerBI when one connects to the Databricks SQL endpoint, e.g. in this w...
- 2079 Views
- 4 replies
- 3 kudos
- 4130 Views
- 6 replies
- 5 kudos
Resolved! How to implement the where not exists pattern in scala?
I have a dataframe with the following columns:Key1Key2Y_N_ColCol1Col2For the key tuple (Key1, Key2), I have rows with Y_N_Col = "Y" and Y_N_Col = "N".I need a new dataframe with all rows with Y_N_Col = "Y" (regardless of the key tuple), plus all Y_N_...
- 4130 Views
- 6 replies
- 5 kudos
- 5 kudos
I'd use a left-anti join.So create a df with all the Y, then create a df with all the N and do a left_anti join (on key1 and key2) on the df with the Y.then a union of those two.
- 5 kudos
- 761 Views
- 0 replies
- 0 kudos
aws logo
What is the future of aws?The future of AWS is very promising. So, if you are thinking of a cloud career or want to switch your position to something related to the cloud, I would highly recommend you going for AWS training. No matter what field you ...
- 761 Views
- 0 replies
- 0 kudos
- 2201 Views
- 0 replies
- 0 kudos
- 2201 Views
- 0 replies
- 0 kudos
- 1280 Views
- 1 replies
- 0 kudos
If data from a Delta table is cached in Databricks SQL and the table is altered in the backend, does it invalidate the cache?
Basically I'm worried about the scenario where data that gets cached on Databricks SQL endpoints becomes out of sync with the source Delta table. If that were to happen and data was read from the cache it would be out of date/incorrect. Is this a con...
- 1280 Views
- 1 replies
- 0 kudos
- 0 kudos
There are 3 types of caching. 1-Databricks SQL UI caching, 2-Query results caching , 3-Delta caching . (1) does not get invalidated. It's like your BI dashboard. BI dashboard needs to be manually refreshed.(2) and (3) gets auto invalidation.pls check...
- 0 kudos
- 3561 Views
- 1 replies
- 1 kudos
Resolved! How to create a temporary file with sql
what are the commands to create a temporary file with SQL
- 3561 Views
- 1 replies
- 1 kudos
- 1 kudos
In Spark SQL, you could use commands like "insert overwrite directory" that indirectly creates a temporary file with the datahttps://docs.databricks.com/spark/latest/spark-sql/language-manual/sql-ref-syntax-dml-insert-overwrite-directory.html#example...
- 1 kudos
- 2329 Views
- 1 replies
- 3 kudos
Resolved! Write mode features in Bigquey using Databricks notebook.
Currently using df.write.format("bigquery") ,Databricks only supports append and Overwrite modes in to writing Bigquery tables.Does Databricks has any option of executing the DMLs like Merge in to Bigquey using Databricks Notebooks.?
- 2329 Views
- 1 replies
- 3 kudos
- 3 kudos
@Sumeet Dora​ , Unfortunately there is no direct "merge into" option for writing to Bigquery using Databricks notebook. You could write to an intermediate delta table using the "merge into" option in delta table. Then read from the delta table and pe...
- 3 kudos
- 7000 Views
- 8 replies
- 9 kudos
Slow performance of VACUUM on Azure Data Lake Store Gen2
We need to run VACCUM on one of our biggest tables to free the storage. According to our analysis using VACUUM bigtable DRY RUN this affects 30M+ files that need to be deleted.If we run the final VACUUM, the file-listing takes up to 2h (which is OK) ...
- 7000 Views
- 8 replies
- 9 kudos
- 9 kudos
@Gerhard Brueckl​ we have seen near 80k-120k file deletions in Azure per hour while doing a VACUUM on delta tables, it's just that the vacuum is slower in azure and S3. It might take some time as you said when deleting the files from the delta path. ...
- 9 kudos
- 7362 Views
- 6 replies
- 2 kudos
Run more than nr-of-cores concurrent tasks.
We are using the terraform databricks provier, which is starting a cluster and checking every mount (since there is no mount rest API!). Each mount takes 20 seconds to check, and 99.9% of that time is idle waiting, and it starts a job per mount. If w...
- 7362 Views
- 6 replies
- 2 kudos
- 2 kudos
hi @Erik Parmann​ ,It is possible to do, but you might need to also enable dynamic allocation at the cluster level to be able to make sure your settings are apply at cluster creation . You can find more details here. As best practice, we do not recom...
- 2 kudos
- 17796 Views
- 3 replies
- 5 kudos
How can I use custom python library in Azure Databricks?
I am trying to access functions in my coreapi.py by importing in the main notebook, but I have error ModuleNotFoundError: No module named 'coreapi'. I tried by uploading the file into the same folder and I tried creating a python egg and uploading it...
- 17796 Views
- 3 replies
- 5 kudos
- 5 kudos
There is also the possibility to use Repos file functionality:https://databricks.com/blog/2021/10/07/databricks-repos-is-now-generally-available.html
- 5 kudos
- 1486 Views
- 0 replies
- 0 kudos
apt-get install texlive error 404
Hi everybody, I'm trying to install on a cluster (Azure Databricks, DBR 7.3LTS) texlive-full using apt-get install texlive-full in an init script.The issue is that, most of the times (not always), I get a 404 when downloading packages from security.u...
- 1486 Views
- 0 replies
- 0 kudos
- 12737 Views
- 9 replies
- 7 kudos
Resolved! Delete row from table is not working.
I have created External table using spark via below command. (Using Data science & Engineering)df.write.mode("overwrite").format("parquet").saveAsTable(name=f'{db_name}.{table_name}', path="dbfs:/reports/testing")I have tried to delete a row based on...
- 12737 Views
- 9 replies
- 7 kudos
- 7 kudos
hi @karthick J​ ,Can you try to delete the row and execute your command in a non high concurrency cluster? the reason why im asking this is because we first need to isolate the error message and undertand why is happening to be able to find the best ...
- 7 kudos
| User | Count |
|---|---|
| 1611 | |
| 768 | |
| 348 | |
| 286 | |
| 252 |
