- 1497 Views
- 1 replies
- 1 kudos
medium.com
Sorting In Spark**How to sort null values First and last of the records in the Spark data frame?Please find the answershttps://medium.com/@sharikrishna26/sorting-in-spark-a57db245ecd4
- 1497 Views
- 1 replies
- 1 kudos
- 1 kudos
Yeah this is really good post,keep it up Man
- 1 kudos
- 2959 Views
- 0 replies
- 31 kudos
Understanding Cluster Pools Sometimes we want to run our databricks code without any delay as reports are very emergency like the upstream team wants ...
Understanding Cluster PoolsSometimes we want to run our databricks code without any delay as reports are very emergency like the upstream team wants to save as much time as they can save in the starting cluster.That time we can use the pool of cluste...
- 2959 Views
- 0 replies
- 31 kudos
- 3157 Views
- 0 replies
- 31 kudos
Databricks New Runtime Version is Available Now  PySpark memory profiling- Memory profiling is now enabled for PySpark user-defined functions. This pr...
Databricks New Runtime Version is Available Now PySpark memory profiling- Memory profiling is now enabled for PySpark user-defined functions. This provides information on memory increment, memory usage, and number of occurrences for each line of code...
- 3157 Views
- 0 replies
- 31 kudos
- 4071 Views
- 1 replies
- 2 kudos
error too large report
hi i am trying to pull the data from quick base but it is giving me error-: too large reportbelow are the code i used@%pythondf = quickbasePull('b5zj8k_pbz5_0_cd5h4wbb77n4nvp95b4u','bq2nq8jm7',4)2) i tried below code but its not displaying in correc...
- 4071 Views
- 1 replies
- 2 kudos
- 2 kudos
Hey @ahana ahana​ ,this code is not working
- 2 kudos
- 17327 Views
- 6 replies
- 6 kudos
- 17327 Views
- 6 replies
- 6 kudos
- 3672 Views
- 1 replies
- 2 kudos
spark.apache.org
Hey fellow co-workers!!I have been doing the Apache Spark programming in Databricks academy and I realized the hyperlinks here it doesn't work.Spark session https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql.html#spark-session-apis
- 3672 Views
- 1 replies
- 2 kudos
- 2 kudos
@Deiry Navas​ Could you share the repo link usued to exectue that notebook? I'll on my side
- 2 kudos
- 2224 Views
- 1 replies
- 0 kudos
Why are the get..Id() functions returning 'some(123456)' instead of just the id?
Hey fellow users, I've successfully retrieved the notebook context during job runs and there are several getId calls. For some reason when the ids are returned, they are wrapped in a some() instead of just the number. Does anyone know why this is the...
- 2224 Views
- 1 replies
- 0 kudos
- 0 kudos
Well, my post for me is irrelevant now!! I just stumbled across this beauty which avoids me having to do any of this and deal with odd return values.How to get the Job ID and Run ID and save into a database (databricks.com)Are the braces {{job_id}} n...
- 0 kudos
- 10738 Views
- 6 replies
- 6 kudos
External Table issue format in databricks
I am new to databricksI am trying to create a external table in databricks with below format :CREATE EXTERNAL TABLE Salesforce.Account( Id string , IsDeleted bigint, Name string , Type string , RecordTypeId string , ParentId string , ShippingSt...
- 10738 Views
- 6 replies
- 6 kudos
- 6 kudos
Databricks is awesome if you have SQL knowledge....I just came across one of my problem in my project and databricks helped me a lot....like a use of low watermark to hold the load success date .....​
- 6 kudos
- 5018 Views
- 2 replies
- 3 kudos
collapse partial code in large cell?
In databricks notebook, we have SQL cells that are over 700 lines long. Is there a way to collapse a portion of the code vs scrolling? Looking for something similar to what exists in Netezza, "--region" and "--end region" where anything between those...
- 5018 Views
- 2 replies
- 3 kudos
- 3 kudos
Hi @james t​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks!
- 3 kudos
- 1803 Views
- 1 replies
- 0 kudos
- 1803 Views
- 1 replies
- 0 kudos
- 0 kudos
it is just for better security and data governance
- 0 kudos
- 43824 Views
- 7 replies
- 0 kudos
How to get the path of files using python os library
unable to find the path to directory with os library
- 43824 Views
- 7 replies
- 0 kudos
- 9246 Views
- 4 replies
- 1 kudos
How to print the path of a .py file or a notebook?
I have stored a test.py in the dbfs at the below location "/dbfs/FileStore/shared_uploads/krishna@company.com/Project_Folder/test.py"I have a print statement in test.py which says the belowprint( os.getcwd() )and it prints the below'/databricks/drive...
- 9246 Views
- 4 replies
- 1 kudos
- 1 kudos
Hey @Krishna Zanwar​ Please use the below code this will work and as you want the specific location you can create a custom code and format the path using a python formatter , it will give you desired result .
- 1 kudos
- 4595 Views
- 1 replies
- 2 kudos
Resolved! org.apache.http.conn.ConnectTimeoutException: What does this mean and how can we resolve it.
My team has run into getting this error pretty frequently on one of our larger jobs. I've set out retry policy to 5 and that seems to fix it and keep the job going. It seems like it's unable to pick up the task immediately but can after it's complete...
- 4595 Views
- 1 replies
- 2 kudos
- 2 kudos
Hey @Coleman Milligan​ ,I also faced this type of issue many times you can add the below configuration in your cluster and it should work.spark.executor.heartbeatInterval 60sspark.network.timeout 120sFor more details, you can explore this doc - https...
- 2 kudos
- 6200 Views
- 2 replies
- 2 kudos
cannot convert Parquet type INT64 to Photon type double
I am trying to read in files via the COPY INTO command but I am getting this error lately for a certain subset of the data;`Error while reading file: Schema conversion error: cannot convert Parquet type INT64 to Photon type double`These are my option...
- 6200 Views
- 2 replies
- 2 kudos
- 2 kudos
hey @Andrew Fogarty​ I also faced the same issue when I moved from the 7.3 LTS version to a higher runtime version so to mitigate this issue you can use the below cluster configuration spark.sql.storeAssignmentPolicy LEGACYspark.sql.parquet.binaryAsS...
- 2 kudos
- 3692 Views
- 4 replies
- 0 kudos
Resolved! Safari problems after the maintenance on 12/9/2022
I'm experience some problems on Safari 15.3 ( MacOS )I would like to know if I am alone in this and how to fix ( if I can ) this.This is the Databricks SQLData science and Engineering. ( is this case Workflows).
- 3692 Views
- 4 replies
- 0 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 582 | |
| 349 | |
| 287 |
