- 4256 Views
- 1 replies
- 0 kudos
How to create a single CSV file with specified file name Spark in Databricks?
I know how to use Spark in Databricks to create a CSV, but it always has lots of side effects.For example, here is my code:file_path = “dbfs:/mnt/target_folder/file.csv”df.write.mode("overwrite").csv(file_path, header=True)Then what I got isA folder ...
- 4256 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @guangyi ,To disable _commited_xxx, _started_xxx and _SUCCSSS you must set below spark options: spark.conf.set("spark.databricks.io.directoryCommit.createSuccessFile","false") spark.conf.set("mapreduce.fileoutputcommitter.marksuccessfuljobs", "f...
- 0 kudos
- 32466 Views
- 5 replies
- 2 kudos
org.apache.spark.sql.AnalysisException: Undefined function: 'MAX'
I am trying to create a JAR for a Azure Databricks job but some code that works when using the notebook interface does not work when calling the library through a job. The weird part is that the job will complete the first run successfully but on an...
- 32466 Views
- 5 replies
- 2 kudos
- 2 kudos
I am facing similar issue when trying to use from_utc_timestamp function. I am able to call the function from databricks notebook but when I use the same function inside my java jar and running as a job in databricks, it is giving below error. Analys...
- 2 kudos
- 6660 Views
- 2 replies
- 0 kudos
- 6660 Views
- 2 replies
- 0 kudos
- 0 kudos
Yes! Cluster's owner/creator can be changed with the REST API - POST /api/2.1/clusters/change-ownerRequest Body sample:{ "cluster_id": "string", "owner_username": "string" }Ref: Clusters API | Change cluster ownerHope this helps!
- 0 kudos
- 6268 Views
- 4 replies
- 0 kudos
How to solve udf performance issue with databricks sql function?
Hi,I am dealing with a situation where i need to secure data at rest on storage (azure data lake), hence saving the data as encrypted text into the delta table. While serving, i want to create dynamic view which will be created from the delta table a...
- 6268 Views
- 4 replies
- 0 kudos
- 0 kudos
I don't clearly understand your full problem. But I do know the following regarding UDFs:1. PySpark UDFs are extremely slow, because it needs to deserialize the Java Object (DataFrame), transform it with the Python UDF, then Serialize it back. This h...
- 0 kudos
- 1895 Views
- 3 replies
- 0 kudos
Unity Catalog external tables
What are the consistency guarantees that DataBricks supply for multi writers, given that the written table is an external table?Are they different from the consistency guarantees given for managed tables? Thanks!
- 1895 Views
- 3 replies
- 0 kudos
- 0 kudos
Thank you @Ajay-Pandey, that is helpful.One thing that I'm not sure about is how does DataBricks can use the same ACID mechanism that external tools use with the external tables? For example, if an external Spark cluster write Delta Logs with a LogSt...
- 0 kudos
- 5571 Views
- 1 replies
- 0 kudos
Resolved! Authentication Issue while connecting to Databricks using Looker Studio
So previously I created source connections from looker with Databricks using my personal access token.I followed this databricks docs. https://docs.databricks.com/en/partners/bi/looker-studio.htmlBut from 10 July, I think basic authentication has bee...
- 5571 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi,You would still connect using OAuth tokens. It is just that Databricks recommends using personal access tokens belonging to service principals instead of workspace users. To create tokens for service principals, see Manage tokens for a service pri...
- 0 kudos
- 6025 Views
- 8 replies
- 1 kudos
Resolved! Question: Issue with Overwatch Deployment on Databricks (on AWS) - Missing Tables in Gold Schema
Hi all,I'm working on setting up Overwatch in our Databricks workspace to monitor resources, and I've encountered an issue during overwatch deployment. I am able to deploy overwatch, but the validation for the `Gold_jobRunCostPotentialFact` module fa...
- 6025 Views
- 8 replies
- 1 kudos
- 1 kudos
Hi @jiteshraut20 , 1) Storage_prefix: It is updated in the documents please reffer config . 2)If the system table is in use, the recommended Databricks runtime version is 13.3 LTS. For other cases, 11.3 LTS should work seamlessly. Please see the docu...
- 1 kudos
- 4127 Views
- 2 replies
- 6 kudos
Getting error popup in databricks
when i migrated to new databricks workspace, I am getting error popup message continuously and also indentation what I changed it is getting changed to other value every with new login .
- 4127 Views
- 2 replies
- 6 kudos
- 6 kudos
Hi I am facing this issue session expired pop-ups frequently every 3 minutes while I start working on databricks.
- 6 kudos
- 3828 Views
- 0 replies
- 0 kudos
Reg- Issue with using Databricks
I created a Community edition account. while i am using for every 30 seconds, it closes by specifying session expired
- 3828 Views
- 0 replies
- 0 kudos
- 1933 Views
- 1 replies
- 0 kudos
Resolved! Question: Decrypt many files with UDF
I have around 20 pgp files in a folder in my volume that I need to decrypt. I have a decryption function that accepts a file name and writes the decrypted file to a new folder in the same volume. I had thought I could create a spark dataframe with th...
- 1933 Views
- 1 replies
- 0 kudos
- 0 kudos
Spark's error happens because the worker nodes can't access your local files. Instead of using Spark to decrypt, try doing it outside of Spark using Python's multiprocessing or a simple batch script for parallel processing. Another option is to move ...
- 0 kudos
- 2636 Views
- 4 replies
- 2 kudos
Delta Table Properties
Hi When I look at the properties of a delta table I see these two properties that are set to a value of 1.I went through the manual for these properties and this is what the manual says.delta.minReaderVersion: The minimum required protocol reader ver...
- 2636 Views
- 4 replies
- 2 kudos
- 2 kudos
Hi Ramdas1,Let me explain you in simple terms with an example.A Delta table is like a special book for data. delta.minReaderVersion is the minimum version of the "reader" you need to open and read the book, while delta.minWriterVersion is the minimum...
- 2 kudos
- 6878 Views
- 5 replies
- 4 kudos
Resolved! Out of Memory/Connection Lost When Writing to External SQL Server from Databricks Using JDBC Connection
I am working on writing a large amount of data from Databricks to an external SQL server using a JDB connection. I keep getting timeout errors/connection lost but digging deeper it appears to be a memory problem. I am wondering what cluster configura...
- 6878 Views
- 5 replies
- 4 kudos
- 1481 Views
- 0 replies
- 1 kudos
PySpark UDFs: Leveraging Custom Functions for Data Transformation
PySpark UDFs offer a powerful mechanism for applying custom transformations to data within Spark DataFrames. While they provide flexibility and code reusability, they also come with performance overhead and debugging challenges. By understanding the ...
- 1481 Views
- 0 replies
- 1 kudos
- 4549 Views
- 3 replies
- 1 kudos
Exclude absent lookup keys from dataframes made by create_training_set()
I've got data stored in feature tables, plus in a data lake. The feature tables are expected to lag the data lake by at least a little bit. I want to filter data coming out of the feature store by querying the data lake for lookup keys out of my inde...
- 4549 Views
- 3 replies
- 1 kudos
- 1 kudos
I'm facing the same issue as described by @mrcity. There is no easy way to alter the dataframe, which is created inside the score_batch() function. Filtering out rows in the (sklearn) pipeline itself is also not convenient since these transformers ar...
- 1 kudos
- 2139 Views
- 1 replies
- 0 kudos
Databricks Liquid Clustering
Liquid Clustering is a combination of Partitioning and Zordering. As we know, partitioning will create folders based on the column values and stores the similar value. I believe Liquid Clustering will not create folders like partitioning, so how it w...
- 2139 Views
- 1 replies
- 0 kudos
- 0 kudos
why do you think Liquid Clustering uses Z ordering? I recommend you read the design paper https://docs.google.com/document/d/1FWR3odjOw4v4-hjFy_hVaNdxHVs4WuK1asfB6M6XEMw/edit#heading=h.skpz7c7ga1wl
- 0 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 584 | |
| 349 | |
| 287 |
