Databricks Community

AmineHY · 11-13-2023

HelloI having an issue with using a window in pyspark. I created a simple example that reproduce the error, basically i want to define a window with a dynamic size (say from another column ) instead of a fix value from pyspark.sql import SparkSessio...

AmineHY · 04-28-2023

I am having this error when running my cluster, any idea why?

AmineHY · 01-11-2023

I am working on Databricks Notebook and trying to display a map using Floium and I keep getting this error > Command result size exceeds limit: Exceeded 20971520 bytes (current = 20973510)How can I get increase the memory limit?I already reduced the...

AmineHY · 11-16-2022

HelloI am trying to read this JSON file but didn't succeed You can see the head of the file, JSON inside a list of lists. Any idea how to read this file?

AmineHY · 11-14-2022

Python Commanddf = spark.read.format('csv').option('sep', ';').option("recursiveFileLookup", "true").load('dbfs:/***/data_files/PREVISIONS/')Here is the content of the folder Each folder contain the following files: Full logorg.apache.spark.sql.stre...

AmineHY · 01-11-2023

Thanks, Daniel, I have read this article, and I guess this problem occurs only while rendering the output, I'll try to save the result to disk in HTML format and visualize the results through the browser.Have a good day.Amine.

AmineHY · 11-21-2022

Here is my solution, I am sure it can be optimizedimport json data=[] with open(path_to_json_file, 'r') as f: data.extend(json.load(f)) df = spark.createDataFrame(data[0], schema=schema)✌️

AmineHY · 11-21-2022

For now I red the file as a Text and it gives :it is the concatenation of multiple JSON files, that why the native JSON parser can't load the data. The surprising thing is that it detects the right schema!

AmineHY · 11-21-2022

Thank you for sharing,these links do not address the exact problem I am facing

AmineHY · 11-16-2022

Yes some of the files I don't have the right to access (mistakenly) In this case, how do you think I can tell DTL to handle this exception and ignore the file, since I can read some files but not all?

Databricks Community

User Stats

User Activity

Using Window with a dynamic size do not work

Error When Starting the Cluster

How to get rid of "Command result size exceeds limit"

How to read JSON files embedded in a list of lists?

My DLT pipeline return ACL Verification Failed

Re: How to get rid of "Command result size exceeds limit"

Re: How to read JSON files embedded in a list of lists?

Re: How to read JSON files embedded in a list of lists?

Re: How to read JSON files embedded in a list of lists?

Re: My DLT pipeline return ACL Verification Failed