- 19969 Views
- 6 replies
- 8 kudos
Resolved! how to flatten non standard Json files in a dataframe
hello, I have a non standard Json file with a nested file structure that I have issues with. Here is an example of the json file. jsonfile= """[ { "success":true, "numRows":2, "data":{ "58251":{ "invoiceno":"58...
- 19969 Views
- 6 replies
- 8 kudos
- 8 kudos
@stale stokkereit​ You can use the below function to flatten the struct fieldimport pyspark.sql.functions as F def flatten_df(nested_df): flat_cols = [c[0] for c in nested_df.dtypes if c[1][:6] != 'struct'] nested_cols = [c[0] for c in nest...
- 8 kudos
- 6021 Views
- 4 replies
- 2 kudos
Resolved! cannot resolve '(CAST(10000 AS BIGINT) div Khe)' due to data type mismatch:
Hi,I'm trying to create a delta table using SQL but I'm getting this errorError in SQL statement: AnalysisException: cannot resolve '(CAST(10000 AS BIGINT) div Khe)' due to data type mismatch: differing types in '(CAST(10000 AS BIGINT) div Khe)' (big...
- 6021 Views
- 4 replies
- 2 kudos
- 2 kudos
Hi @Adalberto Garcia Espinosa​ Do you need khe column to be double? If not, below query is working:%sql CREATE OR REPLACE TABLE Productos(Khe bigint NOT NULL,Fctor_HL_Estiba bigint GENERATED ALWAYS AS (cast(10000 as bigint) div Khe)) seems to be work...
- 2 kudos
- 6448 Views
- 5 replies
- 8 kudos
Resolved! Access azure storage account from databricks notebook using pyspark or SQL
I have a storage account - Azure BLOB StorageThere I had container. Inside the container we had a CSV file. Couldn't read the file using the access Key and Storage account name.Any idea how to read file using PySpark/SQL? Thanks in advance
- 6448 Views
- 5 replies
- 8 kudos
- 8 kudos
@Ambiga D​ you need to mount the storage https://docs.databricks.com/data/data-sources/azure/azure-storage.html#mount-azure-blob-storage-containers-to-dbfs you can follow this,thanks.
- 8 kudos
- 25708 Views
- 5 replies
- 9 kudos
Resolved! How to write *.csv file from DataBricks FileStore
Struggling with how to export a Spark dataframe as a *.csv file to a local computer. I'm successfully using the spark_write_csv funciton (sparklyr R library R) to write the csv file out to my databricks dbfs:FileStore location. Becase (I'm assuming)...
- 25708 Views
- 5 replies
- 9 kudos
- 9 kudos
sparklyr has a different syntax. There is function sdf_coalesce.The code which you paste is for Scala/Python. Additionally, even in python you can only specify folder not file so CSV("dbfs:FileStore/temp/")
- 9 kudos
- 917 Views
- 0 replies
- 2 kudos
www.vandevelde.eu
June Featured Member of the Month ! Werner Stinckens Job Title: Data Engineer @ Van de Velde (www.vandevelde.eu)What are three words your coworkers would use to describe you?Helpful, accurate, inquisitiveWhat is your favorite thing about your curren...
- 917 Views
- 0 replies
- 2 kudos
- 12853 Views
- 13 replies
- 2 kudos
Resolved! Couldn't convert string to float when fit model
Hi, I am very new in databricks and I am trying to run quick experiments to understand the best practice for me, my colleagues and the company.I pull the data from snowflakedf = spark.read \ .format("snowflake") \ .options(**options) \ .option('qu...
- 12853 Views
- 13 replies
- 2 kudos
- 13338 Views
- 5 replies
- 12 kudos
Resolved! how to convert each row of df to array of rows(list of rows)
Hi,How to convert each row of dataframe to array of rows?Here is our scenario , we need to pass each row of dataframe to one function as dict to apply the key level transformations. But as our data is very huge we can't use collect df.toJson().colle...
- 13338 Views
- 5 replies
- 12 kudos
- 12 kudos
@Hubert Dudek​ , Thank you for the reply. We are new to ADB. And using the below code, looking for an optimized way to do itdfJSONString = df.toJSON().collect()stringList = [] for row in dfJSONString: # ==== Unflatten the JSON string ==== # js...
- 12 kudos
- 13735 Views
- 8 replies
- 3 kudos
Resolved! Remote RPC client disassociated error
Hello,I've been trying to submit a job to a transient cluster, but it is failing with this error :Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in ...
- 13735 Views
- 8 replies
- 3 kudos
- 3 kudos
@Alix Métivier​ - The error is thrown from the user code (please investigate the jar file attached to the cluster). at m80.dbruniv_0_1.dbruniv.tFixedFlowInput_1Process(dbruniv.java:941)at m80.dbruniv_0_1.dbruniv.run(dbruniv.java:1654)at m80.dbruniv_...
- 3 kudos
- 53112 Views
- 11 replies
- 1 kudos
Resolved! Trying to check if a column exist in a dataframe or not if not then i have to give NULL if yes then i need to give the column itself by using UDF
from pyspark import SparkContextfrom pyspark import SparkConffrom pyspark.sql.types import *from pyspark.sql.functions import *from pyspark.sql import *from pyspark.sql.types import StringTypefrom pyspark.sql.functions import udfdf1 = spark.read.form...
- 53112 Views
- 11 replies
- 1 kudos
- 1 kudos
Thanks i modified my code as per your suggestion and it worked perfectly Thanks again for all your inputsdflist= spark.createDataFrame(list(a.columns), "string").toDF("Name")dfg=dflist.filter(col('name').isin('ref_date')).count()if dfg==1 : a = a.wi...
- 1 kudos
- 1104 Views
- 0 replies
- 0 kudos
What are Canyon Creek Condos and what do they offer residents?Canyon Creek Condos are a type of housing that is becoming increasingly popular in the U...
What are Canyon Creek Condos and what do they offer residents?Canyon Creek Condos are a type of housing that is becoming increasingly popular in the United States. These types of condos are typically located in rural or suburban areas and offer resid...
- 1104 Views
- 0 replies
- 0 kudos
- 11181 Views
- 3 replies
- 1 kudos
Resolved! Plotting in pyspark.pandas Uncaught ReferenceError Plotly is not defined
Hi, I am trying to plot using pyspark.pandas running this sample code: speed = [0.1, 17.5, 40, 48, 52, 69, 88] lifespan = [2, 8, 70, 1.5, 25, 12, 28] index = ['snail', 'pig', 'elephant', 'rabbit', 'giraffe', 'coyote', 'horse'] psdf = ps.Data...
- 11181 Views
- 3 replies
- 1 kudos
- 1 kudos
Thank you @Werner Stinckens​ . I was able to find the plotly documentation listed below and setting the output_type and calling displayHTML() helped remedy the error.
- 1 kudos
- 3991 Views
- 2 replies
- 1 kudos
SQL Analytics Map Visualization: Map marker size
Hello all, I am trying to use the Map visualization in SQL Analytics Dashboard in Databricks. Does any one knows how or if we can change the size/radius of the markers based on values in another column. This seems like a very trivial parameter but I ...
- 3991 Views
- 2 replies
- 1 kudos
- 6093 Views
- 5 replies
- 1 kudos
Autoloader and "cleanSource"
Hi All, We are trying to use the Spark 3 structured streaming feature/option ".option('cleanSource','archive')" to archive processed files. This is working as expected using the standard spark implementation, however does not appear to work using aut...
- 6093 Views
- 5 replies
- 1 kudos
- 1 kudos
https://docs.databricks.com/ingestion/auto-loader/options.html#common-auto-loader-optionscleanSource is not a listed option so it won't do anything.Maybe event retention is something you can use?
- 1 kudos
- 8241 Views
- 3 replies
- 3 kudos
Is there a way to CONCAT two dataframes on either of the axis (row/column) and transpose the dataframe in PySpark?
I'm reshaping my dataframe as per requirement and I came across this situation where I'm concatenating 2 dataframes and then transposing them. I've done this previously using pandas and the syntax for pandas goes as below:import pandas as pd df1 = ...
- 8241 Views
- 3 replies
- 3 kudos
- 3 kudos
Hi @Kaniz Fatma​ ,I no longer see the answer you've posted, but I see you were suggesting to use `union`. As per my understanding, union are used to stack the dfs one upon another with similar schema / column names.In my situation, I have 2 different...
- 3 kudos
- 1776 Views
- 1 replies
- 0 kudos
Workflow Pipeline in Azure Databrick is throwing error for EventHubsSourceProvider could not be instantiated
I am using the sample code which is available in getting start tutorial. And it is simple read the json file and move in another table. But it is throwing error related to EventHubsSourceProvider
- 1776 Views
- 1 replies
- 0 kudos
| User | Count |
|---|---|
| 1624 | |
| 790 | |
| 504 | |
| 349 | |
| 287 |
