Databricks

cuteabhi32 · 06-06-2022

from pyspark import SparkContextfrom pyspark import SparkConffrom pyspark.sql.types import *from pyspark.sql.functions import *from pyspark.sql import *from pyspark.sql.types import StringTypefrom pyspark.sql.functions import udfdf1 = spark.read.form...

cuteabhi32 · 06-07-2022

Thanks i modified my code as per your suggestion and it worked perfectly Thanks again for all your inputsdflist= spark.createDataFrame(list(a.columns), "string").toDF("Name")dfg=dflist.filter(col('name').isin('ref_date')).count()if dfg==1 : a = a.wi...

cuteabhi32 · 06-07-2022

yes your understanding is correct i need a column either it may be dynamically created on the fly or pre-existing in the dataframe .if that column is present that function shoud return that column as output if its newly getting created it should retu...

cuteabhi32 · 06-07-2022

AttributeError: 'DataFrame' object has no attribute 'ColumnChecker'def func_udf(df,col): column =list(df.columns) if col in column: return df.col else: df.withColumn("col", lit(null))spark.udf.register("ColumnChecker",func_udf)dfg=a.ColumnC...

cuteabhi32 · 06-07-2022

please find attached the files

cuteabhi32 · 06-07-2022

from pyspark.sql.functions import * from pyspark.sql.functions import colimport pyspark.sql.functions as Ffrom datetime import date,datetimeimport time from dateutil.relativedelta import relativedeltafrom dateutil import parserfrom pyspark.sql.window...

Databricks

User Stats

User Activity

Trying to check if a column exist in a dataframe or not if not then i have to give NULL if yes then i need to give the column itself by using UDF

Re: Trying to check if a column exist in a dataframe or not if not then i have to give NULL if yes then i need to give the column itself by using UDF

Re: Trying to check if a column exist in a dataframe or not if not then i have to give NULL if yes then i need to give the column itself by using UDF

Re: Trying to check if a column exist in a dataframe or not if not then i have to give NULL if yes then i need to give the column itself by using UDF

Re: Trying to check if a column exist in a dataframe or not if not then i have to give NULL if yes then i need to give the column itself by using UDF

Re: Trying to check if a column exist in a dataframe or not if not then i have to give NULL if yes then i need to give the column itself by using UDF