Showing results for 
Search instead for 
Did you mean: 
since ‎06-24-2022

User Stats

  • 15 Posts
  • 1 Solutions
  • 3 Kudos given
  • 8 Kudos received

User Activity

I have the below schemaschema = StructType([StructField(name="Test",dataType=StringType(),nullable=False,metadata={"description": "This is to test metadata description."})])data = [('Test1',), ('Test2',), ('Test3',)] df = spark.createDataFrame(data, ...
I am loading a table into a data frame using df = spark.table(table_name) Is there a way to load only the required columns? The table has more than 50+ columns and I only need a handful of column.
I have a class in a python file like this from pyspark.sql import SparkSession from pyspark.dbutils import DBUtils class DatabricksUtils: def __init__(self‌‌): self.spark = SparkSession.getActiveSession() self.dbutils = DBUtil...
I signed up for the data engineering course and downloaded the course material.However I cannot access the link to import the course material into databricks. Below link gives me access denied.
I have several functions accessing the same createorreplacetempview("viewname"). Does this cause any issues with multiple functions accessing it in a distributed environment?def get_data_sql(spark_session, data_frame, data_element): data_fram...
Kudos from