Databricks Community

Mado · 01-18-2024

I have a delta table in Databricks named "prod.silver.control_table". It has a few columns including "table_name" with string data type and "transform_options" with the below structure: |-- transform_options: map (nullable = true) | |-- key: str...

Mado · 01-03-2024

I have a task to revise CSV ingestion in Azure Databricks. The current implementation uses the below settings: source_query = ( spark.readStream.format("cloudFiles") .option("cloudFiles.format", "csv") .schema(defined_schema) .option(...

Mado · 07-19-2023

Hi,When caching a DataFrame, I always use "df.cache().count()".However, in this reference, it is suggested to save the cached DataFrame into a new variable:When you cache a DataFrame create a new variable for it cachedDF = df.cache(). This will allow...

Mado · 07-18-2023

Hi,I have a DataFrame and different transformations are applied on the DataFrame. I want to display DataFrame after several transformations to check the results. However, according to the Reference, every time I try to display results, it runs the e...

Mado · 06-21-2023

I get the following error when getting a list of files stored in an Azure Storage account using "dbutils.fs.ls" command in Databrciks.Failure to initialize configuration for storage account AAAAA.dfs.core.windows.net: Invalid configuration value dete...

Mado · 01-20-2024

@Retired_mod Could you give me more hints on how to use UPDATE command for this specific case?

Mado · 01-04-2024

Also, I am looking for a solution that works with both correct files and malformed files using PySpark.

Mado · 01-04-2024

@Retired_mod Thanks for your help. I receive different CSV files with different number of columns, and column names for different target tables. I mean, it is not only a CSV with specific column names and data types. I am looking for a solution that ...

Mado · 07-19-2023

@-werners- Thanks for your help. If I assign cached DataFrame into a new variable, which one should I use in the next cells for next transformations. For example, cached_df = df.cache()cached_df.count()Then, which of the following options is correct ...

Mado · 07-19-2023

@Vinay_M_R Thanks for your help.I am afraid I didn't understand the reason why it is necessary.This is because caching the DataFrame can cause it to lose any data skipping that can come from additional filters added on top of the cached DataFrame, No...

Databricks Community

User Stats

User Activity

How to update a value in a column in a delta table with Map of Struct datatype?

Read CSV files in Azure Databricks notebook, how to read data when columns in CSV files are in the w

What are the best practices for spark DataFrame caching?

What is the best approach to display DataFrame without re-executing the logic each time we display?

Error "Invalid configuration value detected for fs.azure.account.key" when listing files stored in an Azure Storage account using "dbutils.fs.ls"

Re: How to update a value in a column in a delta table with Map of Struct datatype?

Re: Read CSV files in Azure Databricks notebook, how to read data when columns in CSV files are in t

Re: Read CSV files in Azure Databricks notebook, how to read data when columns in CSV files are in t

Re: What are the best practices for spark DataFrame caching?

Re: What is the best approach to display DataFrame without re-executing the logic each time we displ