Databricks Community

Mado

Hi Team,I'm evaluating Databricks Lakeflow Connect for SQL Server CDC ingestion and have run into a gateway provisioning issue.EnvironmentRegion: Australia EastSource: Azure SQL DatabaseCDC enabled successfully on the database and source tableSQL Ser...

Mado · 01-18-2024

I have a delta table in Databricks named "prod.silver.control_table". It has a few columns including "table_name" with string data type and "transform_options" with the below structure: |-- transform_options: map (nullable = true) | |-- key: str...

Mado · 01-03-2024

I have a task to revise CSV ingestion in Azure Databricks. The current implementation uses the below settings: source_query = ( spark.readStream.format("cloudFiles") .option("cloudFiles.format", "csv") .schema(defined_schema) .option(...

Mado · 07-19-2023

Hi,When caching a DataFrame, I always use "df.cache().count()".However, in this reference, it is suggested to save the cached DataFrame into a new variable:When you cache a DataFrame create a new variable for it cachedDF = df.cache(). This will allow...

Mado · 07-18-2023

Hi,I have a DataFrame and different transformations are applied on the DataFrame. I want to display DataFrame after several transformations to check the results. However, according to the Reference, every time I try to display results, it runs the e...

Mado

Thanks @ThomazNeto for your help. I created a policy based on your instructions.Could you please advise how I can configure the Gateway Pipeline to use this policy?If there are any additional settings or steps required to ensure the Gateway Pipeline...

Mado · 01-20-2024

@Retired_mod Could you give me more hints on how to use UPDATE command for this specific case?

Mado · 01-04-2024

Also, I am looking for a solution that works with both correct files and malformed files using PySpark.

Mado · 01-04-2024

@Retired_mod Thanks for your help. I receive different CSV files with different number of columns, and column names for different target tables. I mean, it is not only a CSV with specific column names and data types. I am looking for a solution that ...

Mado · 07-19-2023

@-werners- Thanks for your help. If I assign cached DataFrame into a new variable, which one should I use in the next cells for next transformations. For example, cached_df = df.cache()cached_df.count()Then, which of the following options is correct ...

Databricks Community

User Stats

User Activity

How can I configure Lakeflow Connect SQL Server CDC Gateway to use a desired VM type?

How to update a value in a column in a delta table with Map of Struct datatype?

Read CSV files in Azure Databricks notebook, how to read data when columns in CSV files are in the w

What are the best practices for spark DataFrame caching?

What is the best approach to display DataFrame without re-executing the logic each time we display?

Re: How can I configure Lakeflow Connect SQL Server CDC Gateway to use a desired VM type?

Re: How to update a value in a column in a delta table with Map of Struct datatype?

Re: Read CSV files in Azure Databricks notebook, how to read data when columns in CSV files are in t

Re: Read CSV files in Azure Databricks notebook, how to read data when columns in CSV files are in t

Re: What are the best practices for spark DataFrame caching?