Databricks

UmaMahesh1 · 04-11-2023

I have a streaming notebook which fetches messages from confluent Kafka topic and loads them into adls. It is a streaming notebook with the trigger as continuous processing. Before loading the message (which is in Avro format), I'm flattening out the...

UmaMahesh1 · 12-19-2022

Hello community, I want to fetch the list of all the tabular models (if possible details about those models too) which are there in a SQL Analysis server using databricks. Can anyone help me out ?Use case: I want to process clear a large number of mo...

UmaMahesh1 · 12-03-2022

Did anyone recently give Databricks associate data engineer certification i.e. after the latest changes ? How was the experience. Also how was the difficulty of the questions and are the resources available in the training path sufficient ?Need exper...

UmaMahesh1 · 12-01-2022

I have a string column which is a concatenation of elements with a hyphen as follows. Let 3 values from that column looks like below, Row 1 - A-B-C-D-E-FRow 2 - A-B-G-C-D-E-FRow 3 - A-B-G-D-E-FI want to compare 2 consecutive rows and create a column ...

UmaMahesh1 · 11-29-2022

I'm a bit new to spark structured streaming stuff so do ask all the relevant questions if I missed any.I have a notebook which consumes the events from a kafka topic and writes those records into adls. The topic is json serialized so I'm just writing...

UmaMahesh1

Does the table have specific permissions to that ?If yes, please check the permission inheritance in the catalog permissions. if it is correctly configured or not.

UmaMahesh1 · 08-01-2023

Hi @Mahajan What exactly do you mean by disabling the cell scroll ? If and all there is an option as such, then it basically means you can't scroll the cell at all and the cell view is fixed. This makes the cell redundant as at any given point of ti...

UmaMahesh1 · 08-01-2023

Hi @Chalki Is the output of the notebook being passed into any other ADF activity ? If yes, check that output.Also, can you give further details like the configuration you are using in ADB and in ADF for compute ?

UmaMahesh1 · 04-13-2023

Are you using the wildcard for the Update and insert ?#DAIS2023

UmaMahesh1 · 04-13-2023

# Set the chunk size in bytes (1MB = 1048576 bytes) chunk_size = 1048576 # Fix the number of partitions needed based on the chunk size num_partitions = (df.rdd.map(lambda x: len(str(x))).sum() // chunk_size) + 1 # Repartition the DataFrame using ...

Databricks

User Stats

User Activity

Checkpoint issue when loading data from confluent kafka

How to get list of all the tabular models in a Analysis server using databricks ?

Did anyone recently give Databricks associate data engineer certification i.e. after the latest changes ? How was the experience. Also how was the dif...

Pyspark dataframe column comparison

Spark Structured Streaming : Data write is too slow into adls.

Re: Ingest __databricks_internal catalog - PERMISSION DENIED

Re: Want to disable cell scrollers.

Re: InternalError, error message: The output of the notebook is too large.

Re: SQL Merge Statement not working

Re: Split dataFrame into 1MB chunks and create a single json array with each row in chunk being an array element