Data Engineering - Databricks

Data Engineering

Forum Posts

Sorted by:

Start a conversation

TypeError : withcolumn() takes 3 positional arguments but 4 were given.

Hi All,Can some one please help me with the error.This is my small python code.binmaster = binmasterdf.withColumnRenamed("tag_number","BinKey")\.withColumn ("Category", when (length("Bin")==4,'SMALL LOT'),(length("Bin")==7,'RACKING'))TypeError : with...

Kafka unable to read client.keystore.jks.

Below is the error we have received when trying to read the stream Caused by: kafkashaded.org.apache.kafka.common.KafkaException: Failed to load SSL keystore /dbfs/FileStore/Certs/client.keystore.jksCaused by: java.nio.file.NoSuchFileException: /dbfs...

A way to run OPTIMIZE, VACUUM, ANALYZE on all schemas and tables

Hello everyone,I am not sure if this was asked, but I am trying to find a way to create one python (or Scala) script that would basically take a list of all the schemas and then run optimize, vacuum, and then analyze tables on them.I see a lot of web...

Resolved! Spark readStream kafka.ssl.keystore.location abfss path

Similar to https://community.databricks.com/t5/data-engineering/kafka-unable-to-read-client-keystore-jks/td-p/23301 - the documentation (https://learn.microsoft.com/en-gb/azure/databricks/structured-streaming/kafka#use-ssl-to-connect-azure-databricks...

Problems with extending DataFrame-object with custom method in DBR 14.0

I noticed that extending the DataFrame object with a custom method does not work in DBR 14.0 This works fine in when using DBR 13.3. The below code throws a "TypeError: 'Column' object is not callable". from pyspark.sql.functions import col,lit from...

Top Kudoed Authors

User

Count

1579

714

342

284

246