topic Re: Scope creation in Databricks or Confluent? in Data Engineering

Scope creation in Databricks or Confluent?

julie — Thu, 05 Jan 2023 18:51:02 GMT

Hello I am a newbie in this field and trying to access confluent kafka stream in Databricks Azure based on a beginner's video by Databricks. I have a free trial of Databricks cluster right now. When I run the below notebook, it errors out on line 5 on scope.

My question is, should I create the scope in confluent or in Databricks. Has anybody seen this before?

youtube video

Thanks so much!!

Julie

Re: Scope creation in Databricks or Confluent?

Hubert-Dudek — Thu, 05 Jan 2023 18:53:47 GMT

For testing, create without secret scope. It will be unsafe, but you can post secrets as strings in the notebook for testing. Here is the code which I used for loading data from confluent:

inputDF = (spark
.readStream
.format("kafka")
.option("kafka.bootstrap.servers", host)
.option("kafka.ssl.endpoint.identification.algorithm", "https")
.option("kafka.sasl.mechanism", "PLAIN")
.option("kafka.security.protocol", "SASL_SSL")
.option("kafka.sasl.jaas.config", "kafkashaded.org.apache.kafka.common.security.plain.PlainLoginModule required username='{}' password='{}';".format(userid, password))
.option("subscribe", topic)
.option("kafka.client.id", "Databricks")
.option("kafka.group.id", "new_group2")
.option("spark.streaming.kafka.maxRatePerPartition", "5")
.option("startingOffsets", "earliest")
.option("kafka.session.timeout.ms", "10000")
 .option("minPartitions", sc.DefaultParallelism)
.load() )

Re: Scope creation in Databricks or Confluent?

julie — Fri, 06 Jan 2023 15:49:38 GMT

Hey Hubert,

Firstly thanks for your response. Sure, I'll try this instead.

For line 11, kafka.group.id , that'll have to be from confluent cloud right?

Thanks,

Julie

Re: Scope creation in Databricks or Confluent?

Hubert-Dudek — Fri, 06 Jan 2023 15:53:10 GMT

It is the name defined by you to control offset. I think that once you put in databricks code it will create it in kafka/confluent automatically.

Re: Scope creation in Databricks or Confluent?

julie — Fri, 06 Jan 2023 17:19:21 GMT

Thanks Hubert, it runs fine and it looks like it is creating an input dataframe. How do I see if it consuming the correct topic? any thoughts?

Re: Scope creation in Databricks or Confluent?

julie — Fri, 06 Jan 2023 17:24:53 GMT

or does this even mean its able to access my topic?

Thanks for your time Hubert. I really appreciate it.

Julie