Workflow Pipeline in Azure Databrick is throwing error for EventHubsSourceProvider could not be instantiated

PawanShukla — Thu, 26 May 2022 09:18:24 GMT

I am using the sample code which is available in getting start tutorial. And it is simple read the json file and move in another table. But it is throwing error related to EventHubsSourceProvider

Re: Workflow Pipeline in Azure Databrick is throwing error for EventHubsSourceProvider could not be instantiated

PawanShukla — Mon, 06 Jun 2022 18:41:37 GMT

Cluster is creating on the fly...not in interactive cluster

and code is :

import dlt

from pyspark.sql.functions import *

from pyspark.sql.types import *

json_path = "/databricks-datasets/wikipedia-datasets/data-001/clickstream/raw-uncompressed-json/2015_2_clickstream.json"

@dlt.table(

comment="The raw wikipedia clickstream dataset, ingested from /databricks-datasets."

)

def clickstream_raw():

return (spark.read.format("json").load(json_path))

@dlt.table(

comment="Wikipedia clickstream data cleaned and prepared for analysis."

)

@dlt.expect("valid_current_page_title", "current_page_title IS NOT NULL")

@dlt.expect_or_fail("valid_count", "click_count > 0")

def clickstream_prepared():

return (

dlt.read("clickstream_raw")

.withColumn("click_count", expr("CAST(n AS INT)"))

.withColumnRenamed("curr_title", "current_page_title")

.withColumnRenamed("prev_title", "previous_page_title")

.select("current_page_title", "click_count", "previous_page_title")

)

@dlt.table(

comment="A table containing the top pages linking to the Apache Spark page."

)

def top_spark_referrers():

return (

dlt.read("clickstream_prepared")

.filter(expr("current_page_title == 'Apache_Spark'"))

.withColumnRenamed("previous_page_title", "referrer")

.sort(desc("click_count"))

.select("referrer", "click_count")

.limit(10)

)

topic Workflow Pipeline in Azure Databrick is throwing error for EventHubsSourceProvider could not be instantiated in Data Engineering

Workflow Pipeline in Azure Databrick is throwing error for EventHubsSourceProvider could not be instantiated

Re: Workflow Pipeline in Azure Databrick is throwing error for EventHubsSourceProvider could not be instantiated