Databricks

hare · 02-19-2023

Hi Team, Can you please suggest to me how to implement the late arriving dimension or early arriving fact with examples or any sample script for reference? I have to implement the same using pyspark.Thanks.

hare · 05-19-2022

Hi All, We are getting JSON files in Azure blob container and its "Blob Type" is "Append Blob".We are getting an error "AnalysisException: Unable to infer schema for JSON. It must be specified manually.", when we try to read using below mentioned scr...

hare · 05-15-2022

We are processing the josn file from the storage location on every day and it will get archived once the records are appended into the respective tables.source_location_path: "..../mon=05/day=01/fld1" , "..../mon=05/day=01/fld2" ..... "..../mon=05/d...

hare · 02-25-2022

Azure DBR - Have to load list of json files into data frame and then from DF to data bricks table but the column has special character and getting below error.Both column(key) and value (as json record) has special characters in the json file. # Can...

hare · 02-26-2023

Hi @Vidula Khanna - I will be helpful i get any sample implementation logic or script/Pseudo code like that.

hare · 02-21-2023

Thanks for sharing, let me check and update.

hare · 02-28-2022

Thanks for the solution @Hubert Dudek . In our scenario we couldn't able to define the schema manually since schema may vary file to file. So i am thinking how to do it dynamically based on the Json schema. Please do let me know if you have any taug...

hare · 02-28-2022

Thanks for your solution werners. Yes,we can able to rename the column as you suggested. But the problem here is the value of the column is also a json (Array of struct). For an instance;"hydra:mapping" - is a column which has the value called "hydra...

Databricks

User Stats

User Activity

Implementation of Late arriving dimension in databricks

"Databricks" - "PySpark" - Read "JSON" file - Azure Blob container - "APPEND BLOB"

Failed to merge incompatible data types

Azure DBR - Have to load list of json files but the column has special character.(ex: {"hydra:xxxx": {"hydra:value":"yyyy", "hydra:value1":"zzzzz"}

Re: Implementation of Late arriving dimension in databricks

Re: Implementation of Late arriving dimension in databricks

Re: Azure DBR - Have to load list of json files but the column has special character.(ex: {"hydra:xxxx": {"hydra:value":"yyyy", "hydra:value1":"zzzzz"}

Re: Azure DBR - Have to load list of json files but the column has special character.(ex: {"hydra:xxxx": {"hydra:value":"yyyy", "hydra:value1":"zzzzz"}