Documentation issue: Invalid JSON example in Lakeflow Connect multi-destination pipeline

prafullkatiyar
New Contributor

Tuesday

Hi Community,

I am trying to implement multi-destination pipelines and came across this code in Databricks documentation: Create multi-destination pipelines | Databricks on AWS

Key 'table' is repeated in all examples - I believe JSON objects should not have duplicate keys; the second table will overwrite the first one.

pipeline_spec = """
{
"name": "<pipeline-name>",
"ingestion_definition": {
"connection_name": "<connection-name>",
"objects": [
{
"table": {
"source_catalog": "<project-1-id>",
"source_schema": "<property-1-name>",
"destination_catalog": "<target-catalog-1>",
"destination_schema": "<target-schema-1>",
},
"table": {
"source_catalog": "<project-2-id>",
"source_schema": "<property-2-name>",
"destination_catalog": "<target-catalog-2>",
"destination_schema": "<target-schema-2>",
}
}
]
}
}
"""

YAML version of this looks correct, could someone from Databricks team confirm and update the docs.

Correct code:

"objects": [
{
"table": {
"source_catalog": "<project-1-id>",
"source_schema": "<property-1-name>",
"destination_catalog": "<target-catalog-1>",
"destination_schema": "<target-schema-1>"
}
},
{
"table": {
"source_catalog": "<project-2-id>",
"source_schema": "<property-2-name>",
"destination_catalog": "<target-catalog-2>",
"destination_schema": "<target-schema-2>"
}
}
]

0 Kudos

aliyasingh
New Contributor III

Tuesday

You are absolutely correct! Great catch.

The JSON example provided in the documentation is structurally invalid. Because standard JSON parsers do not support duplicate keys within the same dictionary object, the second "table" key will silently overwrite the first one. If someone were to deploy the pipeline using that exact snippet, Databricks would only ever register the final destination schema.

Your corrected JSON snippet is the exact right way to format this configuration: passing a list of distinct objects, each containing its own single "table" dictionary, will ensure all destinations are picked up correctly.

I highly recommend submitting this via the "Send us feedback" link at the bottom of the documentation page so the Databricks tech writers can get it updated. Thanks for sharing the fix with the community!

0 Kudos