HI @TheDataMaverick ,
The most efficient approach for your REST API ingestion pipeline on Databricks is to use an external service like Azure Functions (or AWS Lambda) to handle API calls, then land raw JSON/Parquet in ADLS/S3 for Auto Loader ingestion into bronze.
After external API landing and bronze ingestion via Auto Loader, the next step for silver/gold can be either SDP (Spark Declarative Pipelines) or regular Spark jobs/workflows, depending on requirements. SDP suits declarative, incremental medallion transforms with auto-orchestration and serverless efficiency, while jobs fit custom/complex logic.