Model Serving - Shadow Deployment - Azure

[ryojikn]

Hey,

I'm composing an architecture within the usage of Model Serving Endpoints and one of the needs that we're aiming to resolve is Shadow Deployment.

Currently, it seems that the traffic configurations available in model serving do not allow this type of behavior, mixing a mirroring requests effect with "fire and forget" responses from the shadow application.

Do you have this as a feature backlog? Or do you have any already implemented architecture composed within Azure pieces that I could use for that?

Thanks in advance

[Kaniz_Fatma]

Hi @ryojikn, Shadow deployment is a valuable strategy for testing and validating changes in a production environment without impacting live traffic.

Let’s explore some approaches and resources related to shadow deployments:

1. Shadow Deployments:

   • What is Shadow Deployment?: In a shadow deployment, you deploy a new version of your application alongside the existing live version. The key idea is to route a portion of the incoming traffic to the new version (the “shadow” version) while still serving most of the requests from the live version. This allows you to observe how the new version behaves in a real-world scenario without affecting end users.
   • How It Works:
     • A load balancer is typically used to distribute traffic between the live version and the shadow version of the application.
     • When a user makes a request, the load balancer randomly routes the request to either the live versio...¹.
   • Benefits:
     • Validate changes: Shadow deployments help you test if your changes are compatible with dependent services, ensuring smooth interoperability when deployed to production.
     • Minimize risk: By gradually introducing the new version, you reduce the risk of unexpected issues affecting all users.
     • Real-world insights: You gain insights into how the new version performs under actual load and usage patterns.
   • Implementation:
     • Deploy the new version alongside the existing live version.
     • Configure your load balancer to distribute traffic between the live and shadow versions.
     • Monitor and analyze the behavior of the shadow version.
   • Azure-Specific Considerations:
     • Azure provides various services that can be leveraged for shadow deployments.
     • Consider using Azure Traffic Manager or Azure Application Gateway for traffic distribution.
     • Use Azure Monitor to collect telemetry data and analyze the performance of the shadow version.
     • Automate the deployment process using Azure DevOps or Azure Pipelines².

2. Multitenant Solutions in Azure:

   • If you’re working with multitenant solutions, Azure offers guidance on deploying and configuring resources for multiple tenants.
   • Consider whether your shadow deployment scenario involves multiple tenants, as this may impact your deployment strategy.
   • Document the onboarding workflow for new tenants, including acceptance of commercial agreements, resource provisioning, and configuration tasks.
   • Azure Marketplace can also play a role in tenant onboarding and resource management^2.

 If you have specific requirements or need further assistance, consider consulting Azure’s official resources or engaging with the Azure community. 😊🚀²¹.