I’m working on something where I need to deploy multiple versions of my software and validate whether there’s an improvement or regression. The solution heavily relies on deployed language models, so that’s something we want to evaluate first.
The solution I’m working on looks fairly similar to the setup in my Trial & Error GitHub repository, so I have a .NET frontend service and a Python backend service, both invoking language models deployed in Microsoft Foundry.

The baseline

I’ve deployed this using the following services:

• Azure API Management
  This is the public entry point for my services
• Azure Container Apps Environment
  Hosting two Container Apps for both .NET and Python
• Microsoft Foundry + some models For my agents & MAF to work with

Of course, there are some other resources deployed too, but those aren’t important for now.

The way this works is:

1. A client makes an HTTP request
2. APIM processes the request and routes it to the default backend, the .NET Container App
3. The .NET Container App takes the request and processes it
4. The request gets forwarded to the Python Container App, which processes it
5. The processed request is passed along to a language model in Microsoft Foundry
6. The response gets processed and sent back to the client through the .NET Container App and APIM

  sequenceDiagram
    participant Client
    participant APIM as Azure API Management
    participant NET as .NET Container App
    participant Python as Python Container App
    participant Foundry as Microsoft Foundry (LLM)

    Client->>APIM: HTTP Request
    APIM->>NET: Route to default backend
    NET->>Python: Forward request
    Python->>Foundry: Send to language model
    Foundry-->>Python: Model response
    Python-->>NET: Processed response
    NET-->>APIM: Return response
    APIM-->>Client: HTTP Response

I think you’ll find this to be a common design. Obviously, the .NET and Python services also provide some additional value on top of forwarding requests, but that’s not relevant to this post.