A couple of months ago, we migrated from using the Completions endpoints to the Responses endpoints for the language models we use, hosted in Microsoft Foundry. This worked quite well after some refactoring. That is, until a couple of weeks ago…
During my vacation, ‘something’ changed. Suddenly, some of the requests we made to the language models (gpt-4.1 and gpt-4.1-mini) took about 73 seconds to complete. Previously, the responses were returned well within 4 seconds. Because of this, our agents and chat experience no longer worked as expected.

After doing quite a bit of research and testing, I finally found the issue. We had to change our code from non-streaming responses to streaming responses. Responses are now served well within the expected 4-second time slot.

Doing the analysis

I’m not sure exactly when the mentioned regression appeared, as I was on vacation at the time. But it’s safe to say “some time in July or August 2026”.

When starting the analysis, I first tried to discover where the issue was. Because of our extensive logging, it was easy to discover that a couple of agents took ~73 seconds to complete their actions. We have a nice agentic workflow going on, so responses often took more than 200 seconds to complete due to multiple agents collaborating to get to an answer. Needless to say, we received feedback stating “chat doesn’t work!”.

Going over the numbers

Our environment is deployed in Sweden Central, but the models are deployed using the Global Standard SKU. This is a shared SKU, so our issues could be due to a noisy neighbor, throttling for a specific model, or just throttling in general, or something else entirely.
As the performance hit was (and still is) occurring continuously and with consistent timings, the noisy neighbor didn’t seem like the issue. Throttling could have been the cause; however, when inspecting the responses we received, it was never a 429. Normally, when we were throttled, I could see many 429 responses coming from the Microsoft Foundry endpoints. At one point, someone suggested upgrading to a PTU SKU, which should have consistent performance. I was a bit hesitant to do so, as the provisioned throughput SKU has quite a price hike compared to the pay-as-you-go offering we are using. Also, none of the analysis indicated that we were being throttled or that there were performance issues on the cluster(s) we were using.
I know this due to a small reproduction tool/script I used for the analysis.

In my script, I stripped away all (well, most) of the abstraction layers, like the Microsoft Agent Framework, and only relied on the OpenAI package for invoking the endpoints and my Azure CLI login.

pip install "openai>=2.0" azure-identity
az login

Once this is done, you can make a call via the OpenAI client and analyze the responses. In my initial steps, I was mostly interested in the returned headers and whether there was a clue as to why it was so slow.
The relevant piece of code is pasted below.

HEADERS_WORTH_KEEPING = {
    # The two headers that are important for the findings
    "openai-processing-ms",
    "azureai-fe-is-streaming",
    # Which model actually served the request.
    "x-ms-served-model",
    # Quota capacity, to check if we're throttled
    "x-ratelimit-remaining-requests",
    "x-ratelimit-limit-requests",
    "x-ratelimit-remaining-tokens",
    "x-ratelimit-limit-tokens",
    # Where the request was served, and on what tier.
    "x-ms-region",
    "azureml-served-by-cluster",
    "azureai-fe-requested-service-tier",
    # Identifiers to quote in a support case.
    "apim-request-id",
    "x-request-id",
}

raw_http_response = client.chat.completions.with_raw_response.create(...)
answer = extract_answer_text_from_complete_response(raw_http_response.parse())
headers = dict(raw_http_response.headers)

# Some other code

selected_headers={
    key: value
    for key, value in headers.items()
    if key.lower() in HEADERS_WORTH_KEEPING
},

By looking at the selected_headers I could do some good analysis.
The openai-processing-ms header states how long it spent on the request. From what I understand, this is the time spent on the server/service itself, not including the overhead of the client, network and SDK.

With these headers available, it was also easy to see whether we had hit a quota or were being throttled. There were no 429s, and we were well within the quota limits (< 0.5%).

I also considered that the service might be slow because of unhealthy clusters, ongoing updates, high load, or something similar. With these headers, you can see which backends are being used. In my case, it’s two clusters in Sweden Central.

{
    "region": {
        "value": "Sweden Central",
        "source": "response header `x-ms-region`"
    },
    "clusters_observed": {
        "value": ["hyena-swedencentral-01", "hyena-swedencentral-02"],
        "source": "response header `azureml-served-by-cluster`"
    },
    "service_tier": {
        "raw_header_value": "paygo",
        "source": "response header `azureai-fe-requested-service-tier`"
    }
}

Because we had consistent slow responses for multiple weeks, it’s safe to say we can rule out transient issues that might have been occurring at the backends.

Issues in our code can also be ruled out as the performance regression was happening overnight without us changing something to the code. Also, I can reproduce the issue without any abstraction layer, aside from the openai Python package.

I also ruled out the possibility that it was an issue with one or more models. The behavior could consistently be reproduced with gpt-4.1-mini, gpt-4.1-nano, gpt-4.1, and gpt-5.1-mini. I stopped validating other models, but I’m pretty sure the same behavior will apply to those as well.

Some agents did work properly

What struck me as odd was that some agents did have normal response times, both in our .NET and Python projects.
As I couldn’t find any meaningful issues with the Microsoft Foundry backend responses, I started looking at the differences between the agents in the codebase.

While comparing them, there was one thing that stood out.

This is the code used by the agents that were still performant.

options = _run_options(agent, tool_choice)
options["response_format"] = response_type

try:
    async for chunk in agent.run(
        question,
        stream=True,
        session=session,
        tools=tools,
        options=options,
    ):
        if chunk.text:
            full_text += chunk.text

The agents having the 73 seconds delay are invoking the language model like this.

options = _run_options(agent, tool_choice)
options["response_format"] = response_type

try:
    response = await agent.run(
        question,
        stream=False,
        session=session,
        tools=tools,
        options=options,
    )

Can you spot the difference?
That’s correct: it’s the stream=True versus stream=False parameter. It’s quite interesting that this one small parameter might have something to do with it.

Measuring if streaming is broken for responses API

Because of this finding, I continued expanding the reproduction script.

Microsoft’s guidance explicitly states that streaming should not change the total processing time:

Setting stream: true in a request makes the service return tokens as soon as they’re available, instead of waiting for the full sequence of tokens to be generated. It doesn’t change the time to get all the tokens, but it reduces the time for first response.

— Performance and latency — Improve performance

The same page defines the metrics precisely:

“Time to Last Byte (AzureOpenAITTLTInMS) — Total time from prompt submission to the last token, measured by the API gateway… Time to Response (AzureOpenAITimeToResponse) — Time from prompt submission to the first response chunk.”

— Performance and latency — Understanding Azure OpenAI latency

In my reproduction script, the stopwatch for my streaming requests is stopped after fully draining the stream and serializing the response. Therefore, it’s doing its best to provide a fair comparison with non-streaming serialized responses.

With this in mind, I compared the Completions and Responses APIs.

(averages across three runs; Chat Completions did not return an openai-processing-ms header on this account, so client wall clock is shown, which is an upper bound on server time)

This is quite a difference, and it also doesn’t match what the docs state.

I’ve run numerous tests across multiple days, and the results are consistent. Every time the Responses API is used with the non-streaming flag, it takes about 73 seconds to return.

  sequenceDiagram
    autonumber
    participant C as Client<br/>(openai SDK, max_retries=0)
    participant A as Microsoft Foundry

    rect rgb(255, 235, 235)
        note over C,A: SLOW - the anomaly
        C->>A: POST /openai/v1/responses (stream=false)
        note right of A: azureai-fe-is-streaming: False<br/>openai-processing-ms: ~73,000
        A-->>C: 200 OK after ~73 s
    end

    rect rgb(235, 250, 235)
        note over C,A: FAST - same request, one flag changed
        C->>A: POST /openai/v1/responses (stream=true)
        note right of A: azureai-fe-is-streaming: True<br/>openai-processing-ms: ~300
        A-->>C: 200 OK, fully drained, under 1 s
    end

    rect rgb(235, 250, 235)
        note over C,A: CONTROL - also buffered, yet fast
        C->>A: POST /openai/v1/chat/completions (stream=false)
        A-->>C: 200 OK in ~0.6 s
    end

    rect rgb(235, 250, 235)
        note over C,A: CONTROL - baseline
        C->>A: POST /openai/v1/chat/completions (stream=true)
        A-->>C: 200 OK in ~0.6 s
    end

You can look at the script I have used for reproduction & analysis in this Gist: https://gist.github.com/Jandev/1d4a061eb01939149d2f0e126305a9b1

Changes in my codebase

Instead of relying on the capabilities of MAF or the OpenAI SDK to parse a response into a Pydantic model, you now have to do this yourself. It’s not hard, but it does require a bit of code to maintain. For reference, the code excerpt below may provide some inspiration.

async def do_it( 
    # ...
    response_type: type[TResponse])

options = _run_options(agent, tool_choice)
options["response_format"] = response_type

full_text = ""
async for chunk in agent.run(
    question,
    stream=True,
    session=session,
    tools=tools,
    options=options,
):
    # First append all the text into a big string
    if chunk.text:
        full_text += chunk.text

# Do some cleaning of the string
cleaned = full_text.strip()
if cleaned.startswith("```"):
    first_newline = cleaned.index("\n") if "\n" in cleaned else 3
    cleaned = cleaned[first_newline + 1 :]
if cleaned.endswith("```"):
    cleaned = cleaned[:-3]
cleaned = cleaned.strip()

# Now serialize it into the model you want
parsed = json.loads(cleaned)
return response_type.model_validate(parsed)

It’s quite annoying this workaround is required at this point. But at least it’s solvable.