In my previous post, I wrote how to create & host private build agents for Azure DevOps running in Azure Container Instances.
One of the reasons for doing so is to eliminate creating build agent VM’s and performant pipelines for my side projects. But, of course, the build agents also need to be as cheap as possible. Azure Container Instances have per-second billing, which is excellent for build agent containers.

The thing is, I don’t want to turn them on or off manually.
It would be great to have an automated process turn on the agents when necessary and turn them off when done.

I started looking for a trigger or webhook that I can invoke whenever a build or release is put on the queue, but I couldn’t find such a feature in Azure DevOps. However, I did find some functionality in the Azure DevOps REST API which can be helpful in my scenario.

Azure DevOps REST API

When going through the documentation, you will probably get confused about the terminology quite fast, or at least I did. At first, I was going through the Pipelines docs and made some queries, but from this, I didn’t get the responses I expected.

Authorization

The Azure DevOps REST API works with Basic Auth. When invoking an endpoint, you have to specify an empty username and a Personal Access Token as a password to get access.
When testing in Postman, this will look similar to the following screenshot.

Variables in Postman

In the below URLs, I’m using variables in Postman. {{baseUrl}} is https://dev.azure.com/[yourOrganization]/[YourProject]/_apis.
{{instance}} is dev.azure.com/[yourOrganization]
{{teamproject}} is [YourProject]
In Postman, it also makes sense to store your PAT in a variable and specify this variable on the Authorization tab, as I did.

Pipelines

Invoking GET on the Pipelines endpoint will get you some details about the several pipelines you have in a project ({{baseUrl}}/pipelines?api-version=6.1-preview.1). You can use these details (id) to query the Runs history of a specific pipeline ({{baseUrl}}/pipelines/25/runs?api-version=6.1-preview.1).

I could get to make this work by iterating through the run history of all pipelines, but that looks relatively inefficient to me.

Builds

The Builds endpoint exposes functionality specifically for Builds, like listing all (pending) builds ({{baseUrl}}/build/builds?api-version=6.0).

To get all pending builds a status filter can be applied to the query ({{baseUrl}}/build/builds?statusFilter=notStarted&api-version=6.0) and these results are quite useful.

{
    "count": 1,
    "value": [
        {
            // details of the running build(s)
        }
    ]
}

The count states the number of builds that have not yet started (my status filter) which I can use to turn on the build agents.

Release

Of course, there are also release pipelines in Azure DevOps.
Releases have a different endpoint and to make things interesting, they are running on a separate subdomain, prefixed with vsrm. Retrieving all pipelines is rather easy as you only need to perform a GET-request to the specified endpoint: https://vsrm.{{instance}}/{{teamproject}}/_apis/release/releases?api-version=6.0.
A valid request to this endpoint will return a response similar to the Builds schema.

{
    "count": 3,
    "value": [
        {
            // details of the running build(s)
        }
    ]
}            

The documentation states it is also possible to filter the pipelines based on specific filters. However, it appears most (if not all) filters don’t work correctly. I have opened a GitHub issue for this, but seeing this specific repository doesn’t have a lot of activity, I doubt it’ll get fixed any time soon.
If the filters worked, you would have to call the endpoint similar to this: https://vsrm.{{instance}}/{{teamproject}}/_apis/release/releases?api-version=6.0&$expand=environments&environmentStatusFilter=4. The environmentStatusFilter is a property belonging to environments. I might have messed up the query, but having tried multiple alternatives, I’m pretty sure the API does not work. If not, please let me know!

So, as of this time, I don’t have a proper query to get the pending releases in my Azure DevOps environment.

Azure Functions as the ACI orchestrator

The Build endpoint returns a helpful response, and let’s assume the Pipelines endpoint also works as expected.
With these responses, I can turn on the ACI instance when a Build or Release is pending.

To make this possible, I’ve created a small Azure Functions project to handle these actions for me and can be found on GitHub: https://github.com/Jandev/aci-build-agent.

In the PipelineRunner.cs. I’m querying Azure DevOps for the pending Builds and Releases with the same endpoints as I mentioned above.

var request = CreateAuthenticatedRequest(
    $"{settings.AzureDevOpsProjectUrl}/_apis/build/builds?statusFilter=notStarted&api-version=6.0");

var response = await this.httpClient.SendAsync(request);

var content = await response.Content.ReadAsStringAsync();
var builds = JsonSerializer.Deserialize<Builds>(content);

// With the following to create a request with Basic auth
private HttpRequestMessage CreateAuthenticatedRequest(string url)
{
    var credentials = Convert.ToBase64String(
        System.Text.ASCIIEncoding.ASCII.GetBytes(
            string.Format("{0}:{1}", "", settings.AzureDevOpsPAT)));

    var request = new HttpRequestMessage
    {
        RequestUri = new Uri(url),
        Method = HttpMethod.Get,
        Headers = { Authorization = new AuthenticationHeaderValue("Basic", credentials) }
    };
    return request;
}

The AgentController.cs contains all logic for turning the ACI instance on and off.
As it happens, this code required an IAzure object, and it’s not possible to get this working locally out of the box when using the Azure libraries. There is an issue on GitHub with a workaround, but it’s a shame this is necessary.
Once you have the proper context, it’s rather easy to turn the container instances on and off.

// Starting the ACI.
foreach (var containerGroup in await azureContext.ContainerGroups.ListByResourceGroupAsync(settings.AgentResourceGroup))
{
    await containerGroup.Manager.ContainerGroups.StartAsync(settings.AgentResourceGroup, containerGroup.Name);
}

// Stopping the ACI
foreach (var containerGroup in await azureContext.ContainerGroups.ListByResourceGroupAsync(settings.AgentResourceGroup))
{
    await containerGroup.StopAsync();
}

The Scale-function handles the complete workflow. As you can see, this is a timer-triggered function, which is very inefficient. But like I mentioned, I couldn’t find a way to trigger an event or webhook from within Azure DevOps whenever a build or release starts. So now I’m querying the REST API every 5 minutes to see if something is pending. If, after 50 minutes, there hasn’t been any activity, the ACI instance will get turned off to save me some resources & money.

Is this recommended?

In short: NO!

When I started with this, I hoped there would be some trigger to invoke an HTTP endpoint. It would make the overall solution much more efficient.
Another thing I don’t like is not being able to scale ACI as I had expected. For example, you can’t add/remove containers to an ACI instance ad-hoc. To do so, you need to redeploy a complete ARM template and make sure the configured CPU & Memory doesn’t change.

Now I have to say I will keep using this setup. It’s a good enough solution for my side-projects and maybe even professional projects.
This setup can use optimizations, like creating multiple ACI instances, each containing 1 or 2 agent containers. That way, the containers can use a bit more resources, you can ‘scale’ the agents more granular, and make sure you pay the least amount of money for the agents.

Still, I wish there was some container service in Azure that supports autoscaling, all the way back to 0, and pricing per second/minute.
You can also deploy the above solution to a Kubernetes cluster, but to my knowledge, you’ll have to pay for at least 1 VM when using that service.