My current project requires us to create multiple agents running complex algorithms with large volumes of data.
The algorithms work on large datasets and compute outcomes to be used in the next steps of our workflow. Our data scientists are creating these algorithms, and they’re most comfortable in Python, so that’s what our application is running as well, in a setup similar to the one I created in my Trial and Error repository.
If you head to this repository, you’ll notice a file called large_data_analysis.py. This file is responsible for creating agents that are capable of working with large volumes of data.

As you probably know, context windows for agents are limited. They are growing at a rapid pace, but ideally you keep the context small.
When you need to work with millions of records from a repository, you can put them in the agent’s context, but you’ll notice the limitations of this practice quite quickly. Either the agent will start hallucinating or forget details, or you’ll be prompted with an error stating that the context is too large. Both scenarios are disastrous when relying on the agent and its outcome.

Have tools query your repository

In my previous post, you could see how to create tools that can be used in agents.
Oftentimes, these tools require data to work with. If the data already exists inside the prompt, that’s great. The language model will take care of passing the correct parameters based on the descriptions.

You know what’s cool? Having agents talk to each other and letting them figure out how to get to the answer you’re looking for.
One way to do this is by using the Agent-to-Agent protocol in your application. Version 0.3.0 is the latest released version, and there’s an RC v1.0 available already.
The Microsoft Agent Framework (MAF) also has an implementation of this protocol available. The current version of the MAF packages, 1.0.0b260130 at the time of writing, isn’t compatible with the proposed changes of 1.0, but I’m pretty sure this will be supported in upcoming releases. The team is adding and changing features quite quickly. There are also newer versions of MAF available now, but I have yet to validate those.

In my current project, we’re creating a dozen agents, each doing its own little thing. What we could do is create some workflow or state machine, invoking each agent in turn, much like the good old days. However, we don’t always need to run every agent or run them in a specific order. While it is possible to add this dynamic nature to an application, we can also leverage the power of a language model for this. Based on the knowledge of what an agent can do, the language model can figure out which agents to invoke and in what order. The A2A protocol can help here.

It looks like everyone is creating agents nowadays. Most of the time with elaborate prompts to tell a language model what it should do.
Great, but we all know a language model isn’t good at doing everything. Also, I don’t want it to do everything either as it would need to be granted access to every possible resource in my environment.

To extend the capabilities of an agent (and the underlying language model), you can provide tools. With good tool documentation, agents are empowered to do more. Take, for example, counting how many times the letter r appears in Strawberry. Browsing around the internet, it looks like this is one of those important life questions everyone wants an answer to.

The problem is that people are asking this question to a model that’s good at language and guessing what the best possible output should be, not a model that’s good at counting.

What I did was create an agent that’s empowered with a few tools capable of doing simple math, just to see how this is supposed to work. I’m using the Microsoft Agent Framework. The team isn’t shying away from making breaking changes with every release, so newer (and older) versions may need a different implementation. Currently, I’m working in Python with the agent-framework-azure-ai package, version 1.0.0b260130.

There’s a new feature over here, my weekly links archive.
Every week a page will be added automatically based on content I have read and found interesting to share. I’m using a self-hosted Linkwarden instance to collect pages and links for a variety of topics. For the purpose of this weekly links archive I have created a new tag called Newsletter which I use to fetch the weekly links to share with all of you.

Of course, I don’t want to make my Linkwarden service open to the internet. To fetch the content from Linkwarden in my scheduled GitHub Action I use Tailscale as shared earlier on this blog.

Query Linkwarden API

Linkwarden provides a nice API that is documented (somewhat) on their site: https://docs.linkwarden.app/api/api-introduction
Not all features and possibilities are mentioned over here, but with some creativity you’ll be able to figure out a lot.

To use the API you first need an access token.
These tokens can be created on the Settings -> Access Tokens page.

Once you have this token, make sure to store it somewhere safe.

As you can see in the documentation there’s a search endpoint. This one is useful to validate your requests are working. The endpoint I’m most interested in is the links endpoint and using the tagId querystring parameter. To make good use of this, you do need to know which tag to query, in my case it’s 5.
Do note, this endpoint is being deprecated in favor of the search endpoint. At this point, I have not been able to figure out how to use the search-endpoint to retrieve all posts for a specific tag, but this will probably be necessary at some time in the future.

In an earlier post I wrote about using Semantic Kernel to create an Agentic AI solution, all using C#. Of course, similar flows can be created with Python. To try this, I’ve created a sample solution to update a resume so it’s more likely to pass the ATS requirements used by various companies nowadays.

My sample is heavilly inspired by Gian Paolo Santopaolo his CV-Pilot repository, which I was not able to use due to the CrewAI tooling phoning home and my DNS (PiHole) blocking those requests. Even after disabling the tracking features, the library/ies still tracked ‘something’, which caused the logic to break so I decided to create something myself using Semantic Kernel.

What did I create?

A tool/flow to update a resume so it can pass the ATS (Applicant Tracking System), based on the job description. These systems ofen check for specific wording and skills. For a human, it takes quite a bit of time to (re)write a resume to pass the ATS requirements. It’s the perfect job for a LLM to perform, as it’s built to create text based on other text input.

You can try doing this in a single prompt, but there’s a high probability this won’t perform the way you like. In my sample, I’ve created a ‘Project Manager’-, ‘Job Market Analyst’- and ‘Strategist’-agent. Each with their own specific job and goals.

When working on a project that’s being distributed you are often required to create a NOTICE file giving the necessary attribution to work you rely on. As written on the Apache site on this topic:

If the Work includes a “NOTICE” text file as part of its distribution, then any Derivative Works that You distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that do not pertain to any part of the Derivative Works, in at least one of the following places: within a NOTICE text file distributed as part of the Derivative Works; within the Source form or documentation, if provided along with the Derivative Works; or, within a display generated by the Derivative Works, if and wherever such third-party notices normally appear. The contents of the NOTICE file are for informational purposes only and do not modify the License. You may add Your own attribution notices within Derivative Works that You distribute, alongside or as an addendum to the NOTICE text from the Work, provided that such additional attribution notices cannot be construed as modifying the License.

It’s not hard or complex to create these files, but it’s a LOT of work when starting.

The current large language models, like GPT-4, GPT-4 Turbo and GPT-4o are great when you need some output generated based on data you feed in the prompt. Even the small language models, like Phi-3, are doing a great job at this. However, these models often don’t know a lot about the data within your company. Because of this, they can’t do a good job at answering questions that required data from your organization.

There is of course the M365 Copilot available, which is able to index all of the organization its data and provide answers based on it. On a high level, what this is doing, is using Retrieval-Augmented Generation (RAG). There’s a decent post about this on the IBM Research site and there’s also a good post on the AWS site on it.

By using RAG in combination with your LLM, you are able to index your own data and let the model interpret it.
A great way to get started with this, is by using the Azure Open AI Assistants feature. The MS Learn page on this topic is quite good. If you’re interested in the topic, I’d suggest to check it out: https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/assistant.

Get your data

The first thing you need to do is make sure all data is available to the assistant. At this moment, there’s a large list of supported file types, like docx, pptx, pdf, png, txt, etc. The most important file types for us engineers are CSV, JSON, and XML, because these are able to hold (semi-)structured data so the LLM can infer relationships and create appropriate answers.