Right LLM and Configuration
for Your Operations Agent
Not only is GenAI technology evolving rapidly, but the appearance of the technology is generating rapid changes in many of the underlying businesses seeking to take advantage of it.
The verified version is presented to staff members who seek to perfect it. Once perfected, it goes through a final test process. Depending on the outcome of the final test process, it may be
further iterated.
In large enough organizations, separate groups are responsible for the perfection phase and the final test phase. Separating these functions is valuable because it removes normal human
biases.
LLM Monitoring and Updating Plan
Because AI technology is evolving so quickly, it is important to create a plan to evaluate, including ongoing performance of the agent, and emerging GenAI technology improvements in general, and newly released LMMs in particular.
Agents at installation may perform well. Over time, their performance may degrade. Performance may degrade because the conditions that the model was created to deal with have changed, rising user
expectations, or technical limitations.
For example, early medical agent implementations found that their AI agents worked well when first installed, but after a while, their performance degraded. Institutions using them reported that
they had installed the agents in large part because they anticipated significant cost savings. What they found instead was that they had to add expensive staff to monitor and update the
agents.
Industry speculation was that these performance problems occurred because the Context Window of the LLM supporting the agent became full. The context window provides information to the LLM that
is important in making the next inference request. Information on each inference request and result is added to the context window. Frontier models keep increasing the size of the context window.
Frontier models have also been developing ways of editing the context window. But even so, over time, the window can become full. As LLM technology continues its rapid evolution, this and other
technical limitations may fade away. Only to have new ones appear. Thus, having a plan for monitoring performance is necessary.
Not only is GenAI technology evolving rapidly, but the appearance of the technology is generating rapid changes in many of the underlying businesses seeking to take advantage of it. As the
business requirements change, LLM technology evolves, end user expectations rise, or problems appear with existing agents, it will be necessary to consider updating an agent. To do so, it
is important to establish a set of criteria for deciding when it makes sense to consider updating an agent. Over time, it may be necessary to update these criteria as the technology changes. But
care should be taken to make sure that the criteria are not changed a priori just to justify a decision that has been made for political or other reasons.
To support this expected monitoring of performance and possible updating, documentation should be captured and retained in a way that makes it easily available. This documentation should
contain the original requirements, significant experience during development, testing process, and results, and a way to capture the ongoing performance monitoring results. The AI agent
monitoring process needs to be combined with the results of regularly evaluating alternative LLMs as they appear. The result needs to be kept in a secure manner that is easily accessible to
update or when needed in planning and execution.
Conclusion
Choosing the optimal LLM and configuring it correctly is critical for a successful AI agent implementation. Matching the selection process to the requirements is key. Properly configuring the selected LLM is also very important. Monitoring agent performance after installation and monitoring LLM technology evolution are important steps in maintaining good ongoing operational efficiency.
