Choice of LLM

Some might think that the choice of LLM should be first in the development process. However, it is important to develop enough of the application to understand what is required of the LLM. At the current stage of the technology, it appears that smaller and simpler LLM’s have less of a tendency to hallucinate. A smaller model also requires less resources. An understanding of the application will help drive this. It also can help to divide an application into several separate intelligent agents that interact with each other. This can have many benefits including allowing each of the LLMs powering each of the agents to be smaller.

AI Infrastructure Alternatives

There are two basic approaches to AI application infrastructure: Edge or Data Center.

There are a broad range of local hardware alternatives. The Apple Mac Mini or Mac Studio and similar Windows-Linux form factor machines provide an interesting alternative. They can be loaded with an intelligent agent AI help desk system designed to be plugged into the existing help desk workstation. It can be designed such that it doesn’t take technical expertise to plug it in. This allows the AI to be installed quickly and easily. 

Maintenance is simplified by considering the installed package a bundle and not planning on complex software updates in the field. This provides a graceful path to prototype, then determines the optimal mix of intelligent agents and human staff.

Generally, local operation involves using an open-source model. Many of the current open-source models come from China leading to concerns about privacy and intellectual property.

There are two types of AI data centers: IaaS (Inference as a Service) and organization dedicated. An IaaS provider has the advantage of taking on the installation and support of the LLM itself. This may include automatic updates to new more advanced LLMs. Such updates can be a double-edged sword as they may produce unexpected behavior changes. IaaS providers also have the disadvantage of potential problems with privacy and intellectual property leakage.

Organization dedicated AI data centers assume responsibility for LLM installation and maintenance, but they don’t suffer from LLM version control, privacy, and IP leakage problems. In some cases, large organizations may be able to license LLMs from frontier LLM developers. Others will use open-source models. Some large companies have talked about developing and training their own LLMs. How practical is open to debate.

Over time, small organizations are likely to use the edge architecture. While large organizations use the more centralized data center.

Security

Some may be tempted to think that by taking the staff member out of the help desk equation, the security risk of social engineering and other forms of attack may be reduced. Unfortunately, this is not the case. Special consideration needs to be given to security.

Role of User Groups

Because GenAI technology is so new and evolving so quickly, we are still early on the learning curve in how best to use it to achieve optimal results. As a result, it is important for developers, deployers, managers and users to find a way to move up the learning curve quickly. One of the best ways of doing this is to become a member of a user group focused on the technology in your application space. One organization working on creating such domain specific AI user groups is the AIWG. Such user groups provide a way for people working in the domain to share lessons learned, best practices, recent developments, etc. This can be very effective in moving up the learning curve.

Conclusion

With GenAI, the help desk customer experience can be made better while reducing costs. To capture the opportunity, organizations need to move up the learning curve and develop optimal implementations. One of the best ways to prepare to do this is to join a domain focused user group.