Running locally generally involves running OS/OW Ms. Many come from China. However, Google’s Gemma OS/OW Ms are competitive with many of the Chinese models. Also, there are older OS/OW Ms from Meta that may be very practical for small IA applications.

OS/OW Ms have to be downloaded. Hugging Face is a good source for models, infrastructure supporting their use, etc. An organization running models locally can avoid the per-token pricing plans. As pointed out above, there is cost associated with the hardware and maintenance of the OS/OW M software. 

Another consideration is longevity. Local models will not be automatically updated. That can be an advantage or disadvantage. On the advantage side are consistency both for users and for Intelligent Agents. On the disadvantage side is the work required to investigate new models and to manage the upgrading process.

All of these considerations apply to both data center and Edge device implementations. However, the cost/performance comparison between Edge and data center will change over time. Data centers are going to get more powerful. However, Edge devices may improve more dramatically over time. For example, what runs today on a $10,000 device will run in a few years on a $1,000 one. Of course, the frontier models are likely to increase in size, at least for the next couple of years. 

In this evolving environment, what is most efficient to run in one place today may be more efficient in another place tomorrow.

Architecture Alternatives 

Based on the above, a number of different basic architecture approaches emerge:

Online Frontier models; 
Online proprietary models that are no longer Frontier; 
Online OS/OW Ms: 
Local OS/OW Ms: 
Hybrid part online / part local (Apple seems to be taking this approach): 
Hybrid Frontier model / OS/OW Ms.

There is one further architecture alternative. One in which activity is funneled through an AI that figures out which of the above is best suited to efficiently handle the task at hand and helps with crafting complete, accurate, and efficient prompt language.

Organization Decision Making Guidelines 

Currently, organizations have to make usage and architectural decisions based on their functional and efficiency needs. Over time, it is likely that vendors will appear wth products that support a variety of the emerging architecture alternatives. Some of those solutions will be useful for some organizations. Some organizations will want to continue to create their own. What seems clear is that there will be an ongoing high degree of volatility.

Implications for organizations of ongoing volatility are: 1.) well-structured staff training will continue to pay dividends even as change occurs; and 2.) whatever architectural, usage, and billing decisions are made today, they should be made with an eye to being easily changed tomorrow.

Although there are indications that the results of AI can not be copyrighted, it appears that patent protection around what is covered here is still in effect. Thus, some consideration for, and search of, patents is prudent in making decisions in this space.