AI for IoT Latency: the Closer the Better
Just as the advent of the general-purpose microprocessor led to the development of the microcontroller, it is likely that a micro AI specialized processor will be developed.
Smaller models are improving their capabilities. This comes from two directions: more capable general capabilities, and more capable specialized capabilities. General capabilities of smaller
models are coming from a combination of more engineering attention on the problem and new technology. The most dramatic of the new technologies is the use of Distillation. Distillation uses the
outputs of larger models to train small models. Thus, giving the small models capabilities that approach the larger models. A second approach is the use of specialized training. This can occur
after the initial training to ‘fine tune’ the model. Or, where economics justify the effort, training a new specialized model from scratch.
Network Processing
If it is not possible to place the AI in the same package as the sensor/actuator, then attention should turn to placing the AI in the network as close to the package as possible. It is often
tempting to aggregate AI needs for a series of packages into a single, more powerful processor that can support them all. This tends to lengthen latency. Aggregating AI processing into a single
processor can allow for a more powerful processor with larger memory. But the aggregated processor has to be further away from the sensors/actuators. From a latency point of view, multiple
smaller AIs closer tend to be a better solution.
In some cases, there may be enough system resources in the communications processor near the package to run a small LLM there. In any case, effort should be made to avoid a hub and spoke network. That is a network where data travels from an Edge down a spoke to a central hub and then back out again on another spoke.
In some cases, there may be enough system resources in the communications processor near the package to run a small LLM there. In any case, effort should be made to avoid a hub and spoke network. That is a network where data travels from an Edge down a spoke to a central hub and then back out again on another spoke.
IoT Infrastructure Evolution
Just as the advent of the general-purpose microprocessor led to the development of the Microcontroller, it is likely that a micro-AI specialized processor will be developed. For example, a
Micro-GPU/CPU combo.
At the same time, software evolution will continue. Although the frontier model companies are focusing on the largest LLMs, there is still some promising work going on with smaller LLMs.
As the quantity of sophisticated model developer talent continues to expand, it will become economical for companies focused on IoT to develop their own specialized models. For these companies, this means investments in: training existing staff and in adding advanced AI capability staff as market economics allow.
Brian Case is a microcomputer expert whose career started with early microprocessors. He has an interesting perspective on AI evolution. “When we started designing microprocessors in the 80s, we were trying to go beyond minicomputers, and we did. We failed, though, to anticipate tiny, cheap microcomputers that could bring sound and music to greeting cards; we simply failed to extrapolate the effects of Moore's law correctly. I think this time we should anticipate AI so ubiquitous that it will be in our light bulbs.”
At the same time, software evolution will continue. Although the frontier model companies are focusing on the largest LLMs, there is still some promising work going on with smaller LLMs.
As the quantity of sophisticated model developer talent continues to expand, it will become economical for companies focused on IoT to develop their own specialized models. For these companies, this means investments in: training existing staff and in adding advanced AI capability staff as market economics allow.
Brian Case is a microcomputer expert whose career started with early microprocessors. He has an interesting perspective on AI evolution. “When we started designing microprocessors in the 80s, we were trying to go beyond minicomputers, and we did. We failed, though, to anticipate tiny, cheap microcomputers that could bring sound and music to greeting cards; we simply failed to extrapolate the effects of Moore's law correctly. I think this time we should anticipate AI so ubiquitous that it will be in our light bulbs.”
Conclusion
Latency is a key issue for many IoT systems. Adding AI to IoT can dramatically increase capabilities. However, it can bring latency challenges. Keeping sensor, actuator, and processing very
close to each other produces the lowest latency. With today’s hardware and GenAI systems, this may be challenging. There are two ways of dealing with this challenge: finding ways to reduce the
size of the LLM, and placing the LLM in the network as close to the sensors and actuators as possible. As AI hardware and software continue their rapid evolution, the challenging aspects of
placing sensors, actuators, and LLMs in a single package will decrease. Therefore, IoT systems should be designed to fit today’s technology capabilities and to be able to migrate easily to new
emerging capabilities as they arise. That is, don’t wait. But plan for migration.
