SUBSCRIBE NOW
IN THIS ISSUE
PIPELINE RESOURCES

The Future of GenAI


There will be attempts to manage the power consumption problem. As the models become larger, they consume more power. For example, there is a proposal to restart the Three Mile Island nuclear reactor...

From the perspective of an executive in a large organization today, the correct response seems pretty straight forward. From society’s point of view it is not so clear.

GenAI’s Future Development

Some see GenAI getting incrementally better as we go out into the future. Most of the leading-edge models already use MoE (Mixture of Experts) architectures. These appear to have increased efficiency by directing inference requests to subsections of a large model focused on the particular domain of the request. There may be more that can be achieved along this path.

Another incremental approach is the implementation of self-reflective architectures. Users have noticed that if the results of an inference request are fed back to a model with a new request asking if there is anything wrong with the response, they often get refinements. Some LLM technologists have suggested architectures that do this internally. Because details are proprietary, it is not possible to say for sure. But it appears that ChatGPT o1 may already be doing this.

It is possible that there will be combinations of MoE’s and self-reflection. Other hybrid forms may emerge as well. Some are suggesting the combination of GenAI and previously developed Machine Learning (ML) forms. The argument here being that these previous forms (even going back to Knowledge Engineering), while lacking the power of GenAI, do not have hallucinations. Combining the forms may mitigate hallucination. There are also non-ML forms that may be combined with GenAI to mitigate negative side effects.

We may also see an ongoing increase in number of parameters. From billions to trillions and beyond. These increases may reveal new unexpected capabilities in a similar fashion to how previous increases in the numbers of parameters revealed new unexpected capabilities. However, we may be approaching the area of diminishing returns.

There will be attempts to manage the power consumption problem. As the models become larger, they consume more power. For example, there is a proposal to restart the Three Mile Island nuclear reactor to power a single GenAI data center. The increase in efficiency described above will help. But the increase in parameters will more than offset it.

There will be an ongoing back and forth between those developing guardrails for GenAI and those trying to defeat them. The movement of capable models to edge devices favors those seeking to defeat guardrails.

This process of incremental improvement is proceeding very quickly. The cycle time for a major GenAI developer is four to six months, but each developer has a different start date, and so they are leapfrogging each other. The result may be blurring the three-stage evolution process described above. New systems with new capabilities are coming out faster than organizations can figure out how to best exploit the last one.

Likely Course

All of this is happening but is probably unlikely to produce giant steps forward. Two unexpected things happened to make GenAI happen. The first was a theoretical breakthrough — the publication of the Attention paper that created a quantum jump from all the previous work on deep neural networks. Then came a series of accidents. People discovered that, as each time the number of parameters (how symbols are characterized and differentiated) increased from hundreds to hundreds of thousands, to hundreds of millions, to billions, etc., new capabilities emerged.

Thus, what seems likely is that there are likely to be four independent innovation paths:

  1. Increasing the size of today’s models leading to new and unexpected capabilities.
  2. New ways of modifying today’s models to achieve the same performance with smaller sizes.
  3. Ongoing incremental changes as outlined above.
  4. An unpredictable theoretical breakthrough that will leapfrog today’s GenAI technology creating new and unpredictable capabilities (similar to the Attention paper and increases in parameters).


FEATURED SPONSOR:

Latest Updates





Subscribe to our YouTube Channel