systems are primarily central site based. Central site systems struggle with latency. They face the dilemma of quantity of data. The more data you have the more likely you are to be able to find an attack. However, the more data you have the longer it takes to process it. Attack latency is falling dramatically. Central site systems struggle to keep pace. On the remediation side, these defense systems either use pre-canned remediation scripts or rely on manual intervention. In the world of rapidly changing GenAI attacks, the remediation scripts struggle to respond effectively, and manual responses can’t keep up with attack cycle times.

A new approach is needed.

Social Engineering

Social engineering attacks involve manipulation to get innocent people to do things and act in ways they otherwise would not. We have seen that GenAI is capable of creating very convincing video, audio and text that appear to come from trusted sources asking for money, seeking to obtain credentials, attempting to sway public opinion, etc. GenAI is becoming an increasingly formidable weapon for social engineering attacks.

Hallucinations

Hallucination is a term that has come into common usage as a label for a particular set of GenAI negative side effects. In humans, hallucinations are perceptions of sensory data (images, sounds, tastes, smells, etc.) that do not actually exist, yet seem real.

GenAI is capable of creating outputs of increasingly “high fidelity,” so high that they can be very convincing and reliable, but which upon examination may be false, erroneous, misrepresentative, etc. But whereas the use of GenAI in social engineering attacks involves intentional deception, hallucinations can be unintended outcomes caused by a variety of model- and training-related factors. Their potential negative impacts, however, are no less serious. Moreover, when a GenAI system is confronted with a hallucination, the GenAI system generally continues to maintain that it is true.

This is a critical and widespread problem. Recent studies show that for simple hallucinations depending on the particular public GenAI systems deployed, the range of occurrence of hallucinations ranges from 3 percent to 27 percent and may be even higher. For complex questions in an area such as law, rates range from 69 percent to 88 percent.

As shown in the taxonomy above, there are two general types of hallucinations: those concerning corrupted information, and others concerning damaging actions. Within corrupted information, there are two subtypes: documentation; and audio/video.

Negative impacts of corrupted documentation examples include, e.g., citing non-existent cases in legal briefs, or falsely attributing information that negatively affects reputations, etc. Hallucinations resulting from corrupted audio/video purported to be of real events that never happened. These can impact political, business, social, etc. domains.

Within examples of damaging actions are two subcategories: actions on people; and actions on things. Examples concerning damaging actions negatively affecting people include medical systems such as medical GenAI diagnostics, lab tests, treatments, etc. Examples concerning damaging actions negatively affecting both people and things include military systems killing the wrong people, designating wrong targets, etc. Similar examples exist for police, legal, political systems, etc., and a broad range of infrastructure control systems.

Conclusion

With the foundation understanding of this taxonomy and accompanying discussion, future consideration can turn to mitigating these negative side effects.