Glossary

An AI pipeline is a type of data pipeline that supports AI use cases so that development and operations teams can work in a structured and repeatable manner. Data pipelines typically involve the tools necessary for extracting, transforming, and loading (ETL) data from various sources into a target system. AI pipelines provide the workflow structure, tools and data delivery methods that development and operations teams need in order to power their AI systems.

The key stages of an AI pipeline are:

1. Data cleaning and preprocessing
2. Model training
3. Testing and deployment
4. Monitoring and updating

Artificial general intelligence (AGI) refers to AI models that can perform tasks without having been trained with such tasks in mind. Two primary characteristics of AGI as it is envisioned (it is currently theoretical) are being able to self-teach and having human-like cognitive capabilities across a wide range of tasks.

AI bias, also known as algorithmic bias and machine learning bias, arises when AI algorithms reflect prejudice that is inaccurate and unfair. This often happens due to sampling bias present in the data sets used to train AI algorithms. AI bias can result in severely decreased functionality of AI models.

The Chief AI Officer (CAIO), fast becoming a staple of the modern C-suite, is responsible for setting the company’s AI agenda. The CAIO should advance smart AI adoption, recognize and balance AI benefits with risks, and collaborate across departments to foster AI strategy and vision. As part of a nationwide effort to ensure safe AI use in public service, the U.S. federal government recently issued an order requiring that all federal agencies have a senior leader overseeing their AI systems.

What makes AI special is its particular aptitude for finding patterns within data inputs. On the way to zeroing in on these patterns, AI distinguishes between these inputs by categorizing (“classifying”) them. Classifiers are algorithms whose designated function is to assign class labels to data inputs. Often, they supplement AI models to enhance those models’ performance.

Traditional data loss prevention tools are not tailored to handle the risks associated with GenAI.

Organizations’ sensitive data is increasingly being streamed to GenAI tools. When this happens, it is likely that this data will subsequently be used for future training and potentially be generated by these tools on external endpoints.

This is one major reason why more organizations are implementing GenAI security measures to augment their overall DLP strategy.

Deepfakes are visual or audio representations of individually identifiable people that are created or altered using AI. Often, the purpose behind deepfakes is to mislead viewers and listeners regarding the substance of a given image, video or audio recording. Much of the general public considers deepfakes to be the epitome of disinformation and post-truth.

A denial of wallet (DoW) attack is an‍ increasingly common attack whereby an attacker gains illicit access to a GenAI app, usually through the receiving end’s chat interface and often by bypassing an unimplemented or awkwardly implemented rate limit. 

Once the attacker has access to their target’s LLM, they are still constrained by the system prompt. The next step is 'jailbreaking' the system prompt to bypass it. Once the attacker bypasses the system prompt, they essentially have direct and free access to the target LLM's backend.

In March of 2024, the European Parliament approved the Artificial Intelligence Act, the first legislative proposal of its kind in Europe. The AI Act aims to ensure AI safety and compliance with legal requirements and human rights, all the while encouraging innovation. The regulation addresses risks, promotes ethical use, and establishes standards for high-risk applications.

Explainable AI (XAI) refers to AI models whose pathways from prompt to output are intelligible to humans. The term is also used to describe processes and methods that make these pathways intelligible. 

Whether or not AI should be explainable depends on how the model is being put to use. If fairness, accuracy and objectivity are paramount, such as with AI models that generate credit scores, XAI is vital. On the other hand, chatbots for conversational dialogue with humans do not need to present specific justifications for their outputs, so they do not need to be explainable.

Firewalls are network security systems that monitor and control the flow of traffic between trusted sources and untrusted sources. With the growing prominence of GenAI, more and more firewalls are tailored to GenAI-specific threats, such as:

• Prompt injections, jailbreaking, and denial of wallet (DoW) attacks
• Unsafe and other harmful content coming from LLM responses
• Unintentional disclosure of confidential information by in-house GenAI applications

In April 2024, Prompt Security partnered with F5 to launch a firewall for AI, enabling F5 Distributed Cloud Services customers to protect their GenAI applications at every touchpoint. Organizations can make sure they are safeguarded against prompt injections, sensitive data disclosure, harmful responses, and other threats.

Fuzzing in AI refers to real-world testing that makes an AI-application more resistant to harmful data inputs. A form of quality assurance (QA), it involves presenting an application with unexpected and/or random data inputs so as to uncover vulnerabilities that may otherwise go undetected. This allows GenAI developers to better understand and enhance the resilience and safety of their system prompts.

‍Prompt Fuzzer is an interactive tool that lets GenAI developers test their GenAI applications by executing various LLM-based attacks. After testing a given application, Prompt Fuzzer provides a security score and evaluation based on the test’s outcomes.

This refers to the complete range of GenAI tools used in an organization. IT teams should be vigilant in keeping this list updated and accessible.

GenAI Red Teaming identifies potential vulnerabilities of GenAI applications by mimicking adversarial attacks against those applications. This in-depth assessment technique tests the resilience of GenAI interfaces and applications against a variety of threats, such as prompt injections, jailbreaks and toxicity. This helps ensure that the interfaces and applications are safe and secure to face the external world.

Generative AI Security (GenAI Security) encompasses all the measures, technologies, policies, and security controls that protect organizations from risks associated with the use of GenAI.

Broadly speaking, GenAI Security risks can be divided into two areas:

• Usage: Protecting your company from risks of employee GenAI use and applications using ChatGPT, Gemini Copilots and other third-party GenAI apps.

Integration: Protecting your company from its own first-party GenAI apps that rely on first- or third-party LLMs.

AI hallucinations are AI-generated responses presented as factual and sensical despite being incorrect, misleading and/or incoherent. Causes of AI hallucinations include inadequate model training and AI bias.

ISO/IEC 42001 is an international standard that specifies requirements for establishing, implementing, maintaining, and continually improving artificial intelligence management systems (AIMS) within organizations. It is the world’s first AI management system standard, providing guidance on challenges specific to AI, including ethical considerations, transparency, and continuous learning.

ISO/IEC 42001 is designed to facilitate responsible development and use of AI systems by entities that provide AI-based products and services. For such organizations, it sets out a structured way to manage risks and opportunities associated with AI, balancing innovation with governance.

‍

Indirect prompt injection is a type of prompt injection attack where adversarial instructions are introduced through a third-party data source, such as a web search or API call. For instance, when conversing with Bing Chat, which has internet search capabilities, you may instruct the chatbox to explore a certain website. If this website contains malicious prompts, cleverly concealed as white text, Bing Chat may unwittingly read and comply with these instructions. 
‍

What distinguishes this from direct injection is that the threat is initiated not by the user or the language model, but rather by a malicious third party. You are not explicitly instructing Bing Chat to convey certain information; instead, you are guiding it to an external resource that may contain manipulative content.

Jailbreaking represents a specific category of prompt injection where the goal is to coerce a GenAI application into deviating from its intended behavior and predetermined guidelines. This is typically achieved by crafting inputs that exploit system vulnerabilities, which can enable responses that encroach on restrictions or guardrails. Notable examples include the widely discussed "Dan" or "many-shot" jailbreak incidents, where the AI systems acted without their usual constraints.

Large language models (LLMs) are a type of artificial intelligence technology that processes and generates human-like text based on input. 

Built on deep learning architectures, particularly transformer networks, LLMs are trained on vast datasets comprising text from the internet, books, articles, and other written materials. This extensive training enables them to understand context, grasp nuances in language, and respond to prompts with coherence and relevance. LLMs can perform a wide range of tasks, from answering questions and writing essays to composing poetry and generating code. Their ability to understand and produce language has made them valuable tools for applications in natural language processing, content creation, customer service, and education. 

LLMs are becoming increasingly sophisticated. They are improving their understanding of complex instructions and enhancing their ability to generate more accurate, contextually appropriate responses. Nevertheless, despite their capabilities, LLMs are not without challenges. They often demonstrate biases in their outputs and there are numerous ethical considerations surrounding their use.

‍

Just as development operations (DevOps) refers to the tools and methods that automate processes between software developers and IT teams, machine learning operations (MLOps) refers to the tools and practices that automate processes between machine learning engineers, data scientists, software developers and IT teams. MLOps plays a key role in advancing machine learning models to production and monitoring them thereafter.

Multimodal AI systems can process and understand information from various forms of data, such as text, images, audio, and video. These systems integrate and interpret data from multiple sensory channels, enabling more complex understanding and interaction with the world. Multimodal AI can recognize and analyze patterns within and across these different data types, enhancing applications in fields like autonomous vehicles, healthcare diagnostics, and virtual assistants by providing richer, more nuanced responses and actions.

Prompt injection is a prompt where attackers use carefully crafted inputs to manipulate a large language model (LLM) into behaving in an undesired manner. This manipulation tricks the LLM into executing the attacker's intentions. This threat becomes particularly concerning when the LLM is integrated with other tools such as internal databases, APIs, or code interpreters, creating a new attack surface.

Prompt leaks are a specific form of prompt injections where a large language model (LLM) inadvertently reveals its system instructions or internal logic. This issue arises when prompts are engineered to extract the underlying system prompt of a GenAI application. As prompt engineering becomes increasingly integral to the development of GenAI apps, any unintentional disclosure of prompts may be considered exposure of proprietary code or intellectual property.

Key Concerns:

• Intellectual Property Disclosure: The unauthorized revelation of proprietary information embedded in system prompts.
• Recon for Downstream Attacks: System prompt leaks could serve as reconnaissance for more damaging prompt injections.
• Reputational Damage: Fallout from accidental prompt disclosure can include the release of embarrassing information, hurting the organization’s public image.

‍

Responsible AI is a general term encompassing all efforts and practices geared towards maximizing AI’s social benefits and minimizing its negative effects on humans and real-world circumstances. Responsible AI places strong emphasis on how AI is used as well as on how this use is communicated. 

Some of the weight behind responsible AI is explicitly meant to protect humanity and the world from something viewed as potentially harmful. Other weight behind responsible AI comes from AI advocates who believe that a more promising cost-benefit analysis of AI will help advance AI use cases.

Shadow AI refers to any use of GenAI tools that occurs without an organization’s approval or oversight. 

ChatGPT marked the beginning of the widespread adoption of GenAI tools. Today, an average company’s employees use over 50 different GenAI tools in their daily operations. Mastering and managing these tools is crucial for success, yet most of them are being used in an unofficial capacity.

Key Concerns:

• Limited Visibility: Few companies understand the full scope of GenAI tool usage within their ranks.
• Absence of Governance: There is no external body that offers sufficient oversight in regard to GenAI tool usage.
• Compliance Risks: A lack of visibility into GenAI usage means an increased risk of violating regulatory standards.
  ‍

Sensitive Data Exposure: Unauthorized access to or misuse of confidential information.

Artificial superintelligence (ASI) is AI that is superior, even far superior, to human performance.

Current AI models depend on data inputs that enable them to simulate human capabilities. They often function at a high level and outperform humans, but only for specific tasks. Theoretically, ASI would be able to outperform humans across a wide array of disciplines, from cognition and problem-solving to creativity and social interaction, and could even do so with less or no instruction from humans. 

There is no broad consensus regarding if and when ASI will exist in practice. 

‍

Symbolic AI deals with symbolic representations rather than numerical data. In symbolic AI, specific knowledge and rules are placed in models. This makes symbolic AI intelligible to humans, and thus explainable (XAI) in nature. However, this deprives symbolic AI from one of AI’s core strengths, which is establishing rules in logic organically based on patterns identified in and between inputs and outputs.

Synthetic data is data created by GenAI models. The algorithms used to create synthetic data are trained by real-world data, so there is a connection between synthetic data and real-world data, albeit one characterized by a degree (or several degrees) of separation. 

A common motivation for creating synthetic data is to help train AI models. An advantage of synthetic data is that it does not contain personal, sensitive information. A disadvantage is that if the algorithms used to create the synthetic data are poorly trained, there is no mechanism to prevent biases or other imperfections from influencing the synthetic data.

‍

A jailbroken large language model (LLM) behaving unpredictably can pose significant risks, potentially endangering an organization, its employees, and its customers. Repercussions range from embarrassing social media posts to negative customer experiences, and may even include legal complications. To safeguard against such issues, it’s crucial to implement protective measures. These measures are often referred to as toxicity detection.

Key Concerns:

• Toxicity: Preventing harmful or offensive content
• Bias: Ensuring fair and impartial interactions
• Racism: Avoiding racially insensitive or discriminatory content
• Brand Reputation: Maintaining a positive public image
• Inappropriate Sexual Content: Filtering out unsuitable sexual material

‍

Transfer learning (TL) is when knowledge patterns from iterations of a certain task are applied to improve performance of a similar task.

AI models learn by identifying patterns. A model’s ability to learn well often depends on the context in which these patterns may be identified. This is why AI models dedicated to a single task (that is, a single context) or narrow set of tasks are more likely to succeed.

Transfer learning aims to maximize models’ performance across tasks within this current limitation. If a model is pre-trained for one task, and a second task is similar enough to the first task, the model may succeed in performing the second task without having been trained for it.

‍

In October 2023, US President Joe Biden signed Executive Order 14110, also known as Executive Order on Safe, Secure, and Trustworthy Development and Use of Artificial Intelligence, or simply Executive Order on AI. 

The executive order is the most comprehensive piece of US governance on artificial intelligence to date. It outlines policy goals that relate to promoting competition in the field of AI, preventing AI-related threats to civil rights and national security, and maximizing US competitiveness in AI. It also requires US federal agencies to appoint Chief AI Officers.

‍

Unimodal AI systems process and interpret information from a single type of data or sensory input, such as text, images, or audio. These systems specialize in tasks within their specific modality, excelling in areas like text analysis, image recognition, and speech recognition. Unimodal AI focuses on depth within its domain, enabling precise and efficient processing, analysis, and generation of data in its singular mode of operation.

In a visual prompt injection attack, a textual prompt might be benign, but an associated image harbors malicious instructions formatted and colored so as to remain imperceptible to users. With GenAI apps evolving into multi-modal systems capable of processing images and other diverse inputs, the range of potential origins for such injections is expanding.

eBPF is a revolutionary technology that brings a new paradigm for application security, offering unprecedented visibility and control at the kernel level. 

Some of eBPF’s benefits include:

• Foregoing the need to change the app’s source code
• Enabling unprecedented visibility into all GenAI app interactions with the LLM, vector database, APIs, etc.
• Lower downtime risk due to reduced intrusion

‍