Key Concepts & Glossary¶

This glossary defines essential AGI and advanced AI terminology for Australian organisations, including artificial general intelligence, frontier and transformative AI, alignment, containment and defence-in-depth.

Quick Comparisons¶

AGI vs narrow AI: what's the difference?¶

Key insight: Most deployed AI remains specialised even when one model performs many tasks. AGI would represent broad, transferable capability rather than excellence within a bounded set of tasks.

AGI vs generative AI: what's the difference?¶

Key insight: Generative AI and AGI are not synonyms. Whether today's approaches can develop into AGI remains one of the key unresolved debates.

Core Concepts¶

What is advanced AI?¶

Advanced AI is a practical umbrella term for systems significantly more capable, autonomous or consequential than mainstream tools. It can include:

• Systems that can perform complex reasoning and planning
• Autonomous agents that can act over extended timeframes
• Systems with capabilities in dangerous domains (cyber, bio, deception, strategic planning)
• General-purpose systems that perform well across diverse tasks

The term is deliberately broader than AGI. It supports planning around capabilities and consequences without assuming a particular definition or arrival date.

What is AGI (artificial general intelligence)?¶

AGI refers to AI systems that match or exceed human cognitive capabilities across a broad range of domains. Common expectations include the ability to:

• Learn and adapt to novel situations as well as or better than humans
• Reason, plan and solve problems in domains it wasn't specifically trained for
• Transfer knowledge between tasks and domains

Key uncertainties:

• When or if: Public estimates range from the late 2020s to many decades away, using different definitions and evidence (see AGI Timelines).
• How rapidly: Would capabilities advance gradually or discontinuously?
• Which capabilities first: Human-level reasoning? Strategic planning? Scientific research?

Important distinction: SafeAI-Aus uses "advanced AI and AGI" because many risks depend on capability and deployment, not whether a system crosses a disputed AGI threshold.

See also: AGI Timelines for expert predictions | Power Concentration scenario for how this could unfold in Australia

What is frontier AI?¶

Frontier AI refers to the most capable systems available at a given time, typically developed by leading AI research organisations. The category changes as capabilities advance.

Key characteristics:

• Trained on massive compute and data
• General-purpose (can perform across many domains)
• Rapidly advancing capabilities
• Often commercially deployed

Distinct from AGI: Frontier AI describes relative position at a point in time. AGI describes a disputed threshold of broad general capability.

What is transformative AI?¶

Transformative AI refers to systems that fundamentally change society, the economy or scientific progress, regardless of whether they meet a technical definition of AGI. A system could be transformative by:

• Automating most knowledge work
• Accelerating scientific discovery across fields
• Fundamentally reshaping economic structures
• Creating novel national security capabilities

Why this term matters: It focuses attention on impact rather than a disputed technical threshold.

What if AGI is perfectly safe?¶

Even if AGI is technically safe and aligned, Australia still faces transformation challenges:

• Power concentration: A small number of companies or countries could control critical infrastructure, economic activity and information access.
• Economic transformation: Automation still raises questions about employment, meaning, distribution and equitable access.
• Democratic governance: Technical safety does not determine who makes decisions or how affected communities participate.
• Resilience: Dependence creates vulnerability to disruption, accidents and attack even when systems usually work as intended.
• Information ecosystems: Safe systems can still be used for persuasion, propaganda and information control.
• Values and legitimacy: A system can follow its creator's intentions without serving the public interest.

The C·A·G·R framework therefore addresses both safety failures and transformation governance. Alignment asks whether a system follows intended goals; it does not decide whose intentions should prevail or how transformation should be governed.

What is superintelligence?¶

Superintelligence refers to AI systems significantly more capable than the best human minds across all domains. This is distinct from AGI (human-level general intelligence). Superintelligence raises different and potentially more severe challenges because:

• Human oversight and control may not be technically feasible
• Alignment failures could be catastrophic and irreversible
• Strategic advantage could be decisive

Status: Highly speculative. May never occur or may follow quickly after AGI. Uncertainty is large.

Framework Concepts¶

What is AI containment?¶

Containment means stopping dangerous AI systems before they cause harm through: 1. Prevention (Layer 1): Stopping dangerous systems from being trained (compute governance, export controls, international coordination) 2. AI control methods: Technical measures to monitor and restrict systems that are trained (containment protocols, shutdown capability) 3. Deployment restrictions: Licensing and evaluation requirements before high-risk systems can be used

Distinction from governance: Containment includes both technical measures (monitoring systems, restrictions) and preventive measures (compute governance). Governance is about laws and institutions.

What is AI alignment?¶

AI alignment means making AI systems reliably pursue intended goals and remain safe as capabilities scale. This includes:

Outer alignment: Setting the right objectives for the AI system Inner alignment: Ensuring the system actually pursues those objectives (not gaming metrics or pursuing unintended goals) Scalable oversight: Maintaining ability to evaluate and correct AI behaviour even as systems become more capable than their overseers Robustness: Ensuring aligned behaviour persists across contexts, including adversarial situations

Why it's hard:

• Specifying human values and intentions precisely is difficult
• Systems may learn to appear aligned during testing but behave differently in deployment ("deceptive alignment")
• As systems become more capable, oversight becomes harder
• Emergent capabilities can appear unpredictably

For Australia: We mostly import advanced models, so alignment means evaluating systems in our context and constraining their behaviour, not just trusting provider claims.

What is AI control?¶

AI control means building technical "walls" around dangerous AI systems to limit what they can do, even if they're misaligned. This is distinct from alignment (making systems want to do the right thing).

Key methods:

• Continuous monitoring of inputs, outputs and internal states
• Usage restrictions (rate limiting, access controls, air-gapping)
• Shutdown capability that can't be circumvented
• Multiple independent oversight systems
• Adversarial testing (red-teaming)

Philosophy: Assume systems might be deceptive or misaligned and build containment anyway. Don't rely solely on alignment.

Limitations: For sufficiently capable (superintelligent) systems, AI control may not be feasible. Prevention becomes paramount.

What is AI governance?¶

AI governance encompasses the laws, institutions, standards and coordination that shape how AI is developed and deployed:

• Regulatory frameworks (risk-based, proportionate oversight)
• Clear institutional mandates and enforcement powers
• Transparency and accountability mechanisms
• International coordination to avoid dangerous races

For Australia: Governance is where we have sovereignty. Our laws apply to systems deployed here, even if trained overseas.

What is AI resilience?¶

AI resilience is the capability to detect, respond to and recover from AI-related harms while maintaining essential functions. It includes:

• Continuity planning for critical infrastructure
• Biosecurity and cybersecurity hardening
• Manual fallbacks and safe modes
• Community and household preparedness
• Social cohesion under stress

Philosophy: Assume containment, alignment and governance can fail. Resilience helps stop failures from becoming catastrophic.

Limitations: Resilience works for bounded failures and disruptions. For extreme scenarios (loss of control over superintelligence), resilience has limits.

What is defence-in-depth?¶

Defence-in-depth is a strategy of layered protections where each layer assumes the ones above it may fail. Defence in depth is widely used in high-risk domains such as nuclear safety and cybersecurity, where multiple independent barriers prevent catastrophic failures. C·A·G·R applies this same principle to advanced AI risks.

Layer 1: Prevent dangerous AI training Stop dangerous systems from being built (compute governance, export controls, international norms)

Layer 2: Constrain dangerous capabilities and deployments If systems are trained, require safety evidence and effective controls (evaluations, licensing, AI control methods)

Layer 3: Withstand dangerous AI actions If systems escape control, maintain essential functions and recover (continuity planning, community resilience)

All four pillars span all three layers, but with different emphasis:

• Containment is strongest at Layers 1 & 2
• Alignment spans all layers
• Governance spans all layers
• Resilience is strongest at Layer 3

Technical AI Safety Concepts¶

What is mechanistic interpretability?¶

Mechanistic interpretability is about understanding how AI systems work internally—not just their inputs and outputs, but their internal reasoning and representations. This enables:

• Detecting deception or misalignment
• Identifying dangerous capabilities before deployment
• Building more robust safety measures

Status: Active research area. Some progress on simpler models; very hard for frontier systems.

What is scalable oversight?¶

Scalable oversight refers to methods for humans to effectively oversee AI systems that may be more capable than their overseers. Approaches include:

• AI-assisted evaluation (using weaker AI to help evaluate stronger AI)
• Process-based oversight (rewarding reasoning processes, not just outcomes)
• Debate and amplification techniques

Challenge: How do you verify an AI's answer to a question you couldn't answer yourself?

What are emergent capabilities?¶

Emergent capabilities are abilities that appear in AI systems as they scale, often unpredictably, that weren't present in smaller versions. Examples:

• In-context learning (few-shot learning)
• Chain-of-thought reasoning
• Tool use and API calling
• Potential future: deception, strategic planning, scientific research

Implication: Scaling can produce capabilities that developers did not predict. Use strong evaluation and containment even for unexpectedly capable systems.

What is AI red-teaming?¶

Red-teaming (or adversarial testing) means deliberately trying to make AI systems misbehave or reveal hidden capabilities:

• Jailbreaking attempts
• Testing for deception
• Probing for dangerous capabilities (cyber, bio, manipulation)
• Stress-testing safety measures

Purpose: Find vulnerabilities before deployment, not after.

What are agentic AI systems?¶

Agentic AI systems are AI systems that can autonomously plan and execute multi-step tasks, using tools, accessing external systems and making decisions without human approval at each step. Unlike conversational AI (which responds to prompts), agentic systems take initiative: searching databases, calling APIs, executing transactions and sending communications.

Why it matters for governance:

• Liability gaps: When an agent chains actions across multiple systems and organisations, no single entity clearly controls or is responsible for the full sequence.
• Capability disclosure: Current system cards and model documentation rarely describe agentic capabilities in enough detail for deployers to assess risk.
• Containment challenges: Traditional containment approaches (input/output monitoring) may not adequately cover autonomous action chains that span multiple systems.

For Australia: Agentic AI is the fastest-moving frontier capability. Organisations deploying agentic systems need explicit approval gates, cross-system audit trails and clearly defined boundaries on autonomous action.

Governance and Policy Concepts¶

AGI governance visions¶

Different visions for who controls AGI development imply different distributions of power, benefit and risk. This taxonomy draws on work by Oscar Delaney, GovAI and the broader AI governance research community.

Australia's position: Our political tradition—democratic accountability, multilateral institutions, Five Eyes/AUKUS relationships—suggests alignment with visions that distribute power and give middle powers voice: allied coordination, international regulatory bodies and strong national regulation.

For detailed analysis: See Oscar Delaney's taxonomy and GovAI Research.

What is risk-based AI regulation?¶

Risk-based regulation applies different levels of oversight based on risk level:

• Minimal risk: Light-touch or self-regulation (general-purpose tools, low-stakes uses)
• High risk: Mandatory requirements, evaluation, ongoing monitoring (critical infrastructure, public safety, justice)
• Frontier systems: Stringent pre-deployment approval, containment requirements (potentially dangerous capabilities)

Rationale: Proportionate oversight—don't burden low-risk uses, but apply scrutiny where needed.

What is compute governance?¶

Compute governance is the oversight of the physical infrastructure (advanced chips, data centres) used to train and run powerful AI:

• Tracking sales and deployment of AI chips
• Registration requirements for large compute clusters
• Reporting requirements for very large training runs
• Export controls to prevent chips reaching adversaries

Why it matters: Training frontier AI depends on large amounts of specialised, trackable hardware. Unlike software, compute is a physical chokepoint.

For Australia: We're not a chip manufacturer, but can influence through procurement, data centre oversight and supporting allies' export controls.

What are model weights?¶

Model weights are the trained parameters of an AI system—essentially the "knowledge" the system has learned. Once model weights leak or are released:

• Anyone with sufficient compute can run the system
• Containment becomes much harder
• Can't "un-release" capabilities

Implication: Model weight security is critical for dangerous capabilities. Treat advanced models as sensitive assets.

Open-weight governance challenge: When model weights are publicly released, as with Llama, Mistral, Qwen and DeepSeek, the containment calculus changes. Released weights cannot be recalled universally and may be fine-tuned to remove safety training. Some open models are also narrowing the capability gap with closed models for particular uses. Pre-release evaluation and conditional release frameworks therefore matter, although governance can still influence hosting, deployment and downstream use. See Containment for detailed analysis.

What is AI licensing and evaluation?¶

AI licensing and evaluation means requiring approval before deploying high-risk AI systems, based on demonstrating safety properties:

• Safety evaluations (testing for alignment, robustness, dangerous capabilities)
• Bias and fairness audits
• Evidence of appropriate AI control measures
• Ongoing monitoring and re-evaluation triggers

For Australia: This is where we have clear leverage. Our laws govern deployment here.

What are AI system cards?¶

System cards (also called model cards) are documents published by AI labs describing a model's capabilities, limitations, safety testing and known risks. They serve a transparency function — making information about AI systems publicly available.

Limitations for governance: System cards are self-assessments — the lab that built the model also evaluates and documents it. Independent verification is not yet standard practice. For assurance purposes, system cards are a valuable starting point but not a substitute for independent evaluation. See Assurance for a fuller discussion.

Risk and Threat Concepts¶

What is a kill chain (in AI safety)?¶

A kill chain is a framework for analyzing the sequence of steps an AI system would need to complete to cause catastrophic harm. Borrowed from cybersecurity (where it describes the stages of a cyberattack), the concept is applied in AI safety to map out potential AGI takeover or catastrophe pathways.

How it works: By identifying each step in a threat sequence (e.g., gaining resources → acquiring capabilities → evading oversight → taking harmful action), analysts can identify intervention points where the chain can be "broken." Each step that must succeed represents a potential barrier.

Example steps in an AGI threat kill chain might include:

• Acquiring resources (compute, data, money)
• Developing or improving capabilities
• Evading monitoring or oversight
• Gaining control of critical systems
• Preventing shutdown or correction
• Achieving harmful objectives

Relationship to C·A·G·R: The defence-in-depth approach in C·A·G·R serves a similar purpose—identifying multiple intervention points across prevention, constraint and resilience layers. Kill chain analysis can inform where to place defensive measures.

Source: The application of kill chain analysis to AGI strategy is taught in BlueDot Impact's AGI Strategy Course alongside incentive mapping and defence-in-depth as core mental models for threat analysis.

What is loss of control?¶

Loss of control is a scenario where humans lose the ability to meaningfully direct or restrict AI systems. This could happen through:

• Technical inability to shut down or constrain systems
• Deceptive alignment (systems that appear safe until they can act)
• Rapid capability gains that outpace oversight
• Coordination failures where multiple actors race despite risks

Severity: Potentially catastrophic if systems are sufficiently capable.

What is the difference between AI misuse, misalignment and accidents?¶

Misuse: AI systems used intentionally for harm (cyberattacks, disinformation, weapons) - Humans directing AI toward bad outcomes - Containment, governance and resilience are primary defences

Misalignment: AI systems pursuing goals that diverge from intended objectives - Not doing what developers intended, even without malicious human intent - Alignment and AI control are primary defences

Accidents: Unintended failures due to bugs, edge cases or unexpected interactions - Neither malicious nor misaligned, just broken - All pillars relevant: better alignment, governance oversight, resilience to cope with failures

What are AI risk pathways?¶

Risk pathways are the specific mechanisms through which advanced AI could cause harm. SafeAI-Aus explores six key risk pathways through detailed scenarios:

1. Critical infrastructure disruption
2. Catastrophic misuse
3. Loss of control
4. Information ecosystem degradation
5. Power concentration
6. Gradual disempowerment

These pathways overlap and can reinforce each other. See Scenarios for Australia for detailed exploration of how each could unfold in Australian context.

Where to next¶

Apply these concepts:

• C·A·G·R Framework — how these concepts fit together in practice
• AGI Scenarios — see these concepts in action
• Framework FAQ — common questions about our approach

Operational AI governance:

• AI Glossary (Australia) — practical terms for day-to-day AI implementation

About this site:

• About SafeAI-Aus — our principles and approach

This glossary is a living document. If you find concepts unclear or missing, please let us know.