Alignment: Making AI Systems Reliably Safe¶

Purpose: Assess whether AI systems behave compatibly with human intent, Australian law and public safety Audience: Government, business and technical teams | Time: 25-35 minutes

What is AI Alignment?¶

When you deploy an AI system, how do you know it is genuinely pursuing the goals you intended rather than merely appearing to?

AI alignment is about making AI systems more likely to behave in ways compatible with human intent, Australian law and public safety. The AI alignment problem is central to AI safety because advanced systems may pursue unintended goals, leading to AI misalignment or, in extreme scenarios, AI loss of control.

But "the right thing" is more complex than it first appears. A system can be:

• technically aligned with its training objective, but
• deployed in contexts where that objective causes harm, or
• integrated into workflows where human incentives push it toward harmful behaviour

Effective alignment requires attention to technical, socio-technical and contextual levels. This is why the C·A·G·R framework treats alignment as one pillar of defence-in-depth.

What does AI alignment solve—and what doesn't it solve?¶

Why does AI alignment matter for Australia?¶

Australia is likely to import most advanced AI models from overseas—frontier AI from labs such as OpenAI, Anthropic and DeepMind. Australian deployers generally will not control their design or training objectives.

But we will:

• Deploy those systems in Australian organisations, services and infrastructure
• Fine-tune them for specific uses and contexts
• Integrate them into workflows, incentive structures and decision-making processes

For Australia, alignment is therefore primarily about:

Evaluation: Can we independently assess whether imported systems behave safely in our context?

Constraint: Can we prevent misaligned behaviour in deployment, even if we can't change the underlying model?

Accountability: Can we explain and justify AI-mediated decisions to citizens, courts and Parliament?

What are the three layers of AI alignment?¶

For AGI preparedness, distinguishing model alignment (foundation), system alignment (integration) and institutional alignment (governance) avoids treating RLHF as a complete safety solution. Recent work on multi-agent institutional alignment emphasises that AGI risk concerns agents operating under distribution shift and multi-agent dynamics—not just isolated model behaviour.

1. Model alignment (foundation)¶

What it is: The relationship between a system's training objective and its actual behaviour.

Technical alignment concerns:

• What the system was optimised for during training
• How it behaves under normal conditions
• How it behaves under adversarial prompts, edge cases or distribution shift
• Whether it reliably refuses harmful requests
• Whether capabilities emerge that weren't intended or tested

Why it's hard:

• We often can't directly specify what we want; we use proxy objectives
• Systems can find unexpected ways to satisfy objectives (specification gaming)
• Capabilities can emerge at scale that weren't present in smaller versions
• Behaviour that seems aligned in testing may diverge in deployment
• Modern approaches (RLHF, Constitutional AI) have made progress but don't solve deeper challenges

What this means for you:

Independent evaluation capability

For critical deployments, independently test provider claims before approval. Australia should also partner with universities and research institutes to build domestic evaluation capacity.

Red-teaming and adversarial testing

Use adversarial prompts to identify edge cases, failure modes and unexpected behaviour.

Standards for documentation

Require providers to document training objectives, data sources, known limitations and realistic adversarial testing. Make this available to regulators and, where appropriate, the public.

Examples of technical misalignment in practice:

• A content moderation system optimised to maximise user engagement systematically amplifies divisive content
• A credit scoring system trained on historical data reproduces historical discrimination
• An energy grid optimisation system that crashes under conditions outside its training distribution
• A language model that generates plausible but false information because it's optimised for coherence, not truth

2. System alignment (integration)¶

What it is: How AI systems interact with human organisations, incentives and workflows when deployed in real-world contexts.

A system can be technically well-aligned but still cause harm when:

• It's integrated into workflows where humans lose meaningful oversight
• Organisational incentives reward gaming the system
• Staff lack training or authority to challenge AI outputs
• Feedback loops reinforce rather than correct problematic behaviour

Why it matters:

Many practical alignment problems are socio-technical: the system does what it was designed to do, but:

• The organisation deploys it in inappropriate contexts
• Humans defer to it when they shouldn't
• Nobody notices when edge cases arise
• The system optimises for measurable proxies rather than actual goals

What this means for you:

Meaningful human oversight

Oversight requires trained people with the authority and capability to challenge AI systems. Regularly test whether staff can overrule recommendations in practice.

Appropriate workflow design

Map AI systems within decision-making processes. Give people enough information to make informed choices and maintain manual operating capability.

Incentive alignment

Check whether performance metrics reward gaming or discourage staff from challenging AI outputs. Design feedback loops that surface problems.

Contestability and redress

People affected by AI-mediated decisions should understand why they were made and have practical appeal mechanisms. Design for explainability, not just accuracy.

Examples of socio-technical misalignment:

• Hospital staff rubber-stamp AI triage recommendations because they're measured on speed, not accuracy of human oversight
• A benefits assessment system where appeals are technically possible but practically impossible for most people to navigate
• Financial trading systems where no human can understand or challenge decisions fast enough
• Performance management systems where both managers and employees game AI metrics

3. Institutional alignment (governance)¶

What it is: Fit between AI systems and Australian law, values, democratic norms and institutional context.

A system trained overseas may be aligned with its designers' intentions but misaligned with:

• Australian legal requirements (privacy, discrimination, procedural fairness)
• Cultural norms and expectations
• Local context and conditions
• Democratic accountability and transparency expectations

Why it matters:

Frontier AI systems are mostly trained by a small number of US organisations. Their design choices reflect:

• US legal frameworks (which differ from ours)
• Corporate objectives (which may not align with public interest)
• Training data that over-represents some contexts and under-represents others
• Values and assumptions that may not match Australian society

Australia inherits these choices unless it actively assesses and constrains imported systems.

What this means for you:

Clear expectations for Australian deployment

AI deployed in public services or critical infrastructure may engage obligations under Australian privacy and anti-discrimination law, as well as administrative-law requirements such as procedural fairness. Organisations should obtain advice on the law that applies to their context rather than treating this summary as a compliance determination.

Evaluation in Australian context

Test with Australian data, edge cases and scenarios. Check performance for under-represented groups and whether systems respect Australian legal and cultural norms.

Local fine-tuning and adaptation

Where possible, fine-tune imported models for Australian context using Australian data and feedback. Building your capacity to adapt systems—rather than simply accepting them as-is—is essential for contextual alignment.

Democratic oversight

If you're in government, give citizens and their representatives meaningful input on acceptable uses. Explain how systems are used in public services and create practical ways to challenge decisions that conflict with community values.

Examples of contextual misalignment:

• A hiring system trained on US data disadvantages candidates from Australian education systems
• A content moderation system that reflects US First Amendment norms rather than Australian defamation and vilification laws
• A legal reasoning system trained primarily on US law that confidently gives wrong advice about Australian law
• Emergency response systems optimised for US urban contexts that fail in regional Australia

Why all three layers matter for AGI: As systems become more capable and autonomous, the gap between model alignment (what we can partly solve technically) and system/institutional alignment (what we must govern democratically) widens. Model alignment alone doesn't answer "whose intentions should guide transformative technology?" or "how do communities participate in governance?" This is why Containment addresses system and institutional layers through deployment controls, Governance explicitly handles institutional alignment through democratic input, and Resilience assumes partial alignment failure across all three layers. Defence-in-depth is essential.

How does AI alignment connect to threat pathways?¶

Alignment problems connect to multiple threat pathways:

Power concentration: When alignment is defined by a small number of overseas actors, their values and interests shape outcomes more than Australian democratic processes.

Gradual disempowerment: When organisations lose capability to assess alignment themselves, they must trust provider claims — losing agency over critical systems.

Catastrophic misuse: Misaligned systems or alignment failures can be exploited by malicious actors. Systems that reliably refuse harmful requests under normal conditions might be jailbroken or fine-tuned for harm.

Critical infrastructure disruption: Systems optimised for narrow efficiency metrics rather than resilience and safety can fail catastrophically under stress.

Loss of control: As systems become more capable, alignment becomes harder to verify and maintain. Misaligned behaviour may only emerge under novel conditions.

What can different actors do for AI alignment?¶

How does alignment work in defence-in-depth?¶

Layer 3: Withstand (what we must be able to do)

Detect when deployed systems are misaligned, enable rapid response to alignment failures and maintain the capability to switch off or replace systems if needed.

Layer 2: Constrain (where we have sovereignty)

Evaluate alignment before allowing deployment in critical domains, conduct ongoing monitoring of deployed systems and enforce consequences when alignment failures cause harm.

Layer 1: Prevent (where we have limited influence)

Contribute to international research on alignment methods, support development of evaluation standards and set clear alignment expectations for systems deployed in Australia, creating incentives for better practices upstream.

See alignment in practice¶

These scenarios show alignment challenges and why alignment alone isn't sufficient:

• Loss of Control — alignment failure at scale
• Information Ecosystems — aligned AI used for persuasion
• Gradual Disempowerment — safe AI that still transforms society

Where to next¶

Other framework pillars:

• Framework Overview — how alignment combines with containment, governance and resilience
• Containment — preventing misaligned systems from being deployed
• Governance — requiring and incentivising alignment in practice
• Resilience — detecting and responding when alignment fails

For researchers:

• Framework FAQ — how SafeAI-Aus differs from mainstream AI safety
• AGI Concepts — technical AI safety terminology