Automated Failure Attribution in LLM Multi-Agent Systems: A Practical Guide to Identifying Responsibility for Task Failures

Overview

LLM-based multi-agent systems have shown remarkable promise in tackling complex tasks through collaborative agent interactions. Yet, failures remain a persistent challenge—often caused by a single agent's mistake, a miscommunication, or an information transmission error. Identifying which agent is at fault and when the failure occurred is a labor-intensive process, akin to finding a needle in a haystack of logs. Researchers from Penn State, Duke, Google DeepMind, and other institutions have pioneered Automated Failure Attribution and released the Who&When benchmark dataset. This guide walks you through the problem, the dataset, and practical methods to automate failure attribution in your own multi-agent systems.

By the end of this tutorial, you will be able to:

• Understand the core challenge of failure attribution.
• Download and use the Who&When dataset.
• Implement automated attribution methods (baseline, advanced LLM-based).
• Avoid common pitfalls when diagnosing agent failures.

Prerequisites

Knowledge

• Familiarity with LLMs (e.g., GPT-4, LLaMA) and multi-agent architectures.
• Basic understanding of Python (for code examples).
• Conceptual grasp of sequence labeling or classification tasks (helpful but not required).

Tools

• Python 3.8+ with libraries: transformers, torch, datasets, pandas, numpy.
• Access to a GPU (recommended for LLM-based methods) or use cloud services like Google Colab.
• Git to clone the official repository.

Step-by-Step Instructions

1. Understanding the Who&When Dataset

The dataset simulates multi-agent collaborations (e.g., software development, question answering) where agents execute tasks sequentially. Each instance contains:

• Interaction log: a timeline of agent actions and messages.
• Ground truth labels: the responsible agent and the timestamp (step) of the failure root cause.

Download the dataset from Hugging Face:

from datasets import load_dataset

dataset = load_dataset("Kevin355/Who_and_When")
print(dataset['train'][0]['log'])  # inspect first example

2. Data Preprocessing: Creating Input Formats

Attribution methods require the log to be structured. A common approach is:

1. Parse the log into a sequence of turns, each with agent ID, action, and content.
2. Create a candidate list of (agent, step) pairs as potential failure points.
3. For each candidate, build a prompt that asks the LLM to judge if that agent at that step caused the failure.

Example Python snippet:

def build_candidates(log):
    turns = log.split('\n')
    candidates = []
    for i, turn in enumerate(turns):
        # extract agent name from turn format: "Agent_A: ..."
        agent = turn.split(':')[0]
        candidates.append((agent, i))
    return candidates

3. Baseline Method: Heuristic Rules

Start with a simple baseline: attribute failure to the last agent that performed an action before the system detected an error. This achieves modest accuracy but is fast.

def baseline_attribution(log):
    # assume last action is the culprit
    lines = log.strip().split('\n')
    last_turn = lines[-1]
    agent = last_turn.split(':')[0]
    step = len(lines) - 1
    return agent, step

4. Advanced Method: LLM-as-Judge

Leverage an LLM (e.g., GPT-4) to analyze the entire log and output the responsible agent and step. The prompt is critical.

Prompt Design

prompt = f"""
You are analyzing a multi-agent system interaction log.
The system failed at the end. Identify which agent caused the failure and at which turn (0-indexed).
Provide only the agent name and turn number as JSON.

Log:
{log}

Output:
{{"agent": "", "turn": }}
"""

Then parse the LLM response:

import json
from openai import OpenAI

client = OpenAI()
response = client.chat.completions.create(
    model="gpt-4",
    messages=[{"role": "user", "content": prompt}]
)
result = json.loads(response.choices[0].message.content)
fault_agent = result['agent']
fault_turn = result['turn']

5. Evaluation Metrics

Compare your predictions against ground truth labels. Use:

• Agent Accuracy: percentage of correct agent identification.
• Turn Accuracy: percentage of exact turn match.
• Combined F1: average of agent and turn precision/recall.

Example evaluation script:

from sklearn.metrics import accuracy_score

agent_pred = [p['agent'] for p in predictions]
agent_true = [t['agent'] for t in ground_truth]
print(f"Agent Accuracy: {accuracy_score(agent_true, agent_pred):.2f}")

Common Mistakes

Ignoring Contextual Dependencies

A failure may propagate across multiple steps. Do not treat each turn independently; consider the whole chain.

Overlooking Agent Identity Ambiguity

Two agents might have similar names (e.g., "Agent_1" vs "Agent_10"). Use unique IDs and parse carefully.

Using Too Short Prompts

LLMs need the full log to reason. Truncating logs loses critical evidence. If log length exceeds token limits, use sliding windows or hierarchical summarization.

Assuming Single Failure Point

Some failures result from multiple agents. The Who&When dataset labels a primary cause. Focus on that, but be aware of compound failures.

Summary

Automated failure attribution is essential for debugging complex LLM multi-agent systems. This guide introduced the Who&When dataset, provided step-by-step methods from baselines to LLM-based attribution, and highlighted common pitfalls. With the open-source code and dataset, you can integrate attribution into your own development pipeline, drastically reducing manual log analysis. Future work may extend to real-time monitoring and multi-failure attribution.