Critic (experimental)

A ready-to-run example is available here.

What is a critic?

A critic is an evaluator that analyzes agent actions and conversation history to predict the quality or success probability of agent decisions. The critic runs alongside the agent and provides:

Quality scores: Probability scores between 0.0 and 1.0 indicating predicted success
Real-time feedback: Scores computed during agent execution, not just at completion
Iterative refinement: Automatic retry with follow-up prompts when scores are below threshold

You can use critic scores to build automated workflows, such as triggering the agent to reflect on and fix its previous solution when the critic indicates poor task performance.

Quick start

When using the Faheem Code LLM Provider (llm-proxy.*.faheemcode.ai), the critic is automatically configured - no additional setup required.

Understanding critic results

Critic evaluations produce scores and feedback:

score: Float between 0.0 and 1.0 representing predicted success probability
message: Optional feedback with detailed probabilities
success: Boolean property (True if score >= 0.5)

Results are automatically displayed in the conversation visualizer:

Screenshot: Critic results in SDK visualizer.

Accessing results programmatically

from faheemcode.sdk import Event, ActionEvent, MessageEvent

def callback(event: Event):
    if isinstance(event, (ActionEvent, MessageEvent)):
        if event.critic_result is not None:
            print(f"Critic score: {event.critic_result.score:.3f}")
            print(f"Success: {event.critic_result.success}")

conversation = Conversation(agent=agent, callbacks=[callback])

The critic supports automatic iterative refinement - when the agent finishes a task but the critic score is below a threshold, the conversation automatically continues with a follow-up prompt asking the agent to improve its work.

How it works

Agent completes a task and calls FinishAction
Critic evaluates the result and produces a score
If score < success_threshold, a follow-up prompt is sent automatically
Agent continues working to address issues
Process repeats until score meets threshold or max_iterations is reached

Configuration

Use IterativeRefinementConfig to enable automatic retries:

from faheemcode.sdk.critic import APIBasedCritic, IterativeRefinementConfig

# Configure iterative refinement
iterative_config = IterativeRefinementConfig(
    success_threshold=0.7,  # Retry if score < 70%
    max_iterations=3,       # Maximum retry attempts
)

# Attach to critic
critic = APIBasedCritic(
    server_url="https://llm.faheemcode.ai/vllm",
    api_key=api_key,
    model_name="critic",
    iterative_refinement=iterative_config,
)

Parameters

Parameter	Type	Default	Description
`success_threshold`	`float`	`0.6`	Score threshold (0-1) to consider task successful
`max_iterations`	`int`	`3`	Maximum number of iterations before giving up

Custom follow-up prompts

By default, the critic generates a generic follow-up prompt. You can customize this by subclassing CriticBase and overriding get_followup_prompt():

from faheemcode.sdk.critic.base import CriticBase, CriticResult

class CustomCritic(APIBasedCritic):
    def get_followup_prompt(self, critic_result: CriticResult, iteration: int) -> str:
        score_percent = critic_result.score * 100
        return f"""
Your solution scored {score_percent:.1f}% (iteration {iteration}).

Please review your work carefully:
1. Check that all requirements are met
2. Verify tests pass
3. Fix any issues and try again
"""

Example workflow

Here's what happens during iterative refinement:

Iteration 1:
  → Agent creates files, runs tests
  → Agent calls FinishAction
  → Critic evaluates: score = 0.45 (below 0.7 threshold)
  → Follow-up prompt sent automatically

Iteration 2:
  → Agent reviews and fixes issues
  → Agent calls FinishAction
  → Critic evaluates: score = 0.72 (above threshold)
  → ✅ Success! Conversation ends

Troubleshooting

Critic evaluations not appearing

Verify the critic is properly configured and passed to the Agent
Ensure you're using the Faheem Code LLM Provider (llm-proxy.*.faheemcode.ai)

API authentication errors

Verify LLM_API_KEY is set correctly
Check that the API key has not expired

Ensure iterative_refinement config is attached to the critic
Check that success_threshold is set appropriately (higher values trigger more retries)
Verify the agent is using FinishAction to complete tasks

Ready-to-run example

This example demonstrates iterative refinement with a moderately complex task - creating a Python word statistics tool with specific edge case requirements. The critic evaluates whether all requirements are met and triggers retries if needed.

"""Iterative Refinement with Critic Model Example.

This is EXPERIMENTAL.

This example demonstrates how to use a critic model to shepherd an agent through
complex, multi-step tasks. The critic evaluates the agent's progress and provides
feedback that can trigger follow-up prompts when the agent hasn't completed the
task successfully.

Key concepts demonstrated:
1. Setting up a critic with IterativeRefinementConfig for automatic retry
2. Conversation.run() automatically handles retries based on critic scores
3. Custom follow-up prompt generation via critic.get_followup_prompt()
4. Iterating until the task is completed successfully or max iterations reached

For Faheem Code LLM proxy (llm-proxy.*.faheemcode.ai), the critic is auto-configured
using the same base_url with /vllm suffix and "critic" as the model name.
"""

import os
import re
import tempfile
from pathlib import Path

from faheemcode.sdk import LLM, Agent, Conversation, Tool
from faheemcode.sdk.critic import APIBasedCritic, IterativeRefinementConfig
from faheemcode.sdk.critic.base import CriticBase
from faheemcode.tools.file_editor import FileEditorTool
from faheemcode.tools.task_tracker import TaskTrackerTool
from faheemcode.tools.terminal import TerminalTool

# Configuration
# Higher threshold (70%) makes it more likely the agent needs multiple iterations,
# which better demonstrates how iterative refinement works.
# Adjust as needed to see different behaviors.
SUCCESS_THRESHOLD = float(os.getenv("CRITIC_SUCCESS_THRESHOLD", "0.7"))
MAX_ITERATIONS = int(os.getenv("MAX_ITERATIONS", "3"))

def get_required_env(name: str) -> str:
    value = os.getenv(name)
    if value:
        return value
    raise ValueError(
        f"Missing required environment variable: {name}. "
        f"Set {name} before running this example."
    )

def get_default_critic(llm: LLM) -> CriticBase | None:
    """Auto-configure critic for Faheem Code LLM proxy.

    When the LLM base_url matches `llm-proxy.*.faheemcode.ai`, returns an
    APIBasedCritic configured with:
    - server_url: {base_url}/vllm
    - api_key: same as LLM
    - model_name: "critic"

    Args:
        llm: The LLM instance to derive critic configuration from.

    Returns:
        An APIBasedCritic if the LLM is configured for Faheem Code proxy,
        None otherwise.

    Example:
        llm = LLM(
            model="anthropic/claude-sonnet-4-5",
            api_key=api_key,
            base_url="https://llm.faheemcode.ai",
        )
        critic = get_default_critic(llm)
        if critic is None:
            # Fall back to explicit configuration
            critic = APIBasedCritic(
                server_url="https://my-critic-server.com",
                api_key="my-api-key",
                model_name="my-critic-model",
            )
    """
    base_url = llm.base_url
    api_key = llm.api_key
    if base_url is None or api_key is None:
        return None

    # Match: llm-proxy.{env}.faheemcode.ai (e.g., staging, prod, eval)
    pattern = r"^https?://llm\.faheemcode\.ai"
    if not re.match(pattern, base_url):
        return None

    return APIBasedCritic(
        server_url=f"{base_url.rstrip('/')}/vllm",
        api_key=api_key,
        model_name="critic",
    )

# Task prompt designed to be moderately complex with subtle requirements.
# The task is simple enough to complete in 1-2 iterations, but has specific
# requirements that are easy to miss - triggering critic feedback.
INITIAL_TASK_PROMPT = """\
Create a Python word statistics tool called `wordstats` that analyzes text files.

## Structure

Create directory `wordstats/` with:
- `stats.py` - Main module with `analyze_file(filepath)` function
- `cli.py` - Command-line interface
- `tests/test_stats.py` - Unit tests

## Requirements for stats.py

The `analyze_file(filepath)` function must return a dict with these EXACT keys:
- `lines`: total line count (including empty lines)
- `words`: word count
- `chars`: character count (including whitespace)
- `unique_words`: count of unique words (case-insensitive)

### Important edge cases (often missed!):
1. Empty files must return all zeros, not raise an exception
2. Hyphenated words count as ONE word (e.g., "well-known" = 1 word)
3. Numbers like "123" or "3.14" are NOT counted as words
4. Contractions like "don't" count as ONE word
5. File not found must raise FileNotFoundError with a clear message

## Requirements for cli.py

When run as `python cli.py <filepath>`:
- Print each stat on its own line: "Lines: X", "Words: X", etc.
- Exit with code 1 if file not found, printing error to stderr
- Exit with code 0 on success

## Required Tests (test_stats.py)

Write tests that verify:
1. Basic counting on normal text
2. Empty file returns all zeros
3. Hyphenated words counted correctly
4. Numbers are excluded from word count
5. FileNotFoundError raised for missing files

## Verification Steps

1. Create a sample file `sample.txt` with this EXACT content (no trailing newline):
`​`​`
Hello world!
This is a well-known test file.

It has 5 lines, including empty ones.
Numbers like 42 and 3.14 don't count as words.
`​`​`

2. Run: `python wordstats/cli.py sample.txt`
   Expected output:
   - Lines: 5
   - Words: 21
   - Chars: 130
   - Unique words: 21

3. Run the tests: `python -m pytest wordstats/tests/ -v`
   ALL tests must pass.

The task is complete ONLY when:
- All files exist
- The CLI outputs the correct stats for sample.txt
- All 5+ tests pass
"""

llm_api_key = get_required_env("LLM_API_KEY")
# Use a weaker model to increase likelihood of needing multiple iterations
llm_model = os.getenv("LLM_MODEL", "anthropic/claude-haiku-4-5-20251001")
llm = LLM(
    model=llm_model,
    api_key=llm_api_key,
    top_p=0.95,
    base_url=os.getenv("LLM_BASE_URL"),
)

# Setup critic with iterative refinement config
# The IterativeRefinementConfig tells Conversation.run() to automatically
# retry the task if the critic score is below the threshold
iterative_config = IterativeRefinementConfig(
    success_threshold=SUCCESS_THRESHOLD,
    max_iterations=MAX_ITERATIONS,
)

# Auto-configure critic for Faheem Code proxy or use explicit env vars
critic = get_default_critic(llm)
if critic is None:
    print("⚠️  No Faheem Code LLM proxy detected, trying explicit env vars...")
    critic = APIBasedCritic(
        server_url=get_required_env("CRITIC_SERVER_URL"),
        api_key=get_required_env("CRITIC_API_KEY"),
        model_name=get_required_env("CRITIC_MODEL_NAME"),
        iterative_refinement=iterative_config,
    )
else:
    # Add iterative refinement config to the auto-configured critic
    critic = critic.model_copy(update={"iterative_refinement": iterative_config})

# Create agent with critic (iterative refinement is built into the critic)
agent = Agent(
    llm=llm,
    tools=[
        Tool(name=TerminalTool.name),
        Tool(name=FileEditorTool.name),
        Tool(name=TaskTrackerTool.name),
    ],
    critic=critic,
)

# Create workspace
workspace = Path(tempfile.mkdtemp(prefix="critic_demo_"))
print(f"📁 Created workspace: {workspace}")

# Create conversation - iterative refinement is handled automatically
# by Conversation.run() based on the critic's config
conversation = Conversation(
    agent=agent,
    workspace=str(workspace),
)

print("\n" + "=" * 70)
print("🚀 Starting Iterative Refinement with Critic Model")
print("=" * 70)
print(f"Success threshold: {SUCCESS_THRESHOLD:.0%}")
print(f"Max iterations: {MAX_ITERATIONS}")

# Send the task and run - Conversation.run() handles retries automatically
conversation.send_message(INITIAL_TASK_PROMPT)
conversation.run()

# Print additional info about created files
print("\nCreated files:")
for path in sorted(workspace.rglob("*")):
    if path.is_file():
        relative = path.relative_to(workspace)
        print(f"  - {relative}")

# Report cost
cost = llm.metrics.accumulated_cost
print(f"\nEXAMPLE_COST: {cost:.4f}")

LLM_BASE_URL="https://llm.faheemcode.ai" LLM_API_KEY="$LLM_API_KEY" \
  uv run python examples/01_standalone_sdk/34_critic_example.py

Example output

📁 Created workspace: /tmp/critic_demo_abc123

======================================================================
🚀 Starting Iterative Refinement with Critic Model
======================================================================
Success threshold: 70%
Max iterations: 3

... agent works on the task ...

✓ Critic evaluation: score=0.758, success=True

Created files:
  - sample.txt
  - wordstats/cli.py
  - wordstats/stats.py
  - wordstats/tests/test_stats.py

EXAMPLE_COST: 0.0234

Next steps

Observability - Monitor and log agent behavior
Metrics - Collect performance metrics
Stuck Detector - Detect unproductive agent patterns

Critic (experimental)

What is a critic?

Quick start

Understanding critic results

Accessing results programmatically

Iterative refinement with a critic

How it works

Configuration

Parameters

Custom follow-up prompts

Example workflow

Troubleshooting

Critic evaluations not appearing

API authentication errors

Iterative refinement not triggering

Ready-to-run example

Example output

Next steps

What is a critic?​

Quick start​

Understanding critic results​

Accessing results programmatically​

Iterative refinement with a critic​

How it works​

Configuration​

Parameters​

Custom follow-up prompts​

Example workflow​

Troubleshooting​

Critic evaluations not appearing​

API authentication errors​

Iterative refinement not triggering​

Ready-to-run example​

Example output​

Next steps​

What is a critic?

Quick start

Understanding critic results

Accessing results programmatically

Iterative refinement with a critic

How it works

Configuration

Parameters

Custom follow-up prompts

Example workflow

Troubleshooting

Critic evaluations not appearing

API authentication errors

Iterative refinement not triggering

Ready-to-run example

Example output

Next steps