---
title: "eval-cli"
description: "Headless CLI binary for running Zed's agent in evaluation/benchmark environments. Designed to work inside containerized environments like [Harbor](https://harborframework.com/) where the repository is already checked out and API keys are provided via environment variables."
type: skill
canonical_url: https://claudary.paisolsolutions.com/skills/readme-673
source: "Claudary"
difficulty: intermediate
author: "Claude Code Knowledge Pack"
date: 2026-07-10T11:36:48.879Z
license: CC-BY-4.0
attribution: "eval-cli — Claudary (https://claudary.paisolsolutions.com/skills/readme-673)"
---

# eval-cli
Headless CLI binary for running Zed's agent in evaluation/benchmark environments. Designed to work inside containerized environments like [Harbor](https://harborframework.com/) where the repository is already checked out and API keys are provided via environment variables.

## Overview

# eval-cli

Headless CLI binary for running Zed's agent in evaluation/benchmark
environments. Designed to work inside containerized environments like
[Harbor](https://harborframework.com/) where the repository is already
checked out and API keys are provided via environment variables.

Uses the same `NativeAgent` + `AcpThread` pipeline as the production Zed
editor — full agentic loop with tool calls, subagents, and retries, just
without a GUI.

## Building

### Native (for local testing on the same OS)

```
cargo build --release -p eval_cli
```

### Cross-compile for Linux x86_64 (from macOS or other hosts)

Harbor containers run Linux x86_64. Use the Docker-based build script:

```
crates/eval_cli/script/build-linux
```

This produces `target/eval-cli` (an x86_64 Linux ELF binary). You can
also specify a custom output path:

```
crates/eval_cli/script/build-linux --output ~/bin/eval-cli-linux
```

## Standalone usage

```
eval-cli \\
  --workdir /testbed \\
  --model anthropic/claude-sonnet-4-6-latest \\
  --instruction "Fix the bug described in..." \\
  --timeout 600 \\
  --output-dir /logs/agent
```

Reads API keys from environment variables (`ANTHROPIC_API_KEY`,
`OPENAI_API_KEY`, etc.). Writes `result.json`, `thread.md`, and
`thread.json` to the output directory.

### Exit codes

| Code | Meaning                            |
| ---- | ---------------------------------- |
| 0    | Agent finished                     |
| 1    | Error (model/auth/runtime failure) |
| 2    | Timeout                            |
| 3    | Interrupted (SIGTERM/SIGINT)       |

## Harbor integration

The `zed_eval/` directory contains a Python package that
implements Harbor's `BaseInstalledAgent` interface, allowing eval-cli to
be used with `--agent-import-path` without modifying Harbor's source code.

### Setup

```
pip install -e crates/eval_cli/harbor/
```

### Running with a local binary

Build for Linux first, then pass the binary path:

```
crates/eval_cli/script/build-linux

harbor run -d "swebench_verified@latest" \\
  --agent-import-path zed_eval.agent:ZedAgent \\
  --ae binary_path=target/eval-cli \\
  -m anthropic/claude-sonnet-4-6-latest
```

The agent uploads the binary into the container during setup — no
download URL needed during local iteration.

### Running with a download URL

For CI or when the binary is hosted somewhere:

```
harbor run -d "swebench_verified@latest" \\
  --agent-import-path zed_eval.agent:ZedAgent \\
  --ak download_url=https://example.com/eval-cli \\
  -m anthropic/claude-sonnet-4-6-latest
```

### Setting a timeout

Pass `EVAL_CLI_TIMEOUT` via `--ae`:

```
harbor run -d "swebench_verified@latest" \\
  --agent-import-path zed_eval.agent:ZedAgent \\
  --ak binary_path=target/eval-cli \\
  --ae EVAL_CLI_TIMEOUT=600 \\
  -m anthropic/claude-sonnet-4-6-latest
```

---

Source: [Claudary](https://claudary.paisolsolutions.com/skills/readme-673) · https://claudary.paisolsolutions.com