Testing methodology

# Scoring Rubric v0.1: "Tested, not listed"

Every tool in a use case runs the **same fixed tasks** on a clean account, the cheapest paid tier (or free tier if it covers the tasks), the default model, and a fresh project. Run each task **3 times** and record the median. Log the prompt, transcript, time, cost and screenshots to `runs/<tool>/<date>/`.

## Weighted score (0-100)
| Criterion | Weight | How measured |
|---|---|---|
| Task success | 40% | Per task: 0 = fail, 0.5 = partial (needs a manual fix), 1 = pass on the automated acceptance check. Average across tasks. |
| Output quality | 15% | Lint/tests pass, no hard-coded secrets, accessibility (Lighthouse a11y) or code review checklist, scored 1-5 |
| Speed | 10% | Median wall-clock time vs the use-case median: faster = 5, ≥2x slower = 1 |
| Cost to complete | 15% | $ spent (credits/tokens/plan pro-rata) for the full suite, scaled against the cheapest = 5 |
| Ease / onboarding | 10% | Minutes from sign-up to first result, plus count of human interventions, scored 1-5 |
| Ecosystem & lock-in | 10% | Code export, Git sync, self-host/open source, integrations, scored 1-5 |

Score = 100 × Σ(weight × normalised criterion). Ties go to the lower cost. Re-test every 60 days or on a major release. Each page shows the tested date and version.

## Fixed benchmark tasks per use case
1. **AI coding IDEs/agents** (Cursor, Windsurf, Claude Code, Codex...): T1 add a feature to a seeded 2k-LOC repo with tests; T2 fix 3 seeded bugs; T3 refactor a module without breaking tests; T4 write tests to 80% coverage.
2. **App builders** (Lovable, Bolt, v0, Replit...): T1 todo app with auth and persistence; T2 landing page from a fixed brief; T3 CRUD dashboard on a supplied CSV; T4 deploy to a live URL.
3. **Agent frameworks** (LangGraph, CrewAI...): T1 research agent with web search plus a cited summary; T2 a multi-agent plan→execute→review loop; T3 a tool-calling agent with retries; measure LOC and setup time.
4. **Workflow automation** (n8n, Zapier, Make...): T1 RSS → LLM summary → email; T2 webhook → classify → Slack/Sheet; T3 an AI agent node with 2 tools.
5. **Code review & testing**: run on a seeded PR with 10 planted issues (security, logic, style); score recall and precision of findings.
6. **Deploy & backend**: deploy the reference app plus a DB; measure time-to-URL, cold start and free-tier limits.
7. **Agent infra (memory/browser/observability)**: plug into the reference agent; measure integration time, a task-success uplift and a trace completeness check.
8. **Models & APIs for coding**: run a fixed 20-problem coding set via the API; record pass@1, latency, and $/suite.

## Integrity rules
- Rankings are never sold. A paid fast-track only changes *when* a tool is tested, never its score.
- Affiliate status is disclosed on every page, and having one doesn't change the score.
- Publish the raw run logs.