pi-autoresearch

Autonomous optimization for pi

Give pi a metric. Let it run.

It tests ideas, measures every result, keeps improvements, and reverts the rest.

pi install npm:pi-autoresearch

Runs with your user permissions. Review the source and use a dedicated branch or worktree. Security guidance →

Star on GitHub 7,950

MIT licensed. Built for pi.

How it works

One goal. Many measured attempts.

Give pi a metric and a boundary. It explores one hypothesis at a time, keeps evidence, and lets the benchmark decide.

Start with a measurable goal

/autoresearch optimize unit test runtime, monitor correctness

Agents get clever. Benchmarks remain unimpressed. Only measured improvements survive.

Session running

Unit test runtime

Run 06
Baseline12.84s Best1.28s −90.0%
RunHypothesisMetricDecision
01Establish baseline12.84sbaseline
02Isolate worker pool9.62skeep
03Surely 12 threads is faster10.19srevert
04Stop rebuilding every test file4.08skeep
05Transform only the files that need it1.28skeep
06Remove one more suspiciously expensive thingmeasuring running
Next hypothesis is queued from what the last run learned..auto/log.jsonl

Correctness first

Faster is only a win if the code still works.

Add a checks file once. Every promising experiment must pass it before the change can survive.

.auto/checks.sh
#!/bin/bash
set -euo pipefail

pnpm test --run
pnpm typecheck
pnpm lint
01

Measure first

The benchmark records its result before correctness checks begin. The primary metric stays clean.

02

Checks have veto power

A failed test, type check, or lint blocks the keep and reverts the experiment.

03

Keep the evidence

The dashboard distinguishes regressions, crashes, and check failures so the next attempt can learn from them.

Read the configuration reference →

Durable by design

Agents forget. The experiment doesn't.

Every decision is written to the project. Restarts, context resets, and auto-compaction do not erase the work.

A fresh agent reads the session files and continues from evidence—not a fragile memory of the conversation.

.auto/prompt.md

The objective, constraints, scope, tried ideas, dead ends, and key wins. The handoff for the next agent.

.auto/ideas.md
  • Reuse transformed modules
  • Try isolated worker pools
  • Profile setup-file reloads

Promising ideas wait here instead of disappearing with the context window.

.auto/log.jsonl

An append-only record of every metric, commit, decision, description, and confidence score.

.auto/hooks/
before.shexperimentafter.sh

Fetch research before a run; persist learnings or notify afterward.

Read the hook contract →

Evidence, not optimism

A faster number isn't always a win. Know when it is probably noise.

After three runs, pi-autoresearch compares the best improvement with the session's noise floor using Median Absolute Deviation.

Best improvement confidence 2.7× Likely real
2.0× and above Likely real
1.0–2.0× Above noise, but marginal
Below 1.0× Within the noise floor

Confidence is advisory. It encourages re-runs when a result is uncertain; it never discards work for you.

Browser dashboard

Watch the numbers move. Share the evidence.

/autoresearch export opens a live dashboard with the full experiment history. This example uses results from the Shopify Engineering case study.

Autoresearch results

Polaris visual regression build

Complete · 12 experiments
Baseline → Best 19.1s → ≈6.69s
Improvement 65% faster
Runs 12
Kept 9 / 12
Normalized build runtime −65%
Published autoresearch result compared with its baseline The selected Shopify result falls from a normalized baseline of 100 to the published final result.
RunStatusRuntime indexΔ baselineExperiment
01keep100—Full VRT build baseline
02keep91−9%Skip declaration emit for Storybook
03discard94−6%Increase transform workers
04keep82−18%Compile Storybook from source
05keep76−24%Limit TypeScript transform scope
06discard79−21%Disable source maps globally
07keep67−33%Transform matched files only
08keep59−41%Reuse resolved configuration
09discard62−38%Cache all module resolution
10keep48−52%Skip redundant IIFE bundle
11keep41−59%Narrow component entry points
12keep35−65%Transform only 105 matched files
Shopify Engineering case study Autoresearch isn't just for training models Read the story ↗

In the wild

Real codebases. Results people shared publicly.

From Shopify Liquid to a new canvas rendering engine, teams are using the loop to surface improvements nobody would schedule by hand.

Quoted from public posts on X. Dates link to the originals. Read the full Shopify story in Autoresearch isn't just for training models.

Domain agnostic

If you can measure it, the loop can improve it.

The extension provides the experiment infrastructure. Skills bring the knowledge of what to try.

Target Metric Example command
Test speed Seconds ↓ pnpm test
Bundle size Kilobytes ↓ pnpm build && du -sb dist
LLM training Validation loss ↓ uv run train.py
Build speed Seconds ↓ pnpm build
Lighthouse Performance ↑ lighthouse http://localhost:3000

A clean ending

Experiments are messy. Your review shouldn't be.

/skill:autoresearch-finalize groups compatible wins and creates independent branches from the merge-base.

Each branch is focused, reviewable, and carries its measured improvement in the commit history.

Stay close or walk away

Watch the loop at the level you need.

Live widget

Every result stays visible above the pi editor while you work.

Fullscreen dashboard

Open a scrollable terminal view with /autoresearch dashboard. Optional shortcuts are opt-in.

Browser export

Open a live chart and shareable results view with /autoresearch export.

Extension points

The core stays small. Hooks make the loop yours.

Optional scripts run before or after each iteration without coupling custom behavior to the agent.

Fetch external research Keep a learnings journal Send native notifications Detect idea thrashing Rotate hypotheses Tag meaningful wins

Start with one metric

Try an idea. Measure it.
Keep only what works.

pi install npm:pi-autoresearch
View on GitHub View on npm