autoresearch

Autoresearch is an autonomous iterative experimentation skill designed for programming tasks that have a measurable outcome. Instead of performing one-off code changes, the skill establishes a structured optimization loop where the user defines a goal, specifies how success is measured, and sets clear scope and constraint boundaries. Once configured, the agent repeatedly modifies code, runs experiments, measures results, and determines whether to keep or revert changes. The process is inspired by Karpathy's autoresearch concept and extends it beyond machine learning workflows to general software engineering and performance optimization tasks.

The skill emphasizes disciplined experimentation and reproducibility. Before any experimentation begins, it guides the user through an interactive setup phase to define optimization goals, metric commands, extraction methods, editable file scopes, and operational constraints. During execution, the system establishes a baseline measurement, commits every experiment attempt to git for safe rollback, records experiment results in a TSV log, and automatically reverts changes that do not improve the selected metric. The workflow is designed to run autonomously after setup, ensuring continuous experimentation without requiring repeated user confirmation.

This skill is intended for developers, performance engineers, and technical teams working on optimization-focused programming tasks. Typical use cases include improving execution speed, reducing memory usage, optimizing benchmark scores, decreasing build times, increasing throughput, and refining automated test outcomes. It is best suited for projects already managed with git and environments where terminal access is available. The skill is not intended for simple bug fixes, code reviews, or tasks that lack a measurable metric.