Type: Contract, hourly
Location: Remote
Hours: 20–30 per week
Pay: $20–30/hour, based on experience and language coverage
Start: Immediate
ABOUT THE ROLE
We evaluate how well advanced AI coding agents solve real engineering problems. An agent is given a real open source codebase inside a container and a hard task, then works on its own for 80 to 250 steps. A trajectory is the full record of that run — every command, result, and decision.
You will do two jobs, and you should expect either on any given day.
•Annotate — Read a trajectory nobody has looked at yet and judge it step by step.
• Review — Take an existing annotation, written by our AI tooling or another person, and confirm, correct, or reject it.
TASKS YOU'LL SEE
• Feature build — Add a working feature to a live library without breaking anything that already worked.
• Rebuild — Work out what a compiled tool does by running it, then rebuild it to match its output, exit codes, and file effects.
• Bug hunt — Find and fix twenty undocumented bugs across a dozen files with no test suite, then record what caused them.
Mostly Python and Go, with some Rust, C, and Ct+. A trajectory runs about 100 steps.
WHAT YOU JUDGE IN A TRAJECTORY
• Was the command right for the state the environment was actually in?
• Did the agent read the previous output correctly?
• Was the step wrong, or only inefficient — these are scored differently.
• Where did the run first go off course — usually earlier than where it visibly broke.
• Did the agent notice its own mistake and recover, or keep building on a false assumption?
• Did it game the grader instead of solving the task (e.g., weakening a test or hardcoding an expected value)?
WHAT WE NEED FROM YOU
• Experience — 2+ years in software engineering, DevOps, or site reliability, with real debugging in real codebases.
• Languages — Strong in Python or Go, and able to read a language you've never used.
• Linux — Comfortable with logs, running processes, build failures, and containers.
• Workflow — Everyday Git, diffs, pull requests, and issue tracking.
• Debugging — Able to work with no test suite and no error message pointing at the cause.
• Focus — Able to hold context across a long run, because step 74 can depend on step 12.
• Writing — Clear English, since every judgement needs an explanation another engineer can check.
...motivated, self-starter to join our award-winning news team as an Sports Reporte (MultiMedia Journalist). This is a terrific opportunity... ...news landscapeQualification/Requirements:A degree in journalism or a related field, or equivalent experienceLink to a demo reel...
...in Virginia's Blue Ridge Mountains seeks an Activity Guide to engage guests with a breadth of outdoor experiences, from hunting and fishing to equestrian activities and clay shooting. The role requires knowledge of resort offerings and the ability to supervise and adapt...
...Position: Elementary Teacher Location: Gillmore Elementary School Salary: State Salary Schedule Qualifications: Must have or be eligible for valid certification General: This is a 9-month position and includes a full benefit package. The Clarke County...
...Nurse Assistant Under the general supervision or direction of the Chancellor and the school nurse, the nurse assistant will carry out a variety of treatments and procedures necessary for providing health care services to meet the medical and physical needs of all students...
...Call Center Representative Customer Support Employment Type: Full-Time (Part-Time available after training) Compensation... ...clear goals, you may be a great fit. No industry-specific experience required. We provide paid training, ongoing coaching, and the...