Research Engineer, Evaluations
Build the evaluations that tell us, and everyone else, whether reasoning is real: starting with Step Validity.
- Research
- San Francisco (hybrid) or remote in the US
- Full-time
- $170k–$250k a year
What you will do
- Build and maintain evaluation suites, including Step Validity, faithfulness probes and calibration tests.
- Run public benchmarks with published settings so results can be reproduced.
- Build the tooling that runs our safety framework's pre-release evaluations.
- Prepare evaluation sets and grading code for public release.
What we look for
- Strong software engineering in Python, with tests and reproducible pipelines.
- Experience evaluating machine learning models and an eye for leakage and flawed metrics.
- Comfort working with large datasets and model APIs.
Nice to have
- Experience building or maintaining a public benchmark.
- Background in statistics or experimental design.
You do not need to meet every point to apply. We care about what you have built or proved more than titles.
Pay and benefits
The base salary range for this role is $170k–$250k a year. It is the range we expect in good faith to pay; where an offer falls depends on experience and skills. Offers also include equity.
- Compute budget for your own research questions
- Publish by default, with credit to the people who did the work
- Remote or hybrid, with team weeks in person
- Health insurance and generous parental leave
How to apply
Email contact@theoratix.ai with the role in the subject line, a link to your CV or LinkedIn, and something you have built or proved that you are proud of.
- Introductory call. A 30-minute conversation about your work and what you want to do next.
- Technical deep dive. Walk us through something you built or proved, and the decisions behind it.
- Work session. A paid, realistic problem, worked on together rather than set as homework.
- Conversation with the founder. Mission, values and the questions you have for us.
Theoratix Inc. is an equal opportunity employer. We welcome applicants of every background and do not discriminate on the basis of race, color, religion, sex, sexual orientation, gender identity, national origin, age, disability, veteran status or any other protected characteristic. We will consider qualified applicants with arrest and conviction records in accordance with the San Francisco Fair Chance Ordinance. If you need an accommodation during the hiring process, email us.