Model Behavior Engineer
Company websiteoverto.ne (Overtone, opens in new tab)
Base salary $160K – $190K
Employer-published annual base salary, USD. Advertised base range in New York City, NY
You apply on the employer's own site. Six Figure Work does not handle applications.
- LocationNew York City, NY
- WorkplaceOn-site
- Status at last checkSource accepting applications when checked
- Last checked (1 h ago) · observed since
About this job
Your Role
Your job is to work with our relationship scientist and eval engineer to build and execute an eval strategy that keeps our LLM content high quality at scale.
Our relationship scientist defines what good looks like across our content surfaces and drafts golden examples. They also label for LLM judges that need subject matter expertise or calls on tone and voice.
Our eval engineer builds the infrastructure to power our content monitoring and iteration: human labeling tools, automated judge calibration tools, Braintrust experiments, and production monitoring dashboards.
You are the hands-on operator of this system: the eval engineer builds the tools, the relationship scientist defines goldens, and you use both every day to find and fix problems in our content.
The team is early, so you'll have real influence over how this workflow gets built.
What You’ll Do
Prompt iteration and engineering. Fix straightforward errors yourself: when our judges flag a problem in production content, diagnose the failure, change the prompt, and confirm the fix with evals before it ships. When a fix depends on subject matter expertise, partner with our relationship scientist. You’ll draft prompts for our content surfaces and our judge rubrics.
Eval + content engineering strategy. Design our infrastructure for evals and content iteration, including labeling, judge calibration and production quality monitoring and keep evolving it as the product grows.
Define what we measure. Work with our relationship scientist to decide which content features to measure, from proactive baseline quality checks to reactive checks driven by errors we find. Set up automated error analysis so new failure patterns surface on their own, not only when someone spots them.
Labeling strategy and tools. Own how we label and calibrate our judges, including the tools and workflows that scale our relationship scientist's judgment as efficiently as possible. Make sure they label only where their expertise is needed. Everywhere else, label production outputs and match assessments yourself or through outsourced labelers, so we can check that our judges agree with human judgment.
Synthetic testing. Design synthetic tests that tell us whether a new prompt is working before it reaches human testing, and review the results.
Production review and prioritization. Read real production outputs across our core surfaces and spot quality issues. Prioritize what to fix first across everything our judges, automated error analysis and manual reviews surface.
Who You Are
We're open to a very product-minded engineer, or a highly technical product person:
3+ years in product management or engineering, including shipping LLM-powered features to real users
Hands-on experience with evals: writing rubrics, building or running LLM-as-judge setups, and aligning judges against human labels
Hands-on prompt engineering experience: you can diagnose why an output went wrong, rewrite the prompt, and show with evals that the fix worked
Comfortable writing code when the work calls for it, like scripts for error analysis or changes to our eval pipeline
A detail-oriented, editorial eye for tone, structure and voice in AI-written content
Eagerness to dive in and read lots of AI outputs by hand
Great to have:
Experience with Braintrust or similar eval tooling
Experience running human labeling programs
Experience with conversational or voice AI products
Interest in relationships, psychology or matchmaking
Compensation & Benefits
Salary of $160k to $190k. The offer within that range depends on experience and skills. This position will receive meaningful stock options with considerable upside potential.
We provide standard benefits such as:
Medical, dental and vision insurance
401(k)
Commuter benefits, daily lunch and other perks
Paid time off, holidays and sick leave
Our Core Qualities
We look for these qualities in everyone we hire at Overtone.
Curiosity: You seek to understand before you decide. You ask thoughtful questions, test assumptions, and stay open to changing your mind as new information emerges.
Craft: You care deeply about quality and coherence. You simplify complexity, exercise good judgment, and take pride in work that feels intentional, thoughtful, and well-made.
Collaboration: You engage with others honestly and generatively. You challenge ideas with respect, listen deeply, and help synthesize perspectives into clearer, better outcomes.
Care: You take responsibility for people and outcomes. You act with integrity, follow through on commitments, and protect trust—with users, teammates, and the work itself.
About Overtone
Overtone is a modern matchmaking company, from the creator of Hinge. We’re building a voice- and audio-forward service, enabled by AI, grounded in relationship science. There’s no profiles or swiping. We get to know people in their own voice, then make thoughtful introductions, one at a time, to people we think are worth meeting. The service is in beta-testing and not yet open to the public. We’re building a small team of people passionate about relationships and human connection, to work together in person in our NYC office (5 days a week in Flatiron).
Equal Opportunity
Overtone is an equal opportunity employer. We consider all qualified applicants without regard to race, color, religion, creed, sex, pregnancy, sexual orientation, gender identity or expression, national origin, ancestry, age, disability, genetic information, marital or partnership status, caregiver status, veteran or military status, or any other status protected by law. We will consider qualified applicants with arrest and conviction records in a manner consistent with the New York City Fair Chance Act and other applicable law. If you need an accommodation at any point in the application or interview process, email [email protected] and we'll work with you.
Source: Overtone on Ashby · View original posting (opens in new tab)
How this pay compares
Base floor $160,000. The employer's range is $30,000 wide, 19% of the floor. First observed today.
This listingModel Behavior Engineer$160K – $190K
Head of Marketing$250K – $300K
Chart as a table
| Role | Published base |
|---|---|
| Model Behavior Engineer (this listing) | $160K - $190K |
| Head of Marketing | $250K - $300K |
Observed changes
No changes to the pay or listing since we first observed this role on Nov 20, 2026 (today).
- First observed on the employer's board, already publishing a qualifying base salary
- Observed since
- Employer-stated posting date
- 2026-11-20
Where this pay comes from
Location as the employer wrote it: New York City
USD 160,000 – 190,000 · 1 YEAR · Salary
- Method
- Structured pay field published by the employer
- Checked
- Rule set
- us-base-v2
The employer's job board publishes this pay as a dedicated data field. We read the field directly; no interpretation was needed.
What we verified
- The pay figures come from the employer's own listing, quoted exactly above.
- The pay is an annual base salary in US dollars with a minimum of at least $100,000.
- The role was publicly listed on the employer's job board when last checked.
We do not verify what the employer will actually offer, whether the employer is still recruiting for the role, or anything beyond what the employer published. Full methodology.
