RemoteLeads Logo
Sign Up Sign In

AI Rater Guidelines Writer (Linguist / Instructional Designer)

1 month, 2 weeks ago
United States
Contract
Mid Level
AI (Artificial Intelligence)
Artificial Intelligence and Machine Learning
Generative AI
Apply Now
Weekday

Weekday

Weekday helps companies hire engineers who are vouched by other software engineers, enabling passive income for engineers. They offer services like drafting outreach messages, shortlisting candidates, and conducting reference checks. Backed by Y Combin...

Construction & Engineering
11-50
Founded 2020
View All Jobs 107

Description

  • Translate complex and ambiguous program requirements into clear rater guidelines.
  • Design evaluation rubrics and scoring frameworks for consistent decision-making across AI tasks.
  • Review existing documentation to identify ambiguities, inconsistencies, contradictions, and coverage gaps.
  • Revise guidelines to improve clarity, usability, and reviewer confidence.
  • Convert subject-matter requirements from finance, retail, insurance, legal, and sports into structured evaluation instructions.
  • Collaborate with research teams, program managers, and subject matter experts to maintain consistency across guideline sets.
  • Develop documentation for edge cases, exceptions, and complex evaluation scenarios.
  • Minimize reviewer escalation by creating precise and actionable guidance.
  • Support high-quality human evaluation processes for AI training and model assessment.

Requirements

  • Minimum 3 years of professional experience in linguistics, instructional design, technical writing, AI content development, or a related field.
  • Direct experience creating, refining, or maintaining evaluation guidelines, rating rubrics, or reviewer instructions in Generative AI, RLHF, human evaluation, or AI data annotation environments.
  • Demonstrated ability to work across multiple subject areas and convert domain knowledge into structured documentation.
  • Proven experience resolving ambiguity and improving written specifications with measurable results.
  • Strong analytical thinking with exceptional attention to detail.
  • Outstanding written communication skills with the ability to explain nuanced concepts clearly and consistently.
  • Demonstrated career progression and professional growth.
  • Ability to commit reliably to 35+ hours per week during weekdays.
  • Experience supporting AI model training, evaluation, RLHF, or large-scale annotation programs (preferred).
  • Familiarity with structured documentation standards, quality assurance methodologies, and guideline governance (preferred).

Benefits

  • Compensation of $45-$65 per hour.
  • Fully remote engagement with flexible working arrangements.
  • Weekly payments based on approved work completed.
  • Independent contractor engagement.
  • Opportunity to influence the quality and consistency of AI training data across multiple industries.
  • Reasonable accommodations available throughout the application and engagement process.

Interested in this position?

Apply directly on the company website

Apply Now

Similar Roles

Mindrift.ai: Be the “I” in AI

Freelance Agent Evaluation Engineer

Mindrift.ai: Be the “I” in AI Internet Software & Services

Mindrift is seeking experienced software developers for a project-based, remote opportunity creating and evaluating realistic coding tasks and tests for AI coding agents used by leading technology companies.

Poland Freelance Senior AI (Artificial Intelligence) QA Engineer
Up to $83k
Cybersecurity Docker FastAPI JavaScript Kafka Machine Learning PostgreSQL Python React Redis TypeScript
14 hours, 49 minutes ago
Prolific

AI Trainer - Fluent Kazakh Speaker - Stuttgart, Germany

Prolific 51-250 Professional Services

Prolific is seeking fluent Kazakh speakers to evaluate AI-generated text for naturalness, authenticity, tone, and cultural fit, helping improve AI language models.

Germany Freelance Entry Level AI (Artificial Intelligence)
$42k-$54k
NLP
2 days, 13 hours ago
Prolific

AI Trainer - Fluent Kazakh Speaker - Leipzig, Germany

Prolific 51-250 Professional Services

Prolific is seeking fluent Kazakh speakers to evaluate AI-generated text for naturalness, authenticity, tone, and cultural accuracy as part of human-data projects that improve language models.

Germany Freelance Entry Level AI (Artificial Intelligence)
$42k-$54k
2 days, 13 hours ago
Prolific

AI Trainers - United States

Prolific 51-250 Professional Services

Prolific is seeking U.S.-based people from diverse professional and educational backgrounds to participate in academic and applied research studies that train and evaluate AI systems and inform products, services, and research.

United States Freelance Mid Level AI (Artificial Intelligence)
Up to $37k
LLM
2 days, 13 hours ago

You're on a roll! Sign up now to keep applying.

Sign Up

Already have an account? Log in

Used by 14,729+ remote workers