RemoteLeads Logo
Sign Up Sign In

Technology AI Evaluation Expert

1 week, 3 days ago
United States
Contract
Mid Level
AI (Artificial Intelligence)
Data Science and Analytics
Apply Now
Weekday

Weekday

Weekday helps companies hire engineers who are vouched by other software engineers, enabling passive income for engineers. They offer services like drafting outreach messages, shortlisting candidates, and conducting reference checks. Backed by Y Combin...

Construction & Engineering
11-50
Founded 2020
View All Jobs 43

Description

  • Create realistic, multi-step benchmark tasks based on professional technology workflows.
  • Develop evaluation challenges using technical specifications, architecture documents, API documentation, codebases, web research, and code execution.
  • Define clear expected outputs and objective evaluation criteria for each task.
  • Write ground-truth solutions and structured scoring rubrics.
  • Design tasks that assess reasoning, technical understanding, instruction following, and output quality.
  • Maintain high standards of technical accuracy, clarity, and reproducibility.
  • Apply domain expertise from software engineering, data science, or analytics to create authentic evaluation scenarios.
  • Collaborate with research teams to improve benchmark quality and consistency.
  • Continuously refine tasks based on project feedback and evolving evaluation requirements.

Requirements

  • Minimum 3 years of hands-on professional experience in software engineering, data science, or data analytics.
  • Strong understanding of technical documentation, software development workflows, and engineering best practices.
  • Experience working with codebases, APIs, technical specifications, or system architecture documentation.
  • Excellent analytical thinking and problem-solving skills.
  • Strong written communication with the ability to create clear technical instructions and evaluation criteria.
  • Ability to work independently while maintaining high standards of accuracy and consistency.
  • Experience designing evaluation standards, scoring rubrics, or ground-truth solutions is preferred.
  • Comfort working with technical materials such as technical specs, architecture diagrams, and workplace resources.
  • Ability to commit 15–20 hours per week.
  • Visa sponsorship is not available for this opportunity.

Benefits

  • Compensation of $60–$75 per hour.
  • Fully remote work completed on your own schedule.
  • Flexible working hours.
  • Independent contractor engagement.
  • Weekly payments processed through supported payment platforms.
  • Opportunity to work on intellectually challenging AI research projects.
  • Meaningful impact on AI benchmark quality and technical reasoning systems.

Interested in this position?

Apply directly on the company website

Apply Now

Similar Roles

Welo Global

Hydrus -Voice Contributor — English (US)

Welo Global Professional Services

Welo Data is hiring a freelance native-level English (US) speaker to record conversational customer-service audio for a remote AI voice project.

United States Freelance Entry Level AI (Artificial Intelligence) Customer Support
$100k-$100k
1 day, 13 hours ago
Crosslake

AI / ML Consultant

Crosslake 251-1K Capital Markets

Crosslake is hiring a US-based remote consultant to help technology and private equity clients evaluate, improve, and implement AI and software-related initiatives across the investment lifecycle.

United States Contract Lead AI (Artificial Intelligence)
Agile Computer Vision Generative AI LLM Machine Learning MLOps NLP Statistics
1 day, 13 hours ago
Welo Global

Hydrus -Voice Contributor — Spanish (Spain)

Welo Global Professional Services

Welo Data is hiring a freelance native-level Spanish (Spain) speaker to record conversational customer-service voice data for a remote conversational AI project.

Spain Freelance Entry Level AI (Artificial Intelligence) Customer Support
$83k-$83k
1 day, 13 hours ago
Mindrift.ai: Be the “I” in AI

Senior Python Engineer - AI Coding Agent Evaluation (Freelance)

Mindrift.ai: Be the “I” in AI Internet Software & Services

Mindrift is seeking an experienced software development specialist to create realistic coding-agent evaluation tasks and tests for project-based AI opportunities.

Costa Rica Ecuador Jamaica Uruguay Freelance Lead AI (Artificial Intelligence) Backend Engineer
Docker FastAPI JavaScript Kafka PostgreSQL Python React Redis TypeScript
2 days, 13 hours ago

You're on a roll! Sign up now to keep applying.

Sign Up

Already have an account? Log in

Used by 14,729+ remote workers