RemoteLeads Logo
Sign Up Sign In

Freelance Agent Evaluation Engineer

17 hours, 20 minutes ago
Brazil
Freelance
Senior
AI (Artificial Intelligence)
Artificial Intelligence and Machine Learning
Cybersecurity Docker FastAPI JavaScript Kafka Machine Learning NLP PostgreSQL Python React Redis TypeScript
Apply Now
Mindrift.ai: Be the “I” in AI

Mindrift.ai: Be the “I” in AI

Join 10,000+ experts earning $15-50/hr training AI models remotely. Flexible freelance work, weekly payments. No AI experience required. Apply in 5 minutes.

Internet Software & Services
View All Jobs 10

Description

  • Build realistic developer environments with codebases, infrastructure, and supporting context such as tickets, documentation, and conversations.
  • Design tasks from intermediate environment states and define clear criteria for when a task is considered solved.
  • Write evaluation tests that validate agent solutions and distinguish correct approaches from incorrect ones.
  • Review agent solutions, analyze failures, and refine tasks and tests based on QA feedback.
  • Iterate on prompts, task design, and test coverage to ensure the evaluation is fair and robust.
  • Assess whether tasks are solvable by AI agents and challenging enough to evaluate model performance.
  • Help improve the dataset used to benchmark AI coding agents in real-world developer workflows.

Requirements

  • 5+ years of experience in software development.
  • Strong experience with Python (FastAPI), JavaScript/TypeScript (React), Docker, Postgres, Kafka, and Redis.
  • Experience writing functional and integration tests.
  • English proficiency at B2 level or higher.
  • A Master’s degree in Computer Science, Software Engineering, Data Science / Data Analytics, Artificial Intelligence / Machine Learning, Computational Linguistics / NLP, Information Systems, or a related field.
  • A Bachelor’s degree is accepted with 5 years of relevant experience.
  • At least 3 years of professional experience in a related role or domain, specifically QA automation/testing or cybersecurity.
  • Ability to submit a CV in English.

Benefits

  • Paid per accepted task, with earnings up to the equivalent of $30/hr depending on qualification tier and speed.
  • Remote, part-time freelance project work that fits around other professional or academic commitments.
  • Opportunity to work on advanced AI projects and strengthen your portfolio.
  • Chance to influence how future AI models understand and communicate in your field of expertise.

Interested in this position?

Apply directly on the company website

Apply Now

Similar Roles

kake

Chemist with Python Proficiency - AI Trainer

kake 1-10 Internet Software & Services

A remote project-based role with a chemistry-focused AI development team to create and evaluate computational chemistry tasks for frontier AI models.

United States Contract Junior AI (Artificial Intelligence)
Linux Python
16 hours, 35 minutes ago
Newsela

Contractor: Senior Quality Engineer

Newsela 251-1K Diversified Consumer Services

Newsela is hiring a Senior Full-Stack Quality Engineer to ensure reliable, high-quality software for its AI-powered K–12 education products.

Chile Argentina Costa Rica Brazil Contract Senior QA Engineer SDET (Software Development Engineer in Test)
Agile AWS CI/CD CircleCI Datadog GCP Git GitHub Actions JavaScript Jest JIRA JMeter K6 LLM Locust Mocha New Relic Playwright Python Scrum Selenium TypeScript
16 hours, 35 minutes ago
kake

Biologist with Python Proficiency - AI Trainer

kake 1-10 Internet Software & Services

Biology professionals with Python expertise will help evaluate and improve frontier AI models on project-based computational biology tasks for a remote, pipeline-style role.

United States Contract Junior AI (Artificial Intelligence)
Linux Python
16 hours, 50 minutes ago
kake

Civil Engineer with Python Proficiency - AI Trainer

kake 1-10 Internet Software & Services

Civil engineering professionals with Python experience are needed for a remote, project-based role supporting AI model evaluation through domain-specific simulation tasks.

United States Contract Junior AI (Artificial Intelligence)
Linux Python
17 hours, 5 minutes ago

You're on a roll! Sign up now to keep applying.

Sign Up

Already have an account? Log in

Used by 14,729+ remote workers