RemoteLeads Logo
Sign Up Sign In

Freelance Agent Evaluation Engineer

6 days, 13 hours ago
Japan
Contract
Senior
AI (Artificial Intelligence)
Software Development
Docker FastAPI JavaScript Kafka PostgreSQL Python React Redis TypeScript
Apply Now
Mindrift.ai: Be the “I” in AI

Mindrift.ai: Be the “I” in AI

Join 10,000+ experts earning $15-50/hr training AI models remotely. Flexible freelance work, weekly payments. No AI experience required. Apply in 5 minutes.

Internet Software & Services
View All Jobs 65

Description

  • Build realistic developer environments that include a virtual company, codebase, infrastructure, and supporting context such as tickets, documentation, and conversations.
  • Design tasks from intermediate states of these environments and define clear criteria for what counts as a solved task.
  • Write tests that validate agent solutions, accepting all valid approaches and rejecting incorrect ones.
  • Review agent solutions, analyze failures, and refine tasks and tests based on QA feedback.
  • Iterate on evaluation setups until they are fair, robust, and challenging for AI agents.
  • Ensure tasks are solvable by an AI agent while still being sufficiently difficult to test model performance.

Requirements

  • 5+ years of experience in software development.
  • Experience with the core stack: Python (FastAPI), JavaScript/TypeScript (React), Docker, Postgres, Kafka, and Redis.
  • Experience writing functional and integration tests.
  • English proficiency at B2 level or higher.
  • Ability to work on a project-based basis rather than in permanent employment.
  • Ability to submit completed tasks by the required deadline.
  • Strong understanding of how coding models fail and how to create tasks that distinguish good from bad solutions.

Benefits

  • Up to $50/hr equivalent compensation, depending on level and pace.
  • Flexible schedule with work completed on your own time.
  • Project-based engagement with tasks estimated at around 20 hours each.
  • Payment upon completion of accepted tasks.

Interested in this position?

Apply directly on the company website

Apply Now

Similar Roles

Sapsol Technologies

Dynamics Great Plains Developer

Sapsol Technologies 51-250 Internet Software & Services

Microsoft Dynamics GP Developer needed to design, integrate, test, and support custom business applications and reporting solutions for a third-party-connected enterprise environment.

Canada Contract Senior Software Engineer
Azure C# Git Power BI REST API SOAP SQL Server Visual Studio
1 day, 13 hours ago
Welo Global

Hydrus -Voice Contributor — English (US)

Welo Global Professional Services

Welo Data is hiring a freelance native-level English (US) speaker to record conversational customer-service audio for a remote AI voice project.

United States Freelance Entry Level AI (Artificial Intelligence) Customer Support
$100k-$100k
1 day, 13 hours ago
Bounteous

SQL Developer + DBA (AWS & Azure Experience)

Bounteous 1K-5K Internet Software & Services

Bounteous is hiring an experienced SQL Developer + Database Administrator to manage and optimize production database environments across Azure SQL, AWS MySQL, and related applications in a fast-paced digital consultancy setting.

United States Contract Senior Database Administrator Software Engineer
AWS Entity Framework MySQL SQL Server
1 day, 13 hours ago
Crosslake

AI / ML Consultant

Crosslake 251-1K Capital Markets

Crosslake is hiring a US-based remote consultant to help technology and private equity clients evaluate, improve, and implement AI and software-related initiatives across the investment lifecycle.

United States Contract Lead AI (Artificial Intelligence)
Agile Computer Vision Generative AI LLM Machine Learning MLOps NLP Statistics
1 day, 14 hours ago

You're on a roll! Sign up now to keep applying.

Sign Up

Already have an account? Log in

Used by 14,729+ remote workers