Senior Software Engineer - AI Agent Evaluation

4 weeks ago
Contract
Senior
Software Development
Mindrift.ai: Be the “I” in AI

Mindrift.ai: Be the “I” in AI

Join 10,000+ experts earning $15-50/hr training AI models remotely. Flexible freelance work, weekly payments. No AI experience required. Apply in 5 minutes.

Internet Software & Services

Description

  • Build realistic virtual developer environments with codebase, infrastructure, and supporting context such as tickets, documentation, and conversations.
  • Design tasks from intermediate states of those environments and define clear success criteria for what counts as solved.
  • Write functional and integration tests that validate agent solutions and distinguish correct approaches from incorrect ones.
  • Review agent solutions, analyze failures, and refine tasks and tests based on QA feedback.
  • Iterate on evaluation materials to make the dataset fair, robust, and challenging for advanced AI models.
  • Work within the project-based workflow by completing assigned tasks and submitting them by the deadline.
  • Guide and evaluate the agent's work rather than writing most of the code from scratch.

Requirements

  • 5+ years of experience in software development.
  • Experience with the core stack: Python (FastAPI), JavaScript/TypeScript (React), Docker, Postgres, Kafka, and Redis.
  • Experience writing tests, including functional and integration tests.
  • English proficiency at B2 level or higher.
  • CV must be submitted in English.
  • Ability to work on a project-based engagement rather than permanent employment.
  • Understanding of how to assess real-world developer tasks and identify failure modes in AI-generated solutions.
  • Ability to complete tasks estimated at around 20 hours each, subject to complexity and deadline requirements.

Benefits

  • Up to $50/hr equivalent compensation, depending on level and pace.
  • Flexible schedule with tasks completed on your own time.
  • Project-based paid work with estimated 20-hour task assignments.
  • Opportunity to work on cutting-edge AI evaluation projects for leading tech companies.

Interested in this position?

Apply directly on the company website

Apply Now

Similar Roles

Alpheratz Project - Indonesian (Indonesia) Translation Quality Rater

Welo Global Professional Services

Experienced language professionals are needed for an ongoing Indonesian (Indonesia) customer service content project focused on reviewing and refining machine-translated materials for accuracy and naturalness.

17 hours, 41 minutes ago

Senior Software Engineer

Teachable 51-250 Internet Software & Services

Teachable is hiring a Senior Software Engineer to help improve software delivery and deployment processes for its creator-education platform.

AWS CI/CD Docker Go Kubernetes Microservices Ruby Terraform
1 day, 16 hours ago

Alpheratz Project - Indonesian (Indonesia) Translation Quality Reviewer

Welo Global Professional Services

An ongoing language-quality role at a project supporting Indonesian (Indonesia) customer service content, focused on reviewing machine-translated materials and improving linguistic accuracy and consistency.

1 day, 17 hours ago

Senior Software Engineer ML - Contractor position

Janea Systems 11-50 Internet Software & Services

Janea Systems (USA) is seeking a Senior ML Engineer contractor to support remote, client-facing consulting engagements that take enterprise ML and LLM systems from prototype to production.

Apache Airflow AWS Azure CI/CD Dagster Datadog GCP Kubeflow Kubernetes LLM MLOps NLP Prefect Python PyTorch TensorFlow
4 days, 16 hours ago

You're on a roll! Sign up now to keep applying.

Sign Up

Already have an account? Log in

Used by 14,729+ remote workers