Senior Python Engineer - AI Coding Agent Evaluation (Freelance)

1 week, 5 days ago
Freelance
Lead
Software Development
Mindrift.ai: Be the “I” in AI

Mindrift.ai: Be the “I” in AI

Join 10,000+ experts earning $15-50/hr training AI models remotely. Flexible freelance work, weekly payments. No AI experience required. Apply in 5 minutes.

Internet Software & Services

Description

  • Build realistic virtual developer environments with codebase, infrastructure, tickets, docs, and conversation context.
  • Design tasks from intermediate environment states and define what counts as a solved task.
  • Write tests that verify agent solutions and accept all valid approaches while rejecting incorrect ones.
  • Review agent solutions, analyze failures, and refine tasks and tests based on QA feedback.
  • Ensure evaluation tasks are challenging, solvable, and fair for AI coding agents.
  • Iterate on prompts, criteria, and test coverage to improve robustness of the dataset.

Requirements

  • 8+ years of experience in software development.
  • Experience with Python, including FastAPI.
  • Experience with JavaScript/TypeScript, including React.
  • Experience with Docker, Postgres, Kafka, and Redis.
  • Experience writing functional and integration tests.
  • English proficiency at B2 level or higher.
  • Ability to work project-based and submit tasks by the deadline.
  • CV must be submitted in English.

Benefits

  • Up to $200/hr equivalent compensation, depending on level and pace.
  • Project-based work with estimated 30 hours per task.
  • Flexible schedule with self-directed working hours.
  • Work on AI evaluation projects for leading tech companies.

Interested in this position?

Apply directly on the company website

Apply Now

Similar Roles

SWE Fellow - Human Frontier Collective (UK)

Scale AI 251-1K Diversified Consumer Services

The Human Frontier Collective Fellowship at Scale is a fully remote contract role supporting AI research by designing, evaluating, and interpreting advanced generative AI systems with interdisciplinary experts and partner labs.

C++ Generative AI Java JavaScript Machine Learning Python Rust Swift
20 hours, 43 minutes ago

Norwegian Post-Editor | Technology Company

OLIVER 1K-5K Media

A tech company is seeking a Norwegian Post-Editor to support MTPE projects by refining marketing copy and product descriptions for AI-driven content development.

21 hours, 14 minutes ago

Staff Python Engineer, Document App Track

PandaDoc 251-1K Internet Software & Services

PandaDoc is hiring a Staff Backend Engineer for its Document App Track to lead the evolution of the core document domain that powers document creation, editing, and management across the platform.

Agile AWS DDD Django Docker Grafana Kafka Kubernetes Memcached Microservices NATS PostgreSQL Python RabbitMQ Redis SQL
21 hours, 14 minutes ago

Staff Python Engineer, Document App Track

PandaDoc 251-1K Internet Software & Services

PandaDoc is hiring a Staff Backend Engineer for its Document App Track to lead the evolution of the core document domain that powers document creation, editing, and management.

Agile AWS DDD Django Docker Grafana Kafka Kubernetes Memcached Microservices NATS PostgreSQL Python RabbitMQ Redis SQL
21 hours, 29 minutes ago

You're on a roll! Sign up now to keep applying.

Sign Up

Already have an account? Log in

Used by 14,729+ remote workers