RemoteLeads Logo
Sign Up Sign In

Senior Python Engineer - AI Coding Agent Evaluation (Freelance)

1 week, 4 days ago
United States
Contract
Lead
AI (Artificial Intelligence)
Software Development
Docker FastAPI JavaScript Kafka PostgreSQL Python React Redis TypeScript
Apply Now
Mindrift.ai: Be the “I” in AI

Mindrift.ai: Be the “I” in AI

Join 10,000+ experts earning $15-50/hr training AI models remotely. Flexible freelance work, weekly payments. No AI experience required. Apply in 5 minutes.

Internet Software & Services
View All Jobs 20

Description

  • Build realistic developer environments with codebase, infrastructure, and development context such as tickets, documentation, and conversations.
  • Design tasks from intermediate environment states, including prompts and clear definitions of what counts as a solved task.
  • Write tests that verify AI agent solutions and correctly accept valid approaches while rejecting incorrect ones.
  • Review agent solutions, analyze failures, and iterate on tasks and tests based on QA feedback.
  • Refine evaluation criteria to ensure the benchmark is fair, robust, and able to distinguish strong solutions from weak ones.
  • Assess whether tasks are realistically solvable by an AI agent within the simulated environment.

Requirements

  • 8+ years of experience in software development.
  • Experience with Python (FastAPI).
  • Experience with JavaScript/TypeScript (React).
  • Experience with Docker, Postgres, Kafka, and Redis.
  • Experience writing functional and integration tests.
  • English proficiency at B2 level or higher.
  • Submit CV in English.
  • Availability to complete project tasks that are estimated at about 30 hours each and submitted by deadline.

Benefits

  • Up to $200/hr equivalent compensation, depending on level and pace.
  • Project-based work with a self-directed schedule.
  • Ability to choose when and how to work.
  • Payment for completed tasks upon acceptance.
  • Opportunity to work on AI evaluation for leading tech companies.

Interested in this position?

Apply directly on the company website

Apply Now

Similar Roles

Scale AI

SWE Fellow - Human Frontier Collective (UK)

Scale AI 251-1K Diversified Consumer Services

The Human Frontier Collective Fellowship at Scale is a fully remote contract role supporting AI research by designing, evaluating, and interpreting advanced generative AI systems with interdisciplinary experts and partner labs.

United Kingdom Contract Senior AI (Artificial Intelligence) Software Engineer
C++ Generative AI Java JavaScript Machine Learning Python Rust Swift
20 hours, 54 minutes ago
OLIVER

Norwegian Post-Editor | Technology Company

OLIVER 1K-5K Media

A tech company is seeking a Norwegian Post-Editor to support MTPE projects by refining marketing copy and product descriptions for AI-driven content development.

Anywhere Freelance Mid Level AI (Artificial Intelligence) Translator
21 hours, 24 minutes ago
PandaDoc

Staff Python Engineer, Document App Track

PandaDoc 251-1K Internet Software & Services

PandaDoc is hiring a Staff Backend Engineer for its Document App Track to lead the evolution of the core document domain that powers document creation, editing, and management across the platform.

Ukraine Contract Lead Backend Engineer
$0k-$1k
Agile AWS DDD Django Docker Grafana Kafka Kubernetes Memcached Microservices NATS PostgreSQL Python RabbitMQ Redis SQL
21 hours, 24 minutes ago
PandaDoc

Staff Python Engineer, Document App Track

PandaDoc 251-1K Internet Software & Services

PandaDoc is hiring a Staff Backend Engineer for its Document App Track to lead the evolution of the core document domain that powers document creation, editing, and management.

Portugal Contract Lead Backend Engineer
$372k-$556k
Agile AWS DDD Django Docker Grafana Kafka Kubernetes Memcached Microservices NATS PostgreSQL Python RabbitMQ Redis SQL
21 hours, 39 minutes ago

You're on a roll! Sign up now to keep applying.

Sign Up

Already have an account? Log in

Used by 14,729+ remote workers