RemoteLeads Logo
Sign Up Sign In

Senior Python Engineer - AI Coding Agent Evaluation (Freelance)

1 week, 4 days ago
United States
Freelance
Lead
AI (Artificial Intelligence)
Software Development
Docker FastAPI JavaScript Kafka PostgreSQL Python React Redis TypeScript
Apply Now
Mindrift.ai: Be the “I” in AI

Mindrift.ai: Be the “I” in AI

Join 10,000+ experts earning $15-50/hr training AI models remotely. Flexible freelance work, weekly payments. No AI experience required. Apply in 5 minutes.

Internet Software & Services
View All Jobs 20

Description

  • Build realistic developer environments with codebases, infrastructure, and supporting context such as tickets, docs, and conversations.
  • Design tasks from intermediate states of those environments, including prompts and clear definitions of what counts as a solved task.
  • Write tests that verify agent solutions, accepting valid approaches and rejecting incorrect ones.
  • Review agent solutions, analyze failures, and refine tasks and tests based on QA feedback.
  • Iterate on evaluation criteria to make assessments fair, robust, and challenging for AI coding agents.
  • Ensure tasks are solvable by an AI agent within realistic developer scenarios.
  • Assess whether task setups expose meaningful differences between strong and weak model outputs.

Requirements

  • 8+ years of experience in software development.
  • Experience with Python, including FastAPI.
  • Experience with JavaScript/TypeScript, including React.
  • Experience with Docker, Postgres, Kafka, and Redis.
  • Experience writing functional and integration tests.
  • English proficiency at B2 level or higher.
  • Submit CV in English and indicate English proficiency level.
  • Ability to work on a project-based, non-permanent engagement.

Benefits

  • Up to $200/hr equivalent compensation, depending on level and pace.
  • Flexible schedule with no fixed work hours.
  • Estimated 30 hours of work per task, depending on complexity.
  • Project-based paid opportunities with leading tech companies.

Interested in this position?

Apply directly on the company website

Apply Now

Similar Roles

Scale AI

SWE Fellow - Human Frontier Collective (UK)

Scale AI 251-1K Diversified Consumer Services

The Human Frontier Collective Fellowship at Scale is a fully remote contract role supporting AI research by designing, evaluating, and interpreting advanced generative AI systems with interdisciplinary experts and partner labs.

United Kingdom Contract Senior AI (Artificial Intelligence) Software Engineer
C++ Generative AI Java JavaScript Machine Learning Python Rust Swift
21 hours, 1 minute ago
OLIVER

Norwegian Post-Editor | Technology Company

OLIVER 1K-5K Media

A tech company is seeking a Norwegian Post-Editor to support MTPE projects by refining marketing copy and product descriptions for AI-driven content development.

Anywhere Freelance Mid Level AI (Artificial Intelligence) Translator
21 hours, 32 minutes ago
PandaDoc

Staff Python Engineer, Document App Track

PandaDoc 251-1K Internet Software & Services

PandaDoc is hiring a Staff Backend Engineer for its Document App Track to lead the evolution of the core document domain that powers document creation, editing, and management across the platform.

Ukraine Contract Lead Backend Engineer
$0k-$1k
Agile AWS DDD Django Docker Grafana Kafka Kubernetes Memcached Microservices NATS PostgreSQL Python RabbitMQ Redis SQL
21 hours, 32 minutes ago
PandaDoc

Staff Python Engineer, Document App Track

PandaDoc 251-1K Internet Software & Services

PandaDoc is hiring a Staff Backend Engineer for its Document App Track to lead the evolution of the core document domain that powers document creation, editing, and management.

Portugal Contract Lead Backend Engineer
$372k-$556k
Agile AWS DDD Django Docker Grafana Kafka Kubernetes Memcached Microservices NATS PostgreSQL Python RabbitMQ Redis SQL
21 hours, 47 minutes ago

You're on a roll! Sign up now to keep applying.

Sign Up

Already have an account? Log in

Used by 14,729+ remote workers