Senior Python Engineer - AI Coding Agent Evaluation (Freelance)

5 days, 6 hours ago
Freelance
Lead
Software Development
Mindrift.ai: Be the “I” in AI

Mindrift.ai: Be the “I” in AI

Join 10,000+ experts earning $15-50/hr training AI models remotely. Flexible freelance work, weekly payments. No AI experience required. Apply in 5 minutes.

Internet Software & Services

Description

  • Build realistic developer environments with codebases, infrastructure, and supporting context such as tickets, docs, and conversations.
  • Design tasks from intermediate states of those environments, including prompts and clear definitions of what counts as a solved task.
  • Write tests that verify agent solutions, accepting valid approaches and rejecting incorrect ones.
  • Review agent solutions, analyze failures, and refine tasks and tests based on QA feedback.
  • Iterate on evaluation criteria to make assessments fair, robust, and challenging for AI coding agents.
  • Ensure tasks are solvable by an AI agent within realistic developer scenarios.
  • Assess whether task setups expose meaningful differences between strong and weak model outputs.

Requirements

  • 8+ years of experience in software development.
  • Experience with Python, including FastAPI.
  • Experience with JavaScript/TypeScript, including React.
  • Experience with Docker, Postgres, Kafka, and Redis.
  • Experience writing functional and integration tests.
  • English proficiency at B2 level or higher.
  • Submit CV in English and indicate English proficiency level.
  • Ability to work on a project-based, non-permanent engagement.

Benefits

  • Up to $200/hr equivalent compensation, depending on level and pace.
  • Flexible schedule with no fixed work hours.
  • Estimated 30 hours of work per task, depending on complexity.
  • Project-based paid opportunities with leading tech companies.

Interested in this position?

Apply directly on the company website

Apply Now

Similar Roles

Remote Work From Home: Audio Evaluation Project

CrowdGen by Appen Internet Software & Services

CrowdGen is hiring a remote independent contractor for an audio evaluation project to compare paired recordings and select the better one for AI speech improvement.

5 hours, 48 minutes ago

Maps Personalization Relevance Rater - Gujarati(India)

Welo Global Professional Services

Freelance raters are needed in India to evaluate personalized search and location recommendations for a weekly project using Google Maps history and defined guidelines.

5 hours, 48 minutes ago

Remote Work From Home: Audio Evaluation Project

CrowdGen by Appen Internet Software & Services

CrowdGen is hiring a remote, project-based Independent Contractor to compare pairs of audio recordings for AI training and select the better recording according to evaluation guidelines.

5 hours, 48 minutes ago

Generative AI Analyst | Traditional Chinese

Welo Global Professional Services

Welo Data is hiring a remote Generative AI Analyst for Traditional Chinese to review and annotate multilingual, multimodal AI content for a 4-week freelance project.

Generative AI LLM
6 hours, 32 minutes ago

You're on a roll! Sign up now to keep applying.

Sign Up

Already have an account? Log in

Used by 14,729+ remote workers