Online AI Benchmark Evaluator - Remote

Requirements:

• We seek fluent English speakers with clear written communication skills and comfort with natural, everyday conversation.
• We value close attention to detail and patience when reviewing or replaying visual and audio content.
• Contributors should be comfortable responding naturally to realistic scenarios and making clear decisions when both options have flaws.
• Experience writing concise, evidence-based explanations or reviews is beneficial.
• A bachelors degree or higher is a plus.
• To sign up, contributors need a valid ID and mobile phone from their country of residence, an up-to-date resume, and a LinkedIn profile showing their education and work experience.
• Applicants must be authorized to work in their country of residence.

Responsibilities:

• We ask contributors to hold live, scenario-based conversations in English with AI voice models and rate each exchange for naturalness, usefulness, and clarity.
• Compare AI-generated images and videos, judging how well each follows its prompt or source material.
• Inspect outputs closely for issues such as distorted anatomy, flickering, unnatural motion, text errors, or mismatched audio and video.
• Rate outputs for dimensions such as prompt adherence, quality, and realism, and write specific explanations in English based on what you observed or heard.
• Label first-person videos of people or robots performing hands-on tasks by dividing clips into individual actions and correcting AI-drafted captions.
• Evaluate generated speech for naturalness, pronunciation, and similarity to a target voice; some tasks involve recording a short spoken explanation.

Technologies:

• AI
• Mobile

More

Outlier is an AI training platform owned and operated by Scale AI. Our experts contribute to projects that generate training data and evaluate AI models, helping improve how advanced systems reason, learn, and perform in real-world settings. This is remote, flexible project work with no minimum hours; contributors set their own schedules and may work from anywhere. We provide free access to leading LLMs, valued at over $200 per month, and opportunities to collaborate with experts on pre-release models. Outlier reports a community of more than 900,000 experts across 52 countries and over $600 million paid to experts. Project work pays up to USD $15 per hour, with possible additional incentives; rates vary by expertise, assessment results, location, project needs, and other factors. Payments are processed weekly on Tuesdays for the preceding Tuesday-to-Monday pay period via PayPal, AirTM, or ACH transfer. Applicants are currently accepted from Australia, Canada, Ireland, New Zealand, the United Kingdom, and the United States. This is a 1099 freelance/independent-contractor opportunity on the Outlier.ai platform, not employment; internships and sponsorship are not offered. Applicants with questions about their individual tax or immigration circumstances should consult an appropriate advisor.

last updated 44 week of 2026