AI Engineer: Benchmarking and Evaluation

Requirements:

• Bachelors degree in Computer Science, AI, Electrical Engineering, Computer Engineering, or a related field plus at least 4 years of experience developing AI/ML algorithms or technologies; or a relevant masters degree plus at least 2 years of that experience.
• At least 4 years of programming experience with Python, Go, Scala, CUDA, or Java.
• Strong engineering and mathematics fundamentals, with the ability to apply knowledge of hardware, software, and AI to identify optimization opportunities.
• Ability to assess scientific publications and apply relevant research techniques in production; enthusiasm for keeping up with AI research.
• Adaptability and skill in bringing clarity to ambiguous problems, investigating root causes, communicating findings clearly, and proposing new ideas.
• Preferred: Experience leading AI systems development and making tradeoffs among cost, latency, throughput, and accuracy.
• Preferred: 6 or more years deploying scalable, responsible AI solutions on cloud platforms such as AWS, Google Cloud, Azure, or equivalent private cloud.
• Preferred: Experience designing, developing, delivering, and supporting AI services.
• Preferred: Experience developing AI/ML technologies such as LLM inference, similarity search, vector databases, guardrails, or memory using Python, C++, C#, Java, CUDA, or Go.
• Preferred: Experience optimizing training and inference software to improve hardware utilization, latency, throughput, and cost.
• Preferred: Experience building agentic AI systems and workflows.
• Preferred: Proficiency designing distributed systems for model training, evaluation, and online inference at petabyte scale.
• Preferred: Experience defining AI model governance processes, including producibility, lineage tracking, and automated retraining schedules.
• Preferred: Demonstrated ability to influence architectural decisions across multiple AI product lines or platforms.
• We will consider sponsoring a new qualified applicant for employment authorization for this position.

Responsibilities:

• Partner with engineers, research scientists, technical program managers, and product managers to deliver AI-powered products for our associates and customers.
• Design, develop, test, deploy, and support AI software components, including foundation-model training, LLM inference, agents and multi-agent workflows, similarity search, guardrails, model evaluation, experimentation, governance, and observability.
• Use open-source and SaaS AI technologies, including AWS Ultraclusters, Hugging Face, vector databases, and PyTorch.
• Develop and introduce advanced foundation-model optimization techniques to improve production system scalability, cost, latency, and throughput.
• Help shape the technical vision and long-term roadmap for our foundational AI systems.
• Own end-to-end architecture for complex AI systems, ensuring maintainability, observability, and ethical alignment.
• Define and maintain service-level objectives for AI reliability, including latency, uptime, and model-performance drift.
• Work with infrastructure engineering to improve GPU/TPU utilization and accelerate model-inference pipelines.
• Lead cross-functional technical reviews for AI system deployments, ensuring security, data governance, and compliance standards are met.
• Mentor Principal and Senior Associates in scalable design, performance tuning, and translating research into production.

Technologies:

• Agentic AI
• AI
• AWS
• Azure
• C#
• Cloud
• CUDA
• Hardware
• Support
• Java
• LLM
• Model Training
• PyTorch
• Python
• Scala
• Security
• Golang
• Machine Learning

More

At Capital One, we build responsible, reliable AI systems to improve banking and create personalized customer experiences. Our Intelligent Foundations and Experiences team works across the company to advance AI science and engineering, developing proprietary models and platforms that deliver value to millions of customers. This AI Engineer 4 role contributes to foundational AI capabilities and scalable infrastructure. The position is available in Cambridge, Massachusetts; McLean, Virginia; New York, New York; and San Jose, California. The listed full-time annual salary ranges are $197,300–$225,100 in Cambridge and McLean, and $215,200–$245,600 in New York and San Jose; pay for other locations varies. The role is eligible for performance-based incentives and our health, financial, and other benefits, with eligibility dependent on employment status and level. We are an equal opportunity employer committed to nondiscrimination and a drug-free workplace.

last updated 43 week of 2026