AI Engineer (LLM Gateway, FM Hosting) - New York

Requirements:

• Bachelors degree in Computer Science, AI, Electrical Engineering, Computer Engineering, or a related field and at least 4 years of experience developing AI/ML algorithms or technologies; alternatively, a masters degree in one of these fields and at least 2 years of relevant experience
• At least 4 years of programming experience with Python, Go, Scala, CUDA, or Java
• Strong engineering and mathematical foundations, with the ability to apply knowledge of hardware, software, and AI to identify optimization opportunities
• Ability to interpret current research and thoughtfully apply novel techniques in production
• Adaptability in ambiguous situations, strong problem-solving and communication skills, and resilience in pursuing business goals
• Preferred: experience leading AI system development and making tradeoffs among cost, latency, throughput, and accuracy
• Preferred: 6 or more years deploying scalable, responsible AI solutions on cloud platforms such as AWS, Google Cloud, Azure, or equivalent private cloud
• Preferred: experience designing, developing, delivering, and supporting AI services
• Preferred: experience developing AI/ML technologies such as LLM inference, similarity search, vector databases, guardrails, or memory, using Python, C++, C#, Java, CUDA, or Go
• Preferred: experience optimizing training and inference software to improve hardware utilization, latency, throughput, and cost
• Preferred: experience building agentic AI systems and workflows
• Preferred: proficiency designing distributed systems for model training, evaluation, and online inference at petabyte scale
• Preferred: experience defining AI model governance processes, including producibility, lineage tracking, and automated retraining schedules
• Preferred: demonstrated ability to influence architecture across multiple AI product lines or platforms
• Preferred: passion for keeping current with AI research and systems and applying suitable advances in production

Responsibilities:

• Partner with engineers, research scientists, technical program managers, and product managers to deliver AI-powered products for associates and customers
• Design, develop, test, deploy, and support AI software components, including foundation model training, LLM inference, agents and multi-agent workflows, similarity search, guardrails, model evaluation, experimentation, governance, and observability
• Use open-source and SaaS AI technologies, including AWS Ultraclusters, Hugging Face, vector databases, and PyTorch
• Develop foundation model optimization techniques to improve the scalability, cost, latency, and throughput of production AI systems
• Help shape the technical vision and long-term roadmap for foundational AI systems
• Own end-to-end architecture for complex AI systems, ensuring maintainability, observability, and ethical alignment
• Define and maintain AI reliability service-level objectives, including latency, uptime, and model performance drift
• Work with infrastructure engineering to optimize GPU/TPU utilization and accelerate model inference pipelines
• Lead cross-functional technical reviews for AI system deployments, ensuring security, data governance, and compliance standards are met
• Mentor Principal and Senior Associates in scalable design, performance tuning, and translating research into production

Technologies:

• Agentic AI
• AI
• AWS
• Azure
• C#
• Cloud
• CUDA
• Hardware
• Support
• Java
• LLM
• Machine Learning
• Model Training
• PyTorch
• Python
• Scala
• Security
• Golang

More

We are Capital One, where we build responsible, reliable AI systems to transform banking. Our investments in technology and machine learning support personalized customer experiences and scalable AI capabilities. The Intelligent Foundations and Experiences team works across the company to advance AI science and engineering and build proprietary solutions that serve millions of customers. This AI Engineer 4 role is based in Cambridge, Massachusetts; McLean, Virginia; New York, New York; or San Jose, California. Listed annual salary ranges are $197,300–$225,100 in Cambridge and McLean and $215,200–$245,600 in New York and San Jose; pay ranges vary by location, and the actual offer is stated in the offer letter. The role is eligible for performance-based incentives and comprehensive health, financial, and other benefits, with eligibility dependent on employment status and level. We will consider sponsoring a new qualified applicant for employment authorization. Applications are expected to be accepted for at least five business days. We are an equal opportunity employer and promote a drug-free workplace.

last updated 43 week of 2026