AI Engineer, FM Hosting, LLM Inference

Requirements:

• We need a bachelors degree in Computer Science, AI, Electrical Engineering, Computer Engineering, or a related field, plus at least 6 years of experience developing AI and ML algorithms or technologies, or a masters degree in one of those fields plus at least 4 years of relevant experience
• We need at least 6 years of programming experience with Python, Go, Scala, CUDA, or Java
• We value experience leading AI system development with trade-off decisions around cost, latency, throughput, and accuracy
• We prefer 7 years of experience deploying scalable and responsible AI solutions on cloud platforms such as AWS, Google Cloud, Azure, or similar private cloud environments
• We look for experience designing, building, delivering, and supporting complex AI systems
• We value experience developing AI and ML technologies such as LLM inference, similarity search, VectorDBs, guardrails, and memory using Python, C++, C#, Java, CUDA, or Golang
• We prefer experience applying advanced methods to optimize training and inference software to improve hardware utilization, latency, throughput, and cost
• We look for experience building agentic AI systems and agentic workflows
• We value strong communication and presentation skills, with the ability to explain complex AI concepts to peers
• We prefer experience architecting and integrating heterogeneous AI systems, including rule-based, retrieval-augmented, and generative components, into unified production pipelines
• We value experience defining and enforcing standards for ethical AI deployment, including explainability, fairness, and human-in-the-loop review processes
• We look for demonstrated ability to balance model performance and operating cost through dynamic inference strategies and model compression
• We value experience right-sizing models, instance counts, and hardware types based on requirements such as context length, token inputs, and token outputs

Responsibilities:

• We partner with engineers, research scientists, technical program managers, and product managers to deliver AI-powered products that improve how our associates work and how our customers interact with Capital One
• We design, develop, test, deploy, and support AI software components, including foundation model training, large language model inference, agents and multi-agent workflows, similarity search, guardrails, model evaluation, experimentation, governance, and observability
• We leverage a broad stack of open-source and SaaS AI technologies, including AWS Ultraclusters, Hugging Face, VectorDBs, PyTorch, and more
• We invent and introduce state-of-the-art foundation model optimization techniques to improve scalability, cost, latency, and throughput in large-scale production AI systems
• We contribute to the technical vision and long-term roadmap for foundational AI systems at Capital One
• We design, implement, and optimize multi-model orchestration pipelines that integrate LLMs, vector search, and domain-specific models into unified systems
• We establish and lead cost-performance governance reviews across AI systems, tracking GPU utilization, model throughput, and inference cost efficiency
• We lead team design councils or design review boards to maintain technical consistency and compliance with AI engineering standards
• We mentor Principal- and Manager-level AI engineers, encouraging cross-domain learning and raising the organizations technical maturity

Technologies:

• Agentic AI
• AI
• AWS
• Azure
• C#
• Cloud
• CUDA
• Golang
• Hardware
• Support
• Java
• LLM
• Model Training
• PyTorch
• Python
• Scala
• Machine Learning

More

At Capital One, we are building responsible and reliable AI systems to change banking for good. Our Intelligent Foundations and Experiences (IFX) team sits at the heart of that mission, working closely with partners across the company to advance the state of the art in science and AI engineering and to build proprietary solutions that serve millions of customers. We offer a comprehensive, competitive, and inclusive benefits package that supports total well-being, along with performance-based incentive compensation that may include cash bonuses and long-term incentives. This role includes salary ranges that vary by location, with positions listed in Cambridge, McLean, New York, and San Jose. We may consider sponsorship for a new qualified applicant for employment authorization for this position.

last updated 43 week of 2026