QA AI Automation Engineer

Requirements:

• 5+ years of experience in software quality, test automation, or SDET roles, including ownership of automation architecture.
• Production experience with LLM-powered systems, including agent patterns, prompt engineering, tool or function calling, and orchestration.
• Hands-on experience with major model APIs such as OpenAI, Anthropic, or Azure OpenAI, and AI-assisted development tools such as Claude Code, Copilot, or Codex.
• Strong programming skills in at least one of Python, TypeScript/JavaScript, Java, Kotlin, or C#, with the ability to read code in the others.
• Experience defining and interpreting evaluation criteria for systems without a single correct answer.
• Strong CI/CD knowledge, including pipeline integration, gating, monitoring, and logging.
• In-depth API testing experience, including validation of third-party integrations.
• Clear written communication skills, with the ability to explain quality risks to product owners and root causes to engineers.
• Preferred: Familiarity with evaluation and observability tools such as Langfuse, Promptfoo, OpenTelemetry, or New Relic, or equivalent tools.
• Preferred: Understanding of RAG fundamentals, including embeddings, chunking strategies, vector search, and retrieval evaluation.
• Preferred: Experience with workflow orchestration tools such as n8n or similar.
• Preferred: Azure cloud experience and exposure to the .NET ecosystem.
• Preferred: Experience with BDD, Selenium, or comparable UI automation at scale.
• Preferred: Experience in financial services, wealth management, or another regulated industry, or a strong interest in data governance in regulated environments.
• Preferred: Experience mentoring or leading distributed QA engineers.

Responsibilities:

• Build and maintain evaluation suites for our AI assistant and API, measuring correctness, citation fidelity, retrieval quality, refusal and escalation behavior, tone, latency, and cost.
• Curate and version representative golden datasets and adversarial tests based on advisor workflows, updating them as our product evolves.
• Develop LLM-as-judge and rubric-based scoring for non-deterministic behavior, and validate these methods against human-labeled data.
• Integrate evaluations into CI to measure regressions caused by prompt, model, retrieval, or tool changes, and instrument production traces so live failures inform future tests.
• Design and operate a multi-agent CI/CD workflow that runs evaluations, triages failures, documents reproducible defects, proposes or applies fixes, and verifies results.
• Define agent roles, handoffs, safeguards, and trust checks to determine when fixes can be applied automatically or should be reviewed by a person.
• Choose between LLM-based and deterministic automation with cost in mind, and report on autonomous resolution, false positives, and engineering time saved.
• Establish quality standards and automated gates for agent-authored code, covering test coverage, mutation testing, static analysis, security and dependency scanning, architecture, and review practices.
• Detect common agent-generated code failures, including incorrect logic, scope creep, duplicated abstractions, missing edge cases, and tests that do not meaningfully verify behavior.
• Work with architects and delivery teams to maintain quality as agent-assisted development expands.
• Collaborate with product, engineering, and AI Labs to define coverage and acceptance criteria for AI features before development begins.
• Contribute evidence to release-readiness and go/no-go decisions.
• Maintain and modernize our existing API, UI, and integration automation, and mentor engineers.
• In your first 30 days, learn our assistant, API layer, and current evaluation and automation systems, then deliver a baseline evaluation suite for a high-traffic chat workflow with results visible in CI.
• By days 31–60, launch an initial multi-agent defect-detection step for one service, establish the threshold for automatically applied fixes, and publish its metrics.
• By days 61–90, propose and begin rolling out automated quality standards for agent-authored code across delivery teams.

Technologies:

• AI
• API
• Azure
• C#
• CI/CD
• Claude Code
• Cloud
• Copilot
• Java
• JavaScript
• Kotlin
• LLM
• Langfuse
• n8n
• OpenTelemetry
• Python
• RAG
• Security
• Selenium
• TypeScript
• UX UI Design
• ASP.NET
• Product Owner

More

We are building AI systems that advisors and their clients use within a regulated wealth management platform. Our work includes an internal AI assistant, an external-facing API layer, and an expanding set of agentic workflows, where correctness, grounding, and auditability are especially important. This hands-on engineering role focuses on evaluating non-deterministic AI systems and strengthening quality as autonomous coding agents become a meaningful part of development; it is not a test-execution role.

last updated 44 week of 2026