Agentic QA Engineer, Generative AI Systems

Requirements:

• At least 7 years of experience in software QA/testing, including 2 years working with AI/ML or LLM-based systems and hands-on testing of agentic or multi-agent architectures.
• Strong Python programming skills, including experience building test harnesses, simulators, and fixtures.
• Experience with LLM evaluation methods such as exact or soft match, BLEU/ROUGE, BERTScore, embedding-based semantic similarity, guardrails, and prompt testing.
• Expertise in distributed systems testing, latency profiling, resiliency patterns such as circuit breakers and retries, chaos engineering, and message queues.
• Familiarity with orchestration frameworks such as LangChain, LangGraph, LlamaIndex, DSPy, OpenAI Assistants/Actions, Azure OpenAI orchestration, or similar tools.
• Proficiency with CI/CD tools such as GitHub Actions or Azure DevOps; observability tools such as OpenTelemetry, Prometheus/Grafana, or Datadog; and feature flags or canaries.
• Understanding of privacy, security, and compliance for AI systems, including PII handling, content policies, and model safety.
• Excellent communication and leadership skills, with the ability to work across Operations, Data, and Engineering teams.
• Preferred: Experience with multi-agent simulators, agent graph testing, and tooling latency emulation.
• Preferred: Knowledge of MLOps, including model versioning, datasets, and evaluation pipelines, as well as A/B experimentation for LLMs.
• Preferred: Background in AWS or other cloud environments, serverless, containerization, and event-driven architectures.
• Preferred: Previous ownership of cost, latency, or SLAs for production AI workloads.

Responsibilities:

• Define and lead QA strategy for agentic and multi-agent AI systems across development, staging, and production.
• Mentor QA engineers and establish testing standards, test-harness coding guidelines, and review practices.
• Work with Agentic Operations, Data Science, MLOps, and Platform teams to integrate QA into the software development lifecycle and incident response.
• Test agent orchestration, tool calls, planner-executor loops, and inter-agent coordination, including task decomposition, handoffs, and goal completion.
• Validate state management, context windows, memory and knowledge stores, and prompt or graph correctness under varied conditions.
• Develop scenario-fuzzing tests using adversarial inputs, prompt variations, tool latency spikes, and degraded APIs.
• Build resilience test suites covering chaos experiments, failover, retries and backoff, circuit breakers, and degraded-mode behavior.
• Establish latency SLOs and measure end-to-end response times across LLM calls, tool invocations, queues, and orchestration layers.
• Assess reliability through soak tests, canary verification, and automated rollbacks.
• Define ground-truth and reference pipelines for task accuracy, including exact-match, semantic-similarity, and factuality checks.
• Create macro-validation frameworks to assess outcomes across multi-step agent workflows.
• Instrument and validate guardrails for toxicity, PII, hallucinations, and policy compliance.
• Design load and stress tests for multi-agent graphs, covering concurrency, throughput, queue depth, and backpressure.
• Verify orchestrator behavior, including DAG execution, retries, branching, timeouts, and compensation paths.
• Build reusable test assets such as scenario configurations, synthetic datasets, prompt libraries, agent graph fixtures, and simulators.
• Integrate tests into CI/CD and production monitoring, with pre-merge gates, nightly runs, canaries, and KPI-based alerts.
• Define release criteria and conduct operational-readiness assessments for performance, security, compliance, and cost and latency budgets.

Technologies:

• AI
• AWS
• Azure
• CI/CD
• Cloud
• Datadog
• DevOps
• GitHub
• Grafana
• Support
• LLM
• MLOps
• OpenTelemetry
• Prometheus
• Python
• Security
• Serverless
• Windows
• Agentic AI

More

We are Tekfortune, a consulting firm providing permanent, contract, and project-based staffing services to organizations across a range of industries. We are recruiting a hands-on Agentic QA Engineer to support production-grade generative AI and multi-agent systems in partnership with our Agentic Operations Team. The role focuses on reliability, resilience, accuracy, latency, orchestration, and scale. This is a long-term position based in Dallas, Texas, and is onsite five days per week. The interview process includes one virtual interview and one in-person interview.

last updated 44 week of 2026