QA and Performance Testing Engineering Lead

QA & Performance Engineering Lead (AI/LLM Focus) The Role We are seeking a high-caliber QA and Performance Engineering Lead to spearhead the testing strategy for enterprise-grade AI and LLM solutions. In this role, you will define the architecture for functional, non-functional, and performance testing, ensuring that complex AI agent workflows and large-scale applications meet the highest standards of reliability and compliance. You will act as a bridge between traditional QA excellence and the cutting-edge requirements of GenAI evaluation. Core Responsibilities & Technical Expertise • Strategic QA Leadership: Leverage 10+ years of experience leading enterprise-wide testing initiatives within Fortune 500 environments to design comprehensive QA architectures. • AI/LLM Specialized Evaluation: Implement advanced metrics for model assessment, including BLEU, ROUGE, perplexity, and specialized scoring for hallucination and grounding rates. • Performance & Resilience Engineering: Build frameworks for load, stress, and chaos testing to ensure system stability under extreme conditions and peak workloads. • Automation & Orchestration: Engineer robust CI/CD test pipelines using Azure DevOps or GitHub Actions, focusing on automated API testing (Pytest/Postman) and integrated test harnesses. • Agentic Workflow Validation: Design testing strategies for multi-step AI agents, covering tool chaining, orchestration, and context injection accuracy. • Data Governance & Compliance: Apply deep knowledge of data lineage (Purview/Unity Catalog) and maintain strict traceability and auditability standards required in regulated industries. • Lifecycle Management: Oversee model release gates, registry promotions, and the management of synthetic datasets and versioning. Key Deliverables • Unified Testing Framework: A standardized taxonomy and coverage model spanning unit, integration, E2E, and AI agent workflows. • AI Evaluation Suite: A comprehensive suite for validating model consistency, toxicity, and correctness, supported by Proof-of-Concept (PoC) validations. • Automated Performance Harness: Scalable workload models designed for peak-load scenarios and resiliency benchmarking. • Smart Quality Gates: Automated pass/fail scoring mechanisms embedded directly into release pipelines across all quality dimensions. • Advanced Observability: Implementation of "Golden Dashboards" tracking real-time metrics such as latency-per-thought, grounding quality, and functional pass rates. Professional Profile • Expertise in Enterprise QA Architecture (Functional + Non-functional + Performance). • Deep understanding of ML/LLM lifecycle and model promotion pipelines. • Strong background in Regulated Industries (ensuring compliance and audit readiness). • Hands-on experience with Synthetic Data generation and dataset versioning.