WHAT YOU'LL OWN
- Test strategy. Define and own the testing strategy across the platform - what gets tested at which layer (unit, component, contract, integration, end-to-end, production verification), where deterministic assertions end and statistical evaluation begins, and how quality gates map to release risk. Keep it a living document engineers actually use, not a compliance artifact.
- Software test automation. Automated coverage across the React web app, backend APIs and services, auth and authorization, database operations, third-party integrations, email and document ingestion, background jobs and async workflows, multi-tenant configuration and isolation, and end-to-end RFQ and quotation journeys. Playwright, pytest, API clients, SQL, CI/CD. Focus on critical business journeys and high-risk failure modes, not test-case count.
- API and integration testing. Beyond happy-path responses: request and response schemas, permissions, invalid and incomplete inputs, error handling, idempotency and duplicate requests, timeouts and retries, async state transitions, webhooks and event processing, data consistency across services, tenant isolation, and integration failure and recovery - running in CI with enough diagnostic output to explain a failure.
- AI and LLM evaluations. Versioned evaluation datasets built from real business scenarios, edge cases and observed failures. Targets include extraction accuracy, material and product matching, classification and routing, agent decision-making, generated RFQ and quotation content, grounding, hallucination frequency, instruction adherence and final business correctness. You define the scoring methods - exact assertions, semantic comparison, rule-based and model-based graders, human review where needed. Because behaviour is nondeterministic, expect quality thresholds and statistical comparison rather than identical output every run.
- Agent workflow testing. Inspect the whole execution path, not the final response: was the correct tool selected, were its arguments valid, were calls made in the right order, was the right source data used, were permissions and tenant boundaries respected, did retries and recovery work, did the agent stop or escalate when uncertain, did it avoid unnecessary actions, did the outcome match the business objective, and were latency and cost acceptable.
- Controlled use of AI testing agents. Use bounded agents to analyse requirements, suggest missing scenarios, generate candidate tests, explore unusual journeys, create adversarial inputs, reproduce defects and analyse traces. You stay responsible for validating what they generate and for defining authoritative pass/fail criteria - AI output never becomes an uncontrolled release decision.
- Release readiness. Define and maintain the criteria: critical journeys pass, API contracts stable, no cross-tenant access violations, extraction accuracy above the agreed threshold, no material regression in agent task completion, known critical scenarios still succeeding, latency and cost within limits. You communicate what passed, what failed, what's uncertain and the risk of proceeding, and give a recommendation - final release ownership stays shared with engineering and product.
- Failure investigation. Work a regression across layers - browser traces and screenshots, API requests, application logs, agent execution traces, database queries, background-job status, model outputs and grader results, deployment and config changes - and turn every meaningful production failure into a reproducible test or evaluation case.
Automated UI and API suites · integration and end-to-end scenarios · version-controlled AI evaluation datasets · agent-workflow graders · CI/CD quality gates · test and evaluation reports · release-readiness dashboards · regression and failure classifications · defect reproduction packages · quality-risk assessments · a living test strategy documenting layer ownership, coverage standards and evaluation methodology.