We are looking for a QA Automation Engineer to join our team.
We are looking for a QA Automation Engineer to join our team.
About us
- On the market since 2002;
- Operational departments and development hubs worldwide.
Project description
Our partner empowers global organizations to cultivate brand loyalty and accelerate revenue growth through cutting-edge technology.
Their cloud-native infrastructure utilizes proprietary AI to streamline large-scale contact center operations, ensuring seamless interaction management across every communication channel. As a certified telecom provider, they deliver carrier-grade voice quality and robust handling of massive traffic volumes with global numbering capabilities.
They offer a multimodal AI suite tailored for CX, automating workflows and enhancing data integrity. These services integrate natively into the platform or via API across all supported languages.
Their high-performance dialer enables Sales and Service teams to master inbound and outbound engagement, featuring deep CRM integration and real-time analytical monitoring.
Technical stack
-
Automated tests: PyTest, Playwright, k6
-
Infrastructure: Kubernetes
Requirements
- Production-quality test code in Python (PyTest) and TypeScript (Playwright) - readable, maintainable, peer-reviewed.
- Experience designing LLM/AI evaluation frameworks: LLM-as-judge, rubric-based scoring, semantic similarity, or equivalent.
- Practical CI/CD experience with canary releases, automated quality gates, and rollback triggers in a cloud-native environment.
- Ability to define and measure SLOs for AI/ML services - latency, accuracy, reliability - and build automated enforcement.
- Comfortable working with ML engineers on datasets, model outputs, and inference results; understands evaluation metrics.
- Clear written and verbal English for technical and non-technical stakeholder communication.
Nice to have
- Linux proficiency: kubectl, pod inspection, log tailing, cluster debugging.
- Audio data formats and protocols: WebRTC, SIP, or telephony stack experience.
- k6 or equivalent for load testing ML inference APIs.
- Nebius AI Cloud, GPU inference workload testing.
Responsibilities
- Build and maintain automation frameworks (PyTest, Playwright, k6) across four AI products, with quality gates enforced on every PR and canary deployment to the Nebius Kubernetes cluster.
- Own model rollback criteria, canary/A/B deployment quality, and daily cluster debugging via kubectl.
- Design end-to-end test strategies covering API contracts, event streams, and multi-surface call simulation across LiveKit, PSTN, and WebRTC.
- Build LLM evaluation tooling including LLM-as-judge scoring, rubric-based assessment, anomaly detection, and guardrail tests for bias, hallucination, toxicity, and PII across all supported languages.
- Co-design evaluation datasets with R&D and ML engineers, contributing structured quality evidence into model improvement cycles.
- Define and own the SLO framework for the Voice-AI pipeline with production monitoring in Azure Application Insights and weekly quality reporting to leadership.
We offer
- Employment is based on a contract between Cyprus legal entity and a private entrepreneur self-employed or physical person without any reference to a specific location.
- Stable long-term workload (8 hours per day, 40 hours per week), flexible working hours, fully remote.
- Paid vacations (24 working days/year) and sick leave.
- Working on exciting projects with a team of professionals.
- Opportunities for learning and practicing new technologies; internal training.
- Participating in inner meetups and permanent experience exchange with colleagues.
- Well-defined development processes and methodologies.
- English and/or French language learning for free with internal teachers within working hours.
- Reimbursement of medical fees, massage, or sports.
Recognize yourself? We are waiting for your CV!
Please, fill in the form and we will contact you in case if your experience suits our offer