• Python and automated testing.
• Understanding of LLM applications, prompts, and RAG architectures.
• API testing and test automation.
• Familiarity with LLM evaluation methods and quality metrics.
• Logging, tracing, monitoring, and debugging.
• Understanding of application security and data protection.
• Ability to design reproducible tests and analyze results.

• Develop evaluation frameworks for LLM applications and AI agents.
• Create test datasets and automated regression tests for AI outputs.
• Measure response quality, retrieval accuracy, latency, and operating cost.
• Test tool calling, workflow execution, and integration failure scenarios.
• Implement monitoring, tracing, logging, and performance dashboards.
• Identify hallucinations, prompt-injection risks, and unsafe model behavior.
• Design fallback mechanisms, human approval steps, and recovery procedures.
• Collaborate with application engineers to improve production reliability.
• Document evaluation results and communicate risks to technical stakeholders.

• Demonstrable experience with automated testing or AI evaluation.
• Strong analytical and documentation skills.
• Familiarity with tools such as LangSmith, OpenTelemetry, or equivalent monitoring and tracing systems is preferred.
• Understanding of human-in-the-loop workflows and AI governance is an advantage.
• Ability to explain technical risks and recommend practical mitigations.

PKR 220,000–450,000 monthly, depending on experience and specialist knowledge.

• Remote-first working arrangements.
• Opportunity to specialize in AI reliability engineering.
• Exposure to production AI systems and enterprise deployments.
• Professional development and performance reviews, subject to company policy.
• Potential long-term client engagements.