We are seeking a highly technical and versatile Quality Engineer to join our team. This role will focus on developing and executing comprehensive test plans, automating test cases, and ensuring the quality of our cutting-edge AI software and tools. The ideal candidate possesses a strong understanding of AI/LLM concepts and a proven track record in software quality assurance.  This candidate will have a unique opportunity to help shape and influence quality best practices on highly visible AI products in the firm.

Responsibilities:

• Design, develop, and execute test plans and test cases for AI software and tools, covering functional, performance, and security aspects.
• Automate test cases using Python and relevant testing frameworks.
• Implement and maintain automated integration, regression, and end-to-end testing pipelines.
• Work with OpenAI, Anthropic, and other vendor Model APIs, including API testing framework development and maintenance.
• Utilize code quality frameworks like SonarQube to ensure code quality and maintainability.
• Collaborate with development teams to identify and resolve software defects.
• Contribute to the continuous improvement of our QA processes and methodologies.
• Stay up-to-date with the latest advancements in AI/LLM testing and quality assurance.

Required Skills and Experience:

• Strong proficiency in Python, including experience with testing frameworks (FastAPI experience is a plus).
• Experience with front-end testing frameworks for Angular/React.
• Proven experience working with OpenAI, Anthropic, and other vendor Model APIs, including API testing frameworks.
• Hands-on experience with code quality frameworks like SonarQube.
• Familiarity with container runtime environments like Kubernetes.
• Expertise in automating integration, regression, and end-to-end testing.
• Proficiency with synthetic testing and Realtime User Monitoring (RUM) tools.

Desired Skills and Experience:

• Experience with AI/LLM Test Evaluation Harnesses.
• Experience with benchmarking AI/RAG response quality and correctness.
• Familiarity with various AI/LLM models and architectures.