At EY, you’ll have the chance to build a career as unique as you are, with the global scale, support, inclusive culture and technology to become the best version of you. And we’re counting on your unique voice and perspective to help EY become even better, too. Join us and build an exceptional experience for yourself, and a better working world for all. 

Job Title: AI Test Manager

Job Summary:

We are seeking a skilled AI Test Manager who will be responsible in leading a team of QE professionals in designing, implementing, and executing testing strategies for AI-based applications and systems. This role requires a strong understanding of AI/ML concepts, testing methodologies, and project management skills to ensure the delivery of high-quality products. The AI Test Manager will collaborate with cross-functional teams, including data scientists, software engineers, and product managers, to ensure robust testing processes and practices are in place.

Key Responsibilities:

• Leadership and Team Management:
  • Lead and mentor a team of QA engineers focused on AI and machine learning projects.
  • Foster a collaborative and innovative team culture.
  • Conduct performance evaluations, provide feedback, and support career development for team members.
• Testing Strategy Development:
  • Design and implement comprehensive testing strategies for AI applications, including functional, integration, regression, and performance testing.
  • Develop and maintain test plans, test cases, and test scripts tailored to AI-specific challenges.
  • Ensure compliance with industry standards and best practices in AI testing.
• Collaboration with Cross-Functional Teams:
  • Work closely with data scientists, developers, and product managers to understand requirements and provide insights on testing needs.
  • Participate in design reviews and provide input on potential testing challenges and considerations.
• AI Model Validation:
  • Define and execute methodologies for validating AI models, including bias detection, performance evaluation, and accuracy testing.
  • Establish metrics and KPIs to assess the effectiveness of AI systems and ensure they meet specified requirements.
  • Experience in testing LLM models
• Test Automation and Tools:
  • Oversee the implementation and maintenance of automated testing frameworks and tools specific to AI applications.
  • Stay updated on emerging tools and technologies in AI testing and recommend their adoption as appropriate. Knowledge on explainability testing tools (e.g., LIME, SHAP etc.)
  • Experience in any of the AI testing platforms and tools (e.g., Functionize, Applitools, Testim etc.)
  • Knowledge on cloud platforms & services for ML Ops / LLM Ops
• Documentation and Reporting:
  • Maintain detailed documentation of testing processes, results, and lessons learned.
  • Prepare and present reports to stakeholders on testing progress, risks, and outcomes.
• Continuous Improvement:
  • Identify opportunities for process improvement in testing and development methodologies.
  • Advocate for best practices in AI testing across the organization.
• Risk Management:
  • Identify potential risks in AI models and applications and develop mitigation strategies.
  • Ensure timely communication of testing issues and risks to project stakeholders.

Qualifications:

• Education: Bachelor’s degree in Computer Science, Engineering, or a related field; Master’s degree preferred.
• Experience:
  • 10+ years of experience in software testing / development with exposure to Python
  • 1-2  years of experience in defining test strategy and plan for AI/ML applications
  • 2+ years in a managerial or leadership role.
• Technical Skills:
  • Strong understanding of AI/ML concepts, algorithms, and frameworks (e.g., TensorFlow, PyTorch).
  • Knowledge on ML / LLM frameworks & libraries
  • Proficiency in programming languages commonly used in AI testing (e.g., Python, R).
  • Experience with test automation tools (e.g., Selenium, TestNG) and frameworks.
  • Familiarity with data validation, model evaluation, and statistical analysis.
• Soft Skills:
  • Excellent communication and interpersonal skills.
  • Strong problem-solving abilities and analytical thinking.
  • Ability to work collaboratively in a fast-paced, dynamic environment.

EY | Building a better working world 

 
EY exists to build a better working world, helping to create long-term value for clients, people and society and build trust in the capital markets.  

 
Enabled by data and technology, diverse EY teams in over 150 countries provide trust through assurance and help clients grow, transform and operate.  

 
Working across assurance, consulting, law, strategy, tax and transactions, EY teams ask better questions to find new answers for the complex issues facing our world today.