At EY, you’ll have the chance to build a career as unique as you are, with the global scale, support, inclusive culture and technology to become the best version of you. And we’re counting on your unique voice and perspective to help EY become even better, too. Join us and build an exceptional experience for yourself, and a better working world for all. 

Job Title: Senior AI Test Engineer

Job Summary:

We are seeking a skilled Senior AI Test Engineer to join our dynamic team. The ideal candidate will be responsible for developing and executing test plans and test cases for AI models and systems, ensuring their performance, reliability, and accuracy. You will work closely with data scientists, software engineers, and product managers to identify testing requirements and deliver high-quality AI products.

Key Responsibilities:

• Test Planning and Design:
  • Collaborate with cross-functional teams to define testing strategies for AI applications.
  • Develop comprehensive test plans and test cases based on functional and non-functional requirements.
  • Good understanding about the various testing touch points in a model development lifecycle
  • Ability to define test cases to validate input data, Model functional performance, Model response etc
• Test Execution:
  • Execute manual and automated tests for AI models and systems.
  • Monitor and document the results of tests, identifying any defects or areas for improvement.
  • Prior experience in performing AI specific testing methods like Pairwise testing, Metamorphic testing, Back-to-Back testing, Bias Testing, Drift testing etc
  • Knowledge on responsible AI testing
  • Knowledge on explainability testing tools (e.g., LIME, SHAP etc.)
• Model Validation:
  • Good understanding about the various performance benchmarks across the different ML models
  • Validate AI models against performance benchmarks and metrics (e.g., accuracy, precision, recall).
  • Conduct exploratory testing to uncover edge cases and potential failure modes.
  • Exposure to testing LLM models
• Data Management:
  • Work with data engineers to ensure quality and consistency of training and validation datasets.
  • Implement data validation checks to assess the integrity of input data used for AI models.
  • Prior experience in performing various data validation activities like data collection/generation, data augmentation, Exploratory data analysis, data bias and privacy etc
• Automation Development:
  • Develop and maintain automated testing frameworks and scripts for AI applications.
  • Utilize tools and libraries (e.g., TensorFlow, PyTorch, scikit-learn) for testing AI models.
• Defect Tracking and Reporting:
  • Track and manage defects using issue-tracking tools (e.g., JIRA, Bugzilla).
  • Prepare detailed reports on test results, defect status, and overall quality of AI systems.
• Continuous Improvement:
  • Participate in post-mortem analyses of testing processes and results to identify areas for improvement.
  • Stay up-to-date with industry best practices in AI testing and testing methodologies.
• Collaboration and Communication:
  • Communicate effectively with technical and non-technical stakeholders regarding testing progress and outcomes.
  • Provide training and support to team members on AI testing tools and techniques.

Education:

• Bachelor’s degree in Computer Science, Engineering, or a related field; Master’s degree preferred.

Experience:

• 5-9 years of experience in software testing / development with exposure to Python
• 1-2 years in testing on AI/ML applications.

Technical Skills:

• Strong understanding of AI/ML concepts, algorithms, and frameworks (e.g., TensorFlow, PyTorch).
• Knowledge on ML / LLM frameworks & libraries
• Proficiency in programming languages commonly used in AI testing (e.g., Python, R).
• Experience with test automation tools (e.g., Selenium, TestNG) and frameworks.
• Working knowledge on AI testing platforms and tools (e.g., Functionize, Applitools, Testim etc.)
• Familiarity with data validation, model evaluation, and statistical analysis.

Qualifications:

• Bachelor’s degree in Computer Science, Engineering, Data Science, or a related field.
• Proven experience in software testing, with a focus on AI/ML applications preferred.
• Strong understanding of AI/ML concepts and algorithms.
• Proficiency in programming languages such as Python, Java, or C++.
• Experience with testing frameworks and tools (e.g., Selenium, PyTest, unittest).
• Familiarity with version control systems (e.g., Git) and continuous integration/continuous deployment (CI/CD) pipelines.
• Knowledge of data preprocessing, feature engineering, and model evaluation techniques.
• Excellent analytical and problem-solving skills.
• Strong attention to detail and commitment to quality.

Preferred Qualifications:

• Experience with cloud platforms (e.g., AWS, Azure, Google Cloud) and their AI services.
• Familiarity with big data technologies (e.g., Hadoop, Spark).
• Knowledge of Agile methodologies and experience working in Agile teams.

EY | Building a better working world 

 
EY exists to build a better working world, helping to create long-term value for clients, people and society and build trust in the capital markets.  

 
Enabled by data and technology, diverse EY teams in over 150 countries provide trust through assurance and help clients grow, transform and operate.  

 
Working across assurance, consulting, law, strategy, tax and transactions, EY teams ask better questions to find new answers for the complex issues facing our world today.