Director of AI Model Evaluation & Quality Validation - LLMs & Generative AI

Truveta is the world’s first health provider led data platform with a vision of Saving Lives with Data. Our mission is to enable researchers to find cures faster, empower every clinician to be an expert, and help families make the most informed decisions about their care. Achieving Truveta’ s ambitious vision requires an incredible team of talented and inspired people with a special combination of health, software and big data experience who share our company values.

Truveta was born in the Pacific Northwest, but we have employees who live across the country. Our team enjoys the flexibility of a hybrid model and working from anywhere.  In person attendance is required for one week during the year for Truveta Planning Week.

For overall team productivity, we optimize meeting hours in the pacific time zone. We avoid scheduling recurring meetings that start after 3pm PT, however, ad hoc meetings occur between 8am-6pm Pacific time.

Who We Need  

Truveta is rapidly building a talented and diverse team to tackle complex health and technical challenges. Beyond core capabilities, we are seeking problem solvers, passionate and collaborative teammates, and those willing to roll up their sleeves while making a difference. If you are interested in the opportunity to pursue purposeful work, join a mission-driven team, and build a rewarding career while having fun, Truveta may be the perfect fit for you. 

This Opportunity 

We are seeking a highly skilled and experienced Machine Learning Expert to join our team and evaluate the effectiveness of our AI models utilizing Large Language Models (LLMs). The ideal candidate will have a strong background in machine learning, natural language processing (NLP), and a proven track record of working with LLMs. You will be responsible for assessing the performance, accuracy, and efficiency of our AI models, identifying areas for improvement, and providing actionable insights to enhance their effectiveness. You will work in an exciting and fast-paced environment, collaborating closely with multiple teams across the company. You will work as part of an organization that brings together talent from diverse backgrounds including software engineering, big data, machine learning and AI, and clinical experts, making our team an exciting place to work. We value and encourage diversity in the belief that our differences make us and our products better. 

In This Role, You Will: 

• Evaluate the performance of AI models that leverage Large Language Models (LLMs) in various applications.
• Develop and implement metrics to measure the accuracy, efficiency, and overall effectiveness of LLM-based AI models.
• Conduct thorough analyses to identify strengths and weaknesses in model performance.
• Provide detailed reports and recommendations to improve model accuracy, efficiency, and scalability.
• Collaborate with the development team to refine and enhance AI models based on evaluation results.
• Stay up-to-date with the latest advancements in machine learning, NLP, and LLMs to ensure our models remain at the forefront of the industry.
• Ensure that models meet business requirements and deliver tangible value to our clients. 

Key Qualifications 

• Master's or Ph.D. in Computer Science, Machine Learning, Artificial Intelligence, or a related field. 
• 7+ years of experience in machine learning, with a focus on natural language processing and large language models. 
• Proven track record of evaluating and improving AI models, particularly those involving LLMs.
• Strong programming skills in Python and experience with machine learning frameworks such as TensorFlow, PyTorch, or similar. 
• Familiarity with performance evaluation metrics and techniques for LLMs. 
• Excellent analytical and problem-solving skills. 
• Ability to communicate complex technical concepts to non-technical stakeholders. 

Preferred Qualifications 

• Experience with popular LLMs such as GPT, BERT, T5, or similar. 
• Knowledge of deployment and scaling of AI models in a production environment. 
• Previous experience working in a fast-paced, agile development environment. 
• Contributions to the AI/ML community through research papers, conference presentations, or open-source projects. 

This position offers a unique chance to work on state-of-the-art language models and contribute to transformative research with the vision of Saving Lives with Data. You will be part of a diverse and team who are passionate about pushing the boundaries of machine learning and natural language understanding in the healthcare domain. Join us and make a significant impact on the future of healthcare and patient well-being. 

 Why Truveta?

Be a part of building something special. Now is the perfect time to join Truveta. We have strong, established leadership with decades of success. We are well-funded. We are building a culture that prioritizes people and their passions across personal, professional, and everything in between. Join us as we build an amazing company together.

We Offer:

• Interesting and meaningful work for every career stage
• Great benefits package
• Comprehensive benefits with strong medical, dental and vision insurance plans
• 401K plan
• Professional development & training opportunities for continuous learning
• Work/life autonomy via flexible work hours and flexible paid time off
• Generous parental leave
• Regular team activities (virtual and in-person as soon as we are able)
• The base pay for this position is $230,00 to $260,000. The pay range reflects the minimum and maximum target. Pay is based on several factors including location and may vary depending on job-related knowledge, skills, and experience. Certain roles are eligible for additional compensation such as incentive pay and stock options.

If you are based in California, we encourage you to read this important information for California residents linked here.

Truveta is committed to creating a diverse, inclusive, and empowering workplace. We believe that having employees, interns, and contractors with diverse backgrounds enables Truveta to better meet our mission and serve patients and health communities around the world. We recognize that opportunities in technology historically excluded and continue to disproportionately exclude Black and Indigenous people, people of color, people from working class backgrounds, people with disabilities, and LGBTQIA+ people. We strongly encourage individuals with these identities to apply even if you don’t meet all of the requirements.