Position Summary: We are looking for a talented Senior Engineer to build and manage self-serve platforms focused on real-time ML deployment and advanced data engineering. This role combines expertise in cloud-native platform engineering, data pipeline creation, and seamless deployment of machine learning models at scale.
Responsibilities:
Design and develop scalable microservices-based platforms using Kubernetes, Docker, and Python (FastAPI) for managing ML workflows and data pipelines.
Architect and implement real-time ML inference platforms using AWS SageMaker, Databricks, and ensure model versioning, monitoring, and lifecycle management.
Build and enhance ETL/ELT pipelines using PySpark and Pandas, and manage feature stores for high-quality data.
Design distributed data pipelines, integrating DynamoDB, PostgreSQL, MariaDB, and other databases.
Implement data lakes and warehouses to support analytics and ML workflows.
Design, optimize, and maintain CI/CD pipelines with Jenkins and GitHub Actions for continuous deployment and testing.
Automate monitoring and validation to ensure consistent data quality.
Collaborate with cross-functional teams to understand business requirements and deliver integrated solutions.
Maintain comprehensive technical documentation for platforms and workflows.
Required Skills & Qualifications:
5+ years of experience in platform engineering, data engineering, or DevOps roles.
Proficient in Python, PySpark, FastAPI, Kubernetes, and Docker.
Strong experience with AWS services, including SageMaker, Lambda, DynamoDB, and EC2.
Expertise in managing distributed data pipelines with Databricks and PostgreSQL.
Familiarity with CI/CD tools like Jenkins, GitHub Actions, and monitoring tools like New Relic.
