As Data Engineer on our Healthcare Platform, you will own the foundational data ingestion, entity-resolution, and platform infrastructure end-to-end. You will design and operate scalable batch/streaming data pipelines, harden platform orchestration, and ensure system reliability, security, and cost efficiency.
A central challenge of this role is building and operationalizing our unified healthcare data ontology—a consistent semantic model covering hospitals, departments, specialties, metrics, and classification standards. Working closely with Analytics Engineers, Data Scientists, and Methodology experts, you will turn heterogeneous, multi-country hospital data into a coherent, highly queryable data asset.
Key Responsibilities
- Pipeline Infrastructure & Orchestration: Build, optimize, and operate reliable ELT pipelines (using Python, SQL, and Prefect/Airflow) to ingest data from heterogeneous international sources, APIs, databases, and lakehouse storage (S3, Apache Iceberg).
- Healthcare Ontology & Entity Resolution: Drive the implementation of entity resolution and master data management (MDM) for international hospital entities, mapping raw source data to canonical structures and maintaining standardized vocabularies (e.g., ICD/OPS, specialty taxonomies).
- Data Contracts & Schema Governance: Establish strict data contracts (Pydantic, dbt contracts) and schema management, ensuring dataset reproducibility, data lineage tracking, and automated validation across all platform pipelines.
- Platform Efficiency & Cloud Infrastructure: Optimize data storage, query execution, and compute costs across AWS and Snowflake, keeping data assets performant, secure, and cost-effective.
- Automation & CI/CD: Implement automated testing and deployment workflows for data pipelines using GitHub Actions and Infrastructure as Code (Terraform).
- Cross-Functional Data Enablement: Partner directly with Analytics Engineers, Data Scientists, and domain experts to deliver documented, research-grade, and production-ready datasets.
Qualifications
Core Requirements (Must-Haves)
- Data Ingestion & Pipeline Orchestration: Advanced Python and analytical SQL for complex data ingestion across diverse file formats, REST APIs, databases, and cloud lakes (S3/Iceberg). Hands-on experience with modern orchestrators (Prefect, Airflow, or Dagster).
- Entity Resolution & Data Governance: Practical experience with entity resolution/record linkage frameworks (e.g., Splink, dedupe, recordlinkage) and schema management/data contracts (Pydantic, dbt contracts, or JSON Schema).
- Cloud Platform & Warehouse Infrastructure: Deep hands-on experience in an AWS production environment (S3, ECS/EC2) combined with cloud data warehouses (Snowflake).
- Automated CI/CD & Workflow Automation: Proven track record of automating data pipeline deployments, integration tests, and validation workflows via GitHub Actions.