Senior Data Engineer

at  greeHill

JOB OVERVIEW

We are looking for a highly skilled and motivated Senior Data Engineer to join our team. In this role, you will be responsible for designing, implementing, and maintaining data pipelines, with a strong focus on scalability, reliability, and performance. You will work closely with data scientists, analysts, and other engineers to ensure our data infrastructure supports the advanced analytics and machine learning needs of the company.

• Build and orchestrate complex DAGs in Apache Airflow, including advanced use cases involving GPU pools and DAG triggers.
• Develop and manage APIs using FastAPI to enable seamless data flow across different services.
• Implement ETL processes leveraging Python, Postgres, and PostGIS to handle large datasets, including those with spatial data requirements.
• Optimize SQL queries for spatial data within Postgres/PostGIS for efficient data retrieval and processing.
• Design and implement data storage solutions using Azure Blobs and Azure Data Lake.
• Ensure data quality by integrating and maintaining Great Expectations within data pipelines.
• Support the data analytics team by maintaining and optimizing Superset dashboards, enabling self-serve BI for various business stakeholders.
• Utilize Git for version control, ensuring best practices are followed through pre-commit hooks and code reviews.
• Implement and manage Kubernetes for container orchestration to ensure scalable and resilient deployments.
• Use Azure DevOps (CI/CD) pipelines to automate testing, deployment, and monitoring of data engineering solutions.