Key Responsibilities

• Design, develop, test, and maintain reliable and scalable ETL/ELT data pipelines using Apache Airflow.
• Develop efficient SQL queries, stored logic, and Python-based data-processing solutions.
• Work with existing cloud data platforms and extend their schemas, integrations, transformations, and querying capabilities.
• Build and maintain data solutions using cloud technologies such as:
  • Google Cloud Dataflow, Dataform or dbt, BigQuery, and Cloud Spanner
  • Azure Data Factory, Synapse Analytics, and Databricks
  • AWS Glue and Lambda
• Design and implement relational, dimensional, star-schema, NoSQL, and, where appropriate, graph-oriented data models.
• Process and integrate data from structured, semi-structured, and unstructured sources.
• Profile source data, assess data quality, identify patterns and anomalies, and use these findings to inform data-model and pipeline design.
• Implement data-quality controls, reconciliation processes, automated validation, and pipeline testing.
• Monitor data pipelines and troubleshoot issues relating to performance, reliability, scalability, and data integrity.
• Collaborate with software engineers, architects, business analysts, testers, product teams, and client stakeholders to translate requirements into practical data solutions.
• Contribute to technical design discussions and provide recommendations on data architecture, storage, transformation, and integration approaches.
• Manage source code through Git and support automated build, test, and deployment processes using CI/CD and DevOps practices.
• Apply appropriate security, access-control, data-governance, and compliance standards.
• Produce and maintain clear technical documentation covering data models, pipelines, transformations, dependencies, and operational procedures.
• Participate fully in agile delivery activities, including planning, estimation, technical reviews, and continuous improvement.