SENIOR BIG DATA ENGINEER - BATCH & REAL-TIME DATA PIPELINES:

On behalf of our Banking client, Procom is searching for a Senior Big Data Engineer - Batch & Real-time Data Pipelines for a 12-month role. This position is a hybrid position with 2 days onsite at our client’s Toronto office.

SENIOR BIG DATA ENGINEER - BATCH & REAL-TIME DATA PIPELINES - JOB DESCRIPTION:

As a key member of the data engineering team, you will contribute to the development and implementation of GAM’s Data Platform. This role involves designing, implementing, and maintaining data processing pipelines to support data engineering, BI, Machine Learning, and AI capabilities.

SENIOR BIG DATA ENGINEER - BATCH & REAL-TIME DATA PIPELINES - MANDATORY SKILLS:

• 5-7 years of experience building batch and real-time data pipelines leveraging big data technologies.
• 5-7 years of distributed data processing using Spark, Hadoop, Airflow, NiFi, and Kafka.
• Proficiency in writing and optimizing SQL queries and at least one programming language like Python and/or Scala.
• Experience with cloud-based data platforms (Snowflake, Databricks, AWS, Azure, GCP).
• Expertise using CI/CD tools and working with Docker and Kubernetes platforms.
• 5-7 years of experience following DevOps and agile best practices.
• Experience with data modeling tools and methodologies.

SENIOR BIG DATA ENGINEER - BATCH & REAL-TIME DATA PIPELINES – NICE-TO-HAVE SKILLS:

• Experience with OpenShift, S3, Trino, Ranger, and Hive.
• Knowledge of machine learning and data science concepts and tools.
• Knowledge with BI & Analytics tools such as Tableau and Superset.

• Work with business stakeholders and cross-functional teams to understand data requirements and deliver scalable data solutions.
• Design, develop, and maintain robust ETL processes to extract, transform, and load data from various sources into our data platform.
• Build large-scale batch and event-driven data pipelines using cloud and on-premises hybrid data platform topology.
• Work closely with data architects to review solutions and data models and ensure adherence to data platform architecture guidelines and engineering best practices.
• Take ownership of end-to-end deliverables and ensure high-quality software development while fulfilling all operational and functional requirements in a timely manner.
• Implement and enforce data quality standards and best practices while collaborating with data governance teams to ensure compliance with data policies and regulations.
• Optimize data integration workflows for performance and reliability.
• Troubleshoot and resolve data integration and data processing issues.
• Leverage best practices in continuous integration and delivery using DataOps pipelines.
• Apply design-thinking and agile mindset in working with other engineers and business stakeholders to continuously experiment, iterate, and deliver on new initiatives.
• Lead, mentor, and inspire a team of data engineers to achieve high performance levels.