Lead Data Engineer

at  Acerta Analytics Solutions Inc

Toronto, ON, Canada -

Start DateExpiry DateSalaryPosted OnExperienceSkillsTelecommuteSponsor Visa
Immediate07 Feb, 2025Not Specified07 Nov, 2024N/ARegulatory Compliance,Knowledge Sharing,Leadership,Json,Data Modeling,Access Control,Performance Testing,Cloud Storage,Optimization,Coding Standards,Collaborative Environment,Automation,Orchestration,Troubleshooting,EtlNoNo
Add to Wishlist Apply All Jobs
Required Visa Status:
CitizenGC
US CitizenStudent Visa
H1BCPT
OPTH4 Spouse of H1B
GC Green Card
Employment Type:
Full TimePart Time
PermanentIndependent - 1099
Contract – W2C2H Independent
C2H W2Contract – Corp 2 Corp
Contract to Hire – Corp 2 Corp

Description:

ABOUT ACERTA:

Acerta helps manufacturers optimize the quality, safety, and reliability of the products they make. Our machine-learning solutions use industrial data to deliver valuable insights that improve the entire product lifecycle—resulting in greater efficiency and satisfied customers.

JOB SUMMARY:

We are seeking a talented and experienced Lead Data Engineer to join our growing team. The Lead Data Engineer will play a key role in designing, building, and maintaining our data infrastructure, ensuring scalability, reliability, and performance with a team of data engineers around them. This position offers an exciting opportunity to work with large-scale datasets, cutting-edge technologies, and a passionate team of data professionals.

SKILLS AND QUALIFICATIONS:

  • Previous experience in ETL/ELT Architecture: Expertise in designing and optimizing ETL/ELT pipelines to handle various data formats (CSV, JSON, Parquet, etc.) and integrating data from multiple sources (e.g., APIs, cloud storage, client Databricks instances).
  • Strong understanding of REST API principles, experience with high-volume API requests, and ability to optimize API calls and data ingestion strategies.
  • Automation and Orchestration: Proficiency in using orchestration tools like Apache Airflow, or similar tools to automate and manage data workflows.
  • Proficiency using Unity Catalog to establish a consistent security and governance layer, manage access control, and ensure regulatory compliance.
  • Expertise in building efficient ETL/ELT workflows to enable scalable feature engineering.
  • Previous experience working closely with ML/DS teams, understanding of data modeling required for various teams and products.
  • Previous experience in performance testing and optimization (data load testing/performance tuning/monitoring) for various databases, and ETL pipelines.
  • Technical Mentorship: Leadership in coaching junior engineers, setting best practices for coding standards, and fostering a collaborative environment for knowledge sharing and troubleshooting.

Responsibilities:

  • Architect and Develop Data Ingestion to Acerta Platform: Architect, develop, and maintain scalable data pipelines and ETL processes to ingest, process, and analyze large volumes of structured and unstructured data from diverse sources.
  • Collaborate with Cross-Functional Teams: Collaborate with cross-functional teams to understand data requirements, identify opportunities for data-driven insights, and develop solutions to address business challenges.
  • Optimize Data Pipelines: Optimize and tune data pipelines and database queries for performance, efficiency, and reliability.
  • Implement Data Governance: Implement data governance and security best practices to ensure data integrity, privacy, and compliance with regulatory requirements.
  • Support Machine Learning Initiatives: Work closely with data scientists to support their data needs, provide access to relevant datasets, and enable advanced analytics and machine learning initiatives.
  • Stay Abreast of Industry Trends: Stay abreast of industry trends, emerging technologies, and best practices in data engineering and analytics, and apply them to improve our data infrastructure and processes.


REQUIREMENT SUMMARY

Min:N/AMax:5.0 year(s)

Information Technology/IT

IT Software - DBA / Datawarehousing

Software Engineering

Graduate

Proficient

1

Toronto, ON, Canada