Lead Data Engineer

at  Acerta Analytics Solutions Inc

ABOUT ACERTA:

Acerta helps manufacturers optimize the quality, safety, and reliability of the products they make. Our machine-learning solutions use industrial data to deliver valuable insights that improve the entire product lifecycle—resulting in greater efficiency and satisfied customers.

JOB SUMMARY:

We are seeking a talented and experienced Lead Data Engineer to join our growing team. The Lead Data Engineer will play a key role in designing, building, and maintaining our data infrastructure, ensuring scalability, reliability, and performance with a team of data engineers around them. This position offers an exciting opportunity to work with large-scale datasets, cutting-edge technologies, and a passionate team of data professionals.

SKILLS AND QUALIFICATIONS:

• Previous experience in ETL/ELT Architecture: Expertise in designing and optimizing ETL/ELT pipelines to handle various data formats (CSV, JSON, Parquet, etc.) and integrating data from multiple sources (e.g., APIs, cloud storage, client Databricks instances).
• Strong understanding of REST API principles, experience with high-volume API requests, and ability to optimize API calls and data ingestion strategies.
• Automation and Orchestration: Proficiency in using orchestration tools like Apache Airflow, or similar tools to automate and manage data workflows.
• Proficiency using Unity Catalog to establish a consistent security and governance layer, manage access control, and ensure regulatory compliance.
• Expertise in building efficient ETL/ELT workflows to enable scalable feature engineering.
• Previous experience working closely with ML/DS teams, understanding of data modeling required for various teams and products.
• Previous experience in performance testing and optimization (data load testing/performance tuning/monitoring) for various databases, and ETL pipelines.
• Technical Mentorship: Leadership in coaching junior engineers, setting best practices for coding standards, and fostering a collaborative environment for knowledge sharing and troubleshooting.

• Architect and Develop Data Ingestion to Acerta Platform: Architect, develop, and maintain scalable data pipelines and ETL processes to ingest, process, and analyze large volumes of structured and unstructured data from diverse sources.
• Collaborate with Cross-Functional Teams: Collaborate with cross-functional teams to understand data requirements, identify opportunities for data-driven insights, and develop solutions to address business challenges.
• Optimize Data Pipelines: Optimize and tune data pipelines and database queries for performance, efficiency, and reliability.
• Implement Data Governance: Implement data governance and security best practices to ensure data integrity, privacy, and compliance with regulatory requirements.
• Support Machine Learning Initiatives: Work closely with data scientists to support their data needs, provide access to relevant datasets, and enable advanced analytics and machine learning initiatives.
• Stay Abreast of Industry Trends: Stay abreast of industry trends, emerging technologies, and best practices in data engineering and analytics, and apply them to improve our data infrastructure and processes.