Senior Data Engineer

at  Lean Tech

OVERVIEW:

As a Data Engineer, you will play a crucial role in designing, constructing, and maintaining robust data infrastructure to facilitate the efficient processing of large datasets. Your responsibilities will include developing data pipelines, utilizing cloud platforms such as AWS, and programming in Python to handle data processing tasks. Collaborating with cross-functional teams, you will ensure data integrity and support data-driven decision-making processes.

REQUIREMENTS & QUALIFICATIONS:

To excel in this role, you should possess:

• Bachelor’s degree in Computer Science, Engineering, or a related field.
• Minimum of 4 years of experience in data engineering or a similar role.
• Proficient in Python, Pyspark, and SQL for data manipulation, transformation, and automation tasks.
• Experience with AWS cloud services for data storage, processing, and analytics, including EMR, Lambda Functions, and Sagemaker Notebook.
• Deep understanding of platform engineering principles, including designing and building robust, scalable, and fault-tolerant data platforms.
• Strong communication and collaboration skills to work effectively with cross-functional teams.
• Experience with version control systems like Git is preferred.

NICE TO HAVE SKILLS:

• Experience with API integration for data exchange between systems.
• Knowledge of testing frameworks like Pytest for automated testing of data pipelines.
• Familiarity with continuous integration and delivery tools like CodePipeline.
• Experience with NoSQL databases like DynamoDB for storing and querying semi-structured data.
• Proficiency in data visualization tools like Tableau, and PBI for creating insightful dashboards and reports.

WHAT YOU WILL BE DOING:

• Design, develop, and deploy scalable and reliable data pipelines for ingesting, processing, and storing large volumes of data.
• Collaborate with data scientists and analysts to understand data requirements and implement solutions that meet business needs.
• Implement and maintain data infrastructure on cloud platforms like AWS, ensuring scalability, reliability, and security.
• Write efficient and optimized code in Python and SQL for data manipulation, transformation, and automation tasks.
• Utilize tools like Sagemaker Notebook for data exploration, model training, and deployment.
• Architect, implement and optimize data pipelines and workflows to ensure efficient data ingestion, processing, and delivery within the data ecosystem.
• Monitor and troubleshoot data pipelines to ensure smooth operation and performance optimization.
• Collaborate with cross-functional teams to define and implement data governance policies and best practices.
• Stay updated with the latest trends and technologies in data engineering and recommend improvements to existing systems and processes.

To excel in this role, you should possess:

• Bachelor’s degree in Computer Science, Engineering, or a related field.
• Minimum of 4 years of experience in data engineering or a similar role.
• Proficient in Python, Pyspark, and SQL for data manipulation, transformation, and automation tasks.
• Experience with AWS cloud services for data storage, processing, and analytics, including EMR, Lambda Functions, and Sagemaker Notebook.
• Deep understanding of platform engineering principles, including designing and building robust, scalable, and fault-tolerant data platforms.
• Strong communication and collaboration skills to work effectively with cross-functional teams.
• Experience with version control systems like Git is preferred