Lead Data Engineer

at  ASTEK SINGAPORE INNOVATION TECHNOLOGY PTE LTD

REQUIREMENTS

• Education :Bachelor’s degree in Computer Science, Data Engineering, Information Technology, or a related field.
• Experience :Minimum of 10 years in data engineering roles with expertise in Hadoop , Spark , and related tools (e.g., HDFS , Hive , Ranger ).Hands-on experience with OpenShift Container Platform (OCP) and container orchestration tools like Kubernetes .Proven track record of building and scaling data infrastructure in large, complex environments.
• Technical Skills :Proficiency in programming languages such as Scala , Python , or Java .Deep understanding of DevOps practices, including CI/CD pipelines and automation tools like Docker , Jenkins , and Ansible .Familiarity with monitoring and logging tools such as Grafana , Prometheus , and Splunk .Exposure to cloud platforms (AWS, Azure, GCP) and their data services is a plus.
• Soft Skills : Exceptional problem-solving abilities and a proactive approach to resolving technical challenges.Strong communication and collaboration skills to work effectively across diverse teams.Ability to balance multiple priorities and deliver high-quality results in a dynamic environment.

PREFERRED QUALIFICATIONS

• Experience in cloud-native data services (e.g., AWS Redshift, Azure Data Factory, GCP BigQuery).
• Knowledge of advanced analytics and machine learning pipelines.

YOUR ROLE

As a Lead Data Engineer , you will spearhead the development and optimization of our data infrastructure to enable advanced analytics capabilities. Collaborating with cross-functional teams, you will ensure seamless integration and delivery of robust, scalable, and efficient data solutions. In this leadership role, you will mentor and guide a team of engineers while contributing to the strategic direction of the organization’s data landscape.

KEY RESPONSIBILITIES

• Data Engineering & Development :Design and implement large-scale data processing pipelines using Spark Scala , handling structured and unstructured data.Integrate Elasticsearch with Spark for efficient indexing, querying, and data retrieval.Optimize Spark applications for performance and scalability, ensuring minimal latency and high throughput.
• Containerization & Orchestration :Deploy data solutions on OpenShift Container Platform (OCP) using containerization and orchestration techniques.Optimize workflows for containerized environments and ensure efficient resource utilization.
• DevOps Integration :Collaborate with DevOps teams to streamline deployment using CI/CD pipelines and tools such as Jenkins , Bitbucket , and Ansible .Implement robust monitoring and logging frameworks using tools like Grafana , Prometheus , and Splunk to maintain platform stability.
• Collaboration & Stakeholder Management :Work with data scientists, analysts, and infrastructure engineers to understand requirements and translate them into technical solutions.Communicate technical designs and processes effectively to both technical and non-technical stakeholders.
• Leadership & Mentorship :Lead and mentor a team of data engineers, fostering a culture of continuous learning and innovation.Advocate for best practices in data engineering, DevOps, and cloud technologies.
• Documentation & Research :Document workflows, pipelines, and configurations to ensure team knowledge sharing.Stay abreast of emerging technologies and best practices in data engineering and cloud infrastructure.