Big Data Engineer

at  ALTEN Romania

WHAT A TYPICAL DAY MIGHT INCLUDE

• Design & Build ingestion pipelines using ELT and schema on read in Databricks Delta lake
• Design & Develop “Transformations” aspects using ELT framework to modernize the Ingestion pipelines and build data transformations at scale
• Provide technical expertise in the areas of design and implementation of Ratings Data Ingestion pipelines with modern AWS cloud and other technologies such as S3, Hive, Databricks, Scala, Python, and Spark
• Build and maintain a data environment for speed, accuracy, consistency, and “up” time
• Work closely with other data teams & Data Science team and participate in the development of Ingestion pipelines
• Ensure data governance principles are adopted, data quality checks and data lineage are implemented in each hop of the data
• Be in tune with emerging trends Big data and cloud technologies and participate in the evaluation of new technologies
• Ensure compliance through adopting enterprise standards and promoting best practices/guiding principles aligned with organization standards

EXPERIENCE & SKILLS NEEDED

• A minimum of 8+ years of significant experience in application development
• Previous experience in the areas of design and implementation of Ratings Data Ingestion pipelines with modern AWS cloud and other technologies such as S3, Hive, Databricks, Scala, Python, and Spark
• Development, design, and architecture exposure & the ability to ensure quality across various technology components that are developed by geographically diversified software engineers with superior knowledge of system architecture, object-oriented design, and design patterns
• Proficient with software development lifecycle (SDLC) methodologies like Agile and test-driven development
• Experience working with high-volume data and computationally intensive systems
• Garbage collection friendly programming experience - tuning Java garbage collection & performance is a must
• Proficiency in the development environment, including IDE, web & application server, GIT, Continuous Integration, unit-testing tools, and defect management tools Domain knowledge in the Financial Industry and Capital Markets is a plus
• Excellent communication skills are essential, with strong verbal and writing proficiencies
• Mentor teams, innovate and experiment, give a face to business ideas, and present to key stakeholders

Please refer the Job description for details