Big Data Engineer AWS (EMR, S3, Glue, Spark, Hadoop)
Toronto
Role and ResponsibilitiesUnderstand requirement from product owners and translate into requirement and scope documentsDecide on the best fitment of technologiesservices that are available in scopeCreate Solution for data ingestion and data transformation using Hadoop services like Spark, Spark streaming, Hive, etc.Create technical design documents to communicate solutions to the team and mentor the team to develop the solutionBuild the solution with AWS and Hadoop Services as per design specificationsAssist team teams to build test cases and support with testing the solutionCoordinate with Upstream, Downstream and other supporting teams for production implementationProvide post-production support for Solutions implemented Please list 5 mandatory MUST HAVE skills and experience for this requirement. Please include skills related to technical as well as domain and non-technical skills and experience as applicable to the position.Candidate should have strong working experience with data services (EMR, S3, Glue) on AWS cloud.Strong hands-on experience on AWS EMR Glue (Spark with Scala) and AWS S3.Strong hands-on experience in Hadoop services like SparkIn-depth knowledge and extensive experience in building batch workloads on AWS using AWS EMR.Adept in analyzing and refining requirements, consumption query patterns and choosing the right technology fit like RDBMS, data lake and data warehouse.Should have knowledge of analytical data modelling on any of the RDBMSMPP platform Knowledge of PythonProven practical experience in migrating Hadoop based data lake from on-prem to AWS EMR on EMRFSPlease list 5 NICE TO HAVE but not mandatory skills and experience for this requirement Hands-on experience working with Terra bytes peta bytes scale data and millions of transactions per day.Skills to develop ETL pipeline using AirflowKnowledge of Spark streaming or any other streaming jobsAbility to deploy code using AWS Code Pipeline and Bit bucket is an added plusExpert in any of the following programming language Scala, Java and comfortable with working on Linux platform.Knowledge of Cloud based MPP Platfor

Please refer the Job description for details