Project overview: Supporting AML migrate their pricing airflow data jobs from hive on emr to spark on kubernetes

• Onboard onto AML systems and gain access to: AWS, Datadog, Windfarm Airflow, AML Airflow, GitHub, Google Docs, Slack
• Assist with the conversion, refactoring and migration of AML Pricing Airflow DAGs and tasks from Hive running on AWS EMR to Spark running on AWS EKS
• This would include performing data validation to ensure the new DAG produces an output equivalent to the original one
• Collaborate with David Shu from the DIGIT team as well as Robert Gay towards the overall objective of retiring the AML-hosted Pricing Airflow cluster
• As we learn more about the nature of modernizing Hive to Spark as well as organizing our jobs to efficiently perform data validation, a change in direction may be necessary; this will be confirmed by Robert Gay.

Redfin provides equal employment opportunities to all employees and applicants for employment and prohibits discrimination based on race, color, religion, age, sex, national origin, disability status, genetics, protected veteran status, sexual orientation, gender identity or expression, and any other characteristic protected by applicable federal, state or local law. If you need accommodation in the application or recruitment process because of a disability or special need, please contact recruitingteam@redfin.com.
Redfin accepts applications on an ongoing basis.

Please refer the Job description for details