JOB DESCRIPTION:

• Develop and maintain data pipelines using Apache Airflow, including DAG creation, scheduling, and monitoring
• Design and implement backend services and microservices using Scala, ensuring high performance and scalability
• Collaborate with data engineers and architects to integrate Airflow workflows with Spark, Kafka, and other components for real-time and batch processing
• Optimize and troubleshoot Airflow DAGs and Scala-based applications for performance and reliability.
• Ensure code quality through unit testing, code reviews, and adherence to best practices.
• Participate in Agile development cycles, contributing to sprint planning, retrospectives, and daily stand-ups.
• Document technical designs, workflows, and operational procedures.

REQUIREMENTS:

• Proficiency in Scala: Strong understanding of functional programming, concurrency, and JVM-based development.
• Experience with Apache Airflow: DAG design, scheduler configuration, and integration with cloud or on-prem data platforms.
• Familiarity with Spark Structured Streaming, Kafka, and data lake architectures is a plus.
• Knowledge of CI/CD pipelines and software composition analysis tools like Git/SonarQube is beneficial.
• Experience with cloud platforms (AWS, Azure, IBM Cloud) and containerization (Docker, Kubernetes) is advantageous.
• Strong problem-solving skills and ability to work independently or in a team.

Please refer the Job description for details