Kafka Developer at Encora

Kuala Lumpur, Kuala Lumpur, Malaysia -

Full Time

Start Date

Immediate

Expiry Date

29 Apr, 26

Salary

0.0

Posted On

29 Jan, 26

Experience

2 year(s) or above

Remote Job

Yes

Telecommute

Yes

Sponsor Visa

Skills

Kafka, Java, Spring Boot, Microservices, Event-Driven Architecture, CI/CD, Cloud Platforms, Kafka Streams, Schema Management, Security Best Practices, Error Handling, Performance Optimization, Monitoring, Agile, Technical Documentation, Streaming Technologies

Industry

IT Services and IT Consulting

Description

The Kafka Developer is responsible for designing, developing, and maintaining event-driven data pipelines and streaming solutions within the project ecosystem. The role focuses on enabling scalable, reliable, and loosely coupled microservices using Apache Kafka, tightly integrated with Spring Boot applications, CI/CD pipelines, and cloud platforms. Day-to-Day Responsibilities Design and develop Kafka producers and consumers using Java and Spring Kafka to support event-driven workflows. Collaborate with Full Stack and backend teams to define event models, message contracts, and topic strategies. Design and manage Kafka topics, partitions, replication factors, and retention policies based on business and performance needs. Implement event-driven integration patterns, enabling asynchronous communication between microservices. Develop stream processing applications using Kafka Streams, including filtering, transformations, aggregations, and joins. Implement schema management using Avro, JSON, or Protobuf, ensuring backward and forward compatibility. Apply Kafka security best practices, including SSL/TLS, SASL authentication, ACLs, and secure data transmission. Design robust error-handling mechanisms, including retries, dead-letter topics, and idempotent message processing. Optimize Kafka performance by tuning producer/consumer configurations, partition strategies, and batching. Integrate Kafka-based services with Spring Boot microservices deployed on AWS, VMware TAS, or container platforms. Contribute to CI/CD pipelines for automated build, test, and deployment of Kafka applications. Monitor Kafka clusters and applications using Prometheus, Grafana, ELK, or native Kafka metrics. Perform production support and BAU activities, including troubleshooting message lag, consumer failures, and cluster issues. Conduct root cause analysis for incidents and implement preventive improvements. Participate in Agile ceremonies, including sprint planning, daily stand-ups, reviews, and retrospectives. Maintain technical documentation, including topic catalogs, event schemas, and integration guidelines. Continuously improve system reliability, scalability, and maintainability by adopting industry best practices. Stay updated on Kafka ecosystem enhancements and streaming technologies and proactively suggest improvements.

Responsibilities

The Kafka Developer is responsible for designing, developing, and maintaining event-driven data pipelines and streaming solutions. This includes collaborating with teams to define event models and implementing integration patterns for microservices.