Senior Event Streaming Engineer at BDIPlus

New York, New York, United States -

Full Time

Start Date

Immediate

Expiry Date

17 Sep, 26

Salary

190000.0

Posted On

19 Jun, 26

Experience

5 year(s) or above

Remote Job

Yes

Telecommute

Yes

Sponsor Visa

Skills

Apache Kafka, Kafka Streams, Schema Registry, Kafka Connect, Apache Iceberg, CDC Tools, SQL, Python, Java, Scala, AWS, Observability, Event-Driven Architecture, Data Pipelines, S3 Sink Connector, Stateful Processing

Industry

Information Technology & Services

Description

OVERVIEW BDIPlus is seeking a Senior Event Streaming Engineer to support a Fortune 100 financial services client’s enterprise data and AI transformation. You will help build an event-driven intelligence platform. This role sits at the intersection of real-time data engineering, streaming architecture, and AI enablement. You’ll design scalable pipelines that convert raw system activity into high-value, real-time insights used by analytics platforms and AI agents. Your work will directly influence how the business understands and engages with its customers. ---------------------------------------- WHAT YOU’LL DO * Instrument administration systems to capture and emit key customer lifecycle events into Apache Kafka * Design and build Kafka-to-Iceberg data pipelines using S3 Sink Connector with exactly once delivery guarantees * Develop real-time behavioral aggregates (7/30/90-day windows) using Kafka Streams with stateful processing and windowed joins * Configure and manage Schema Registry (Confluent or AWS Glue), enforcing BACKWARD_TRANSITIVE compatibility * Implement resilient DLQ (Dead Letter Queue) patterns with ordering-safe replay and escalation workflows * Build and maintain observability frameworks (ELK, Prometheus) for pipeline health, consumer lag, and alerting * Partner with AI/Agent engineering teams to deliver production-ready streaming data via MCP endpoints * Develop CDC pipelines for legacy and modern systems (mainframe, AIX/AS400, and policy admin platforms) * Participate in client demos and contribute to knowledge transfer with internal client teams * Leverage AI-assisted development tools (Claude Code / Codex) to accelerate delivery ---------------------------------------- REQUIRED QUALIFICATIONS * 5+ years of hands-on experience with Apache Kafka (topic design, partitioning, scaling, consumer group management) * Strong expertise in Kafka Streams (stateful processing, windowing, exactly once semantics) * Experience with Schema Registry (Confluent or AWS Glue) and schema evolution (Avro/Protobuf) * Proven experience with Kafka Connect and S3 Sink Connector; familiarity with Lakehouse patterns (Apache Iceberg preferred) * Experience with monitoring and observability (JMX, consumer lag analysis, ELK dashboards) * Hands-on experience with CDC tools (e.g., Debezium, Informatica IDMC CDC, or equivalent) * Strong programming skills in SQL and at least one of: Python, Java, or Scala * Familiarity with AWS services (S3, Glue, Athena, Step Functions) * Comfort working with AI-assisted development tools (Claude Code experience preferred; training available) ---------------------------------------- PREFERRED QUALIFICATIONS * Experience working with mainframe data (COBOL copybooks, VSAM, AIX/AS400) * Background in financial services or insurance * Hands-on experience with Apache Iceberg (partitioning, schema evolution) * Familiarity with Denodo or semantic data layer tools * Experience with infrastructure-as-code (Terraform or CloudFormation) * Consulting or client-facing experience, including knowledge transfer responsibilities ---------------------------------------- ABOUT BDIPLUS BDIPlus is a data and AI consulting firm with deep experience delivering for Fortune 500 clients, including American Express and Morgan Stanley. We specialize in enterprise data products, identity resolution, semantic layers, and event-driven architecture. Our engineers are AI-native, leveraging tools like Claude Code from day one to deliver faster, smarter, and more scalable solutions than traditional consulting teams.

Responsibilities

Design and build scalable event-driven intelligence platforms and real-time data pipelines using Apache Kafka and Iceberg. Implement behavioral aggregates, CDC pipelines, and observability frameworks to support AI enablement and analytics.