Data Flow Engineer (Warsaw, 40% remote/60% onsite) – EU Public Institutions at The White Team

Capon Bridge, West Virginia, United States -

Full Time

Start Date

Immediate

Expiry Date

13 Sep, 26

Salary

0.0

Posted On

15 Jun, 26

Experience

10 year(s) or above

Remote Job

Yes

Telecommute

Yes

Sponsor Visa

Skills

Apache NiFi, Python, REST API, Change Data Capture, Apache Iceberg, Apache Atlas, Apache Ranger, Apache Kafka, Apache Avro, Data Integration, Data Governance, Data Flow Management, Big Data Streaming, SQL, Data Processing, Data Transformation

Industry

IT Services and IT Consulting

Description

Data Flow Engineer (Warsaw, 40% remote/60% onsite) – Frontex Profile: Data Flow Engineer (Data Flow / Data integration / NiFi / Big Data Streaming & Flow Management). Place of performance: Frontex Headquarters in Warsaw (Poland), 40% remote/60% onsite. Duration of the mission: 48 months. Security Clearance: Restricted / EU Restricted. Minimum level of education: Level 6. Minimum English language skills: B2. Minimum IT relevant experience: 8 years (6 years in relevant roles). Award Criteria: 50% Price / 50% Quality. Minimum required scoring for interview: 60%. Rate: The rate offered depends on the candidate’s level, in accordance with the European Commission’s public grading system. Further details are available upon discussion, for the following estimations: · Estimated NWH: 230days x4 years. · Estimated EWH: 23days x4 years. · Estimated OnC: 3000h x4 years. Required technical certificates: At least 1 certification among: · Cloudera Certified Developer for Apache NiFi or an equivalent certification. · Cloudera Data Flow (CFM) related certification or an equivalent certification. · Each equivalent certification must be recognized internationally (subject to acceptance as a valid credential by the Contracting Authority). Knowledge and Skills · 01] Expert knowledge in defining, designing, implementing and maintaining complex data flows in Apache NiFi (Cloudera DataFlow). · [02] Adanced Python programming skills (data processing, NiFi custom logic, flow automation, integrations). · [03] Advanced in building integrations based on REST API (endpoints' calling, OAuth/JWT authentication, rate limiting, error recovery). · [04] Hands-on experience in building CDC-based data flows (Change Data Capture) - using native NiFI processes/connectors as well as with SQL Builder. · [05] Good knowledge of Apache Iceberg (tables, schema evolution, partitioning). · [06] Knowledge on data governance and catalog in CDP: Apache Atlas (metadata, lineage, tagging) and Apache Ranger (security policies, authorization). · [07] Experience with Apache Kafka as message broker (topics, producers/consumers, schema registry, NiFi integration) and Apache Avro as serialization standard (schema evolution, compatibility etc.). Specific Expertise · [01] Hands-on experience in daily working with Apache NiFi, preferably in CDP environment - design, deployment, monitoring, troubleshooting of advanced flows - min. 2-3 years of hands-on. · [02] Documented experience in realisation of at least one big integration project using NiFi as central tool (API calling, db integrations, transformations, data routing and delivery). · [03] Practical knowledge and experience with Apache Iceberg, preferably in CDP environment - table creation/management, NiFi/Spark/Flink integration etc. · [04] Experience in implementation of CDC pipelines to/from relational databases. · [05] Practical knowledge of configuring and managing governance/lineage in Apache Atlas + Ranger in the context of NiFi flows (tagging, policies, audit). · [06] Experience in working with Apache Kafka in CDP ecosystem (integration NiFi -> Kafka -> downstream consumers, schema management with Avro). Tasks and Responsibilities · Design, implementation, testing and maintenance of complex data flows in Cloudera DataFlow (Apache NiFi) - ingest, transform, enrich, route, egress. · Building and optimisation of CDC-based pipelines (real-time / near-real-time) with usage of NiFi, Kafka, Debezium / SQL CDC connectors. · Integration with external systems via REST API, JDBC, Kafka and other protocols. · Managing data schemas (Avro), metadata and lineage in Apache Atlas. · Configuration of security and governance (Ranger policies) for data flows. · Monitoring, alerting and troubleshooting performance / reliability of data pipelines. · Collaboration with data engineers, architects and business stakeholders on defining requirements and architecture of data flows. · Creation and maintenance of SOPs and documentation for data flows, and runbooks. · Participation in CDP / NiFi / Kafka upgrades and migrations. Travel: By default, travelling in the interest of service is not foreseen for this position/profile. · Nevertheless, Frontex may exceptionally request to carry out some services at other locations than Frontex Headquarters or other Contracting Authority’s premises.

Responsibilities

The Data Flow Engineer will design, implement, test, and maintain complex data flows using Cloudera DataFlow (Apache NiFi). Responsibilities include building CDC-based pipelines, integrating with external systems, and managing data schemas and governance.