Lead Data Engineer

at  Kualty

WHAT WERE LOOKING FOR LEAD DATA ENGINEER

We’re looking for a seasoned Lead Data Engineer with 4-6 Years of work experience in data management disciplines including data integration, modeling, optimization and data quality, and/or other areas directly relevant to data engineering responsibilities and tasks required

RESPONSABILITIES:

• Design, create and maintain data pipelines will be the primary responsibility of the data engineer.
• Drive automation through effective metadata management.
• Assist with renovating the data management infrastructure to drive automation in data integration and management.
• Utilize modern data preparation, integration and AI-enabled metadata management tools and techniques.
• Track data consumption patterns.
• Perform intelligent sampling and caching.
• Monitor schema changes.
• Recommend and automate integration flows.

EXPERIENCE AND KNOWLEDGE

• 4-6 Years of work experience in data management disciplines including data integration, modeling, optimization and data quality, and/or other areas directly relevant to data engineering responsibilities and tasks required
• At least 3 years of experience working in cross-functional teams and collaborating with business stakeholders in the banking business domain, in support of a departmental and/or multi-departmental data management and analytics initiative required
• Strong experience with advanced analytics tools for object-oriented/object function scripting using languages such as R, Python, Java, and Scala required
• Strong experience with popular database programming languages including SQL and PL/SQL for relational databases and certifications on upcoming NoSQL/Hadoop oriented databases like MongoDB and Cassandra for nonrelational databases required
• Strong experience in working with large, heterogeneous datasets in building and optimizing data pipelines, pipeline architectures and integrated datasets using traditional data integration technologies These should include ETL/ELT, data replication/CDC, message-oriented data movement, API design and access and upcoming data ingestion and integration technologies such as stream data integration, CEP and data virtualization required
• Strong experience in working with SQL on Hadoop tools and technologies including HIVE, Impala, Presto and others from an open source perspective and Hortonworks Data Flow (HDF), Dremio, Informatica, Talend among others from a commercial vendor perspective required
• Strong experience in working with and optimizing existing ETL processes and data integration and data preparation flows and helping to move them in production required
• Strong experience in working with both open-source and commercial message queuing technologies (Kafka, JMS, Azure Service Bus, Amazon Simple queuing Service), stream data integration technologies such as Apache Nifi, Apache Beam, Apache Kafka Streams, Amazon Kinesis, others and stream analytics technologies (Apache Kafka, KSQL, Apache Spark) required
• Basic experience working with popular data discovery, analytics and BI software tools like Tableau, and OBI for semantic-layer-based data discovery required
• Strong experience in working with data science teams in refining and optimizing data science and machine learning models and algorithms required
• Basic experience in working with data governance teams and specifically business data stewards and the CISO in moving data pipelines into production with appropriate data quality, governance and security standards and certification required
• English level B2+ (Please apply ONLY if you meet this requirement).
• Only applies LATAM.

• Work with data science teams and with business (data) analysts to refine their data requirements for various data and analytics initiatives.
• Propose appropriate (and innovative) data ingestion, preparation, integration and operationalization techniques.
• Train counterparts such as data scientists, data analysts, LOB users or any data consumers in data pipelining and preparation techniques.
• Ensure that data users and consumers use the data provisioned to them responsibly through data governance and compliance initiatives. Participate in vetting and promoting content created in the business and by data scientists to the curated data catalog for governed reuse.
• Become a data and analytics evangelist by promoting the available data and analytics capabilities and expertise to business unit leaders and educating them in leveraging these capabilities in achieving their business goals.
• Define coding standards, including Object Oriented design standards and design patterns
• Implement a standardized library of reusable objects across the enterprise. This would include reusable libraries to integrate with the major source systems, such as the core banking platform, for example.
• Lead the assessment and implementation of advanced integration patterns, such as real-time streaming and real-time API integrations as an example