Sign up with

Already have an account? Log in here

Need some help?
Talk to us at +91 7670800001

Senior Data Engineer (Python, Pandas & Numpy) at Lumiq

Noida, Uttar Pradesh, India -

Full Time

Start Date

Immediate

Expiry Date

26 Sep, 26

Salary

0.0

Posted On

28 Jun, 26

Experience

5 year(s) or above

Remote Job

Yes

Telecommute

Yes

Sponsor Visa

Skills

Python, Pandas, NumPy, Polars, SQL, PostgreSQL, Elasticsearch, MongoDB, Docker, Kubernetes, ETL Pipeline Design, Data Migration, Performance Optimization, Distributed Data Systems, Data Integrity, Containerization

Industry

Data Infrastructure and Analytics

Description

Job Purpose : We are seeking a highly skilled ETL Data Engineer to re-engineer our existing data pipelines to extract data from a new data source (PostgreSQL / CURA system) instead of the current Microsoft SQL Server (CRM persistence store), while preserving the existing load patterns to Elasticsearch and MongoDB. The engineer will ensure this migration has zero impact on data quality, system performance, or end-user experience. Key Responsibilities : - Analyze existing ETL pipelines and their dependencies on Microsoft SQL Server as source systems. - Design and implement modifications to repoint ETL extractions from PostgreSQL (CURA) while preserving the current transformations and load logic into Elasticsearch and MongoDB. - Ensure end-to-end data integrity, quality, and freshness remain unaffected after the source switch. - Write efficient and optimized SQL queries to extract data from the new source. - Conduct performance testing to confirm no degradation of pipeline throughput or latency in production. - Work closely with DevOps and platform teams to containerize, orchestrate, and deploy the updated ETLs using Docker and Kubernetes. - Monitor post-deployment performance and handle any production issues proactively. - Document design, code, data mappings, and operational runbooks. Required Skills and Qualifications : - Strong experience building and maintaining large-scale distributed data systems. - Expert-level proficiency in Python, especially data analysis/manipulation libraries like pandas, NumPy, and Polars. - Advanced SQL development skills with proven experience in performance optimization. - Working knowledge of Docker and Kubernetes. - Familiarity with Elasticsearch and MongoDB as data stores. - Experience working in production environments with mission-critical systems.

Responsibilities

The role involves re-engineering ETL pipelines to migrate data sources from Microsoft SQL Server to PostgreSQL while maintaining existing load patterns to Elasticsearch and MongoDB. The engineer will ensure data integrity, optimize SQL queries, and collaborate with DevOps for containerized deployment using Docker and Kubernetes.