Senior Data Engineer at CivicDataLab

, delhi, India -

Full Time

Start Date

Immediate

Expiry Date

14 Mar, 26

Salary

0.0

Posted On

14 Dec, 25

Experience

5 year(s) or above

Remote Job

Yes

Telecommute

Yes

Sponsor Visa

Skills

Python, SQL, Excel, Data Engineering, ETL, API Development, Data Pipelines, Data Documentation, Public Sector Experience, Agile Methodologies, Microservice Architecture, Open Source Software, Data Analytics, Data Transformation, Data Management, Collaboration

Industry

Civic and Social Organizations

Description

Job Description - Senior Data Engineer ─ What We're Looking For: CivicDataLab is seeking a Senior Data Engineer to support our current and upcoming interventions through open data platforms and open-source tools, focusing on building robust data pipelines for automated data mining, cleaning, standardisation and transformation for AI readiness. We are looking for people who are strongly aligned with our values and have an innate sense of problem-solving, automating processes and adapting well to dynamic environments. They will work alongside data strategists, public policy researchers and other stakeholders to develop automated ETL (Extract, Transform and Load) data pipelines and know how to model, store and manage bulk datasets. This will help us co-create comprehensive data analytics tools and dashboards for our diverse stakeholders. At CivicDataLab, we believe in using data, tech, design, and social science to strengthen civic engagement and drive evidence-based decision-making. Our projects centre on building data strategy, platforms, and applications to foster data-driven governance. Key Responsibilities: Take ownership over developing scalable data infrastructure for our existing platforms and products from data orchestration pipelines using Prefect and Airflow, to Data Sharing Protocols for our open data collaboratives. Create and oversee data APIs responsible for collecting, managing, and analysing data from diverse public data sources. Maintain and monitor our existing open data platforms like Open Budgets India, Justice Hub, Open Contracting India, and oversee dataset migration into new platforms. Thoroughly document code, processes, and all activities performed by the data team, ensuring clarity and comprehensiveness. This includes documenting algorithms, methodologies, data transformations, and the overall workflow. Lead strategic planning and scoping for data engineering projects, aligning them with product roadmaps and stakeholder needs. Mentor and provide technical guidance to a team of data engineers, conducting code reviews and capacity-building sessions. Quality & compliance: Run structured checklists (licensing/ownership validation, PII checks, metadata completeness) and coordinate fixes with publishers. Requirements Recommended Skills & Requirements: Any graduate, however, candidates with a B.Sc./M.Sc. (or higher) in Computer Science, Data Science, or a related field will be preferred. Proficiency in Python, SQL, and Excel. Experience building dashboards (Superset, Tableau, or Python frameworks). Excellent written/verbal communication and documentation skills. Public-sector experience: Demonstrated work with government departments/PMUs or allied institutions; comfortable with consultations and public-sector processes. Onboarding literacy: Familiarity with dataset discovery, metadata, licensing/ownership, and basic PII/privacy checks; rigorous documentation and follow-through. Operations & collaboration: Excellent coordination and communication skills; readiness for periodic intra-state travel. Extensive working knowledge of API or Stream-based data extraction processes. Ability to gauge requirements for the project, make decisions and coordinate with other team members. Prior experience in actively working and contributing to FOSS (Free and Open Source Software) communities. Familiarity in working with Agile methodologies and Scrum processes. Experience with scalable infrastructure practices such as microservice architecture, infrastructure as code, distributed systems, scaling methods, load balancing and more. Benefits How we work CivicDataLab is based out of Delhi and has the presence of project teams located in Assam and Himachal Pradesh. We follow a hybrid model where our bandhus work out of the office for a minimum of 12 days per month (i.e.) 3 days a week. We use open-source tools and agile methodologies in organising our work. Perks of Working with Us Wellness Allowance At CivicDataLab, we always emphasise the wellness of our bandhus. This includes any Expenditure done for the purpose of Wellness Setup, except any financial instrument, any expense that can be claimed as a deductible expense under Income Tax rules, any goods and services that attract a combined tax, cess or duty of more than 28%. If you're interested in taking classes that enhance your overall physical or mental well-being, you have an INR 60,000 annual stipend to do so. For some people, that might mean a monthly massage. Some take photography lessons or learn a musical instrument or buy a gym membership. It's up to you; the point is to learn something that you feel enriches you as a person. Professional growth and development Allowance At CivicDataLab, we encourage everyone to take up things that help one grow professionally, and you get an annual kitty of INR 60,000 to do so. This includes attending or speaking at conferences and workshops, taking courses, acquiring hardware or software licenses or even joining summer schools. We feel that learning a skill should never be a hurdle to solve important problems for the community.

Responsibilities

The Senior Data Engineer will develop scalable data infrastructure and oversee data APIs for managing diverse public data sources. They will also document processes and mentor a team of data engineers.