Data & AI Engineer Lead at NTT DATA

Bengaluru, karnataka, India -

Full Time

Start Date

Immediate

Expiry Date

13 Jan, 26

Salary

0.0

Posted On

15 Oct, 25

Experience

5 year(s) or above

Remote Job

Yes

Telecommute

Yes

Sponsor Visa

Skills

Python, PySpark, SQL, Data Quality, Data Architecture, Data Engineering, Metadata, Orchestration Tools, Data Lineage, Compliance, Databricks, Airflow, Azure Data Factory, Power BI, Tableau

Industry

IT Services and IT Consulting

Description

Framework Design & Architecture Architect a metadata-driven, Python/Spark-based framework for automated data validation across high-volume production datasets. Define DQ rule templates for completeness, integrity, conformity, accuracy, and timeliness. Establish data quality thresholds, escalation protocols, and exception workflows. Automation & Integration Lead the implementation of automated DQ processes integrated with CI/CD and orchestration platforms such as Airflow, Databricks, or Azure Data Factory. Define parameterized rule execution models for scalability and reuse across multiple data domains. Align the DQ framework with organizational governance, privacy, and compliance standards (e.g., HIPAA, SOC 2). Ensure traceability, auditability, and explainability of all DQ rules and outcomes. Define metadata lineage and rule catalogs in collaboration with governance teams. Partner with engineering and data platform teams to operationalize the framework and integrate into existing pipelines. Provide technical guidance to data engineers and analysts to ensure consistent adoption and quality delivery. Present framework outcomes and improvement recommendations to executive leadership. Monitoring & Continuous Improvement Develop metrics and dashboards to monitor DQ framework performance and rule effectiveness. Drive continuous improvement through rule refinement, automation enhancement, and defect prevention analysis. Minimum Skills Required: | **Category** | **Requirement** | | ---------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Experience** | 8-10 years of experience in data architecture, data quality, or data engineering within enterprise data ecosystems. | | **Technical Expertise** | - Expert in **Python** and **PySpark** for large-scale data processing and validation. - Strong command of **SQL** and data profiling methodologies. - Proven experience designing **metadata-driven DQ frameworks** or validation engines. - Familiarity with **orchestration tools** (Airflow, Databricks, ADF). | | **Governance & Compliance** | Deep understanding of data lineage, retention, auditability, and compliance (HIPAA, SOC 2). | | **Platforms & Tools** | Databricks, Spark, Power BI, Tableau, cloud d"

Responsibilities

The Data & AI Engineer Lead will architect a metadata-driven framework for automated data validation and lead the implementation of automated data quality processes. They will also develop metrics and dashboards to monitor framework performance and drive continuous improvement.