Solution Architect- Databricks at Thakral One

Mumbai, maharashtra, India -

Full Time

Start Date

Immediate

Expiry Date

29 Jul, 26

Salary

0.0

Posted On

30 Apr, 26

Experience

10 year(s) or above

Remote Job

Yes

Telecommute

Yes

Sponsor Visa

Skills

Azure Databricks, PySpark, Spark SQL, Delta Lake, Unity Catalog, Medallion Architecture, Data Engineering, Data Architecture, ETL/ELT Pipelines, CDC Integration, Data Modeling, Azure DevOps, Azure Data Factory, Git, CI/CD, Data Quality

Industry

IT Services and IT Consulting

Description

Requirement: Solution Architect - Databricks We are looking for an experienced Solution Architect - Databricks to join our Data delivery Team for a large-scale data modernization program with a leading telecommunications company. This role involves designing and architecting enterprise-grade data solutions on Azure Databricks, migrating legacy data warehousing systems to a modern lakehouse architecture. Key Responsibilities: Architecture & Design Design and architect end-to-end data pipelines using Azure Databricks with Medallion Architecture (Bronze → Silver → Gold → Semantic layers) Lead the design of hybrid orchestration frameworks combining DLT-Lakeflow for CDC ingestion and metadata-driven frameworks for transformation layers Define and implement Unity Catalog governance strategies for data access, security, and lineage Architect Databricks Workflows for job scheduling, dependency management, and replacing legacy Control-M orchestration Design SCD Type 2 implementations and complex transformation patterns for EDW migration Technical Leadership Provide technical guidance on PySpark and Spark SQL best practices for large-scale data processing Define coding standards, modular notebook organization, and configuration-driven development approaches Lead technical decisions on schema evolution, data quality frameworks, and reconciliation strategies Evaluate and recommend approaches for surrogate key generation, batch control mechanisms, and selective reprocessing Migration & Modernization Architect migration strategies from legacy systems (ODS, EDW, Informatica PowerCenter) to Azure Databricks lakehouse Design patterns for CDC ingestion using Oracle GoldenGate and Informatica IDR integration Define approaches for historical data migration and parallel run strategies Ensure minimal refactoring of existing SQL/PySpark code through metadata-driven frameworks Operational Excellence Design solutions supporting BAU operations including manual data patching, hold/resume/skip controls, and entity-level reprocessing Architect Job/Cycle/Batch auditing frameworks aligned with existing operational patterns Define monitoring, alerting, and logging strategies across all pipeline layers Ensure solutions support targeted fixes without full refresh requirements Stakeholder Engagement Collaborate with Amdocs and customer architecture teams on design approvals and technical alignment Participate in discovery sessions for EDW entities (DDS, SDS, DNF) Present technical solutions and trade-off analyses to leadership and steering committees Work closely with delivery teams to translate designs into implementable solutions Required Qualifications: Experience 8+ years of experience in data engineering and data architecture roles 4+ years of hands-on experience with Databricks (Delta Lake, Spark, Unity Catalog) 3+ years of experience with Azure cloud services (ADLS Gen2, Azure DevOps, Azure Data Factory) Proven experience in large-scale data migration projects (EDW modernization preferred) Technical Skills Must Have: Azure Databricks - Expert level (Delta Lake, DLT/Lakeflow, Databricks Workflows, Unity Catalog) PySpark & Spark SQL - Strong proficiency in writing optimized transformations Medallion Architecture - Hands-on experience implementing Bronze/Silver/Gold patterns Data Modelling - Dimensional modeling, SCD implementations, surrogate key strategies ETL/ELT Pipelines - Experience with metadata-driven and configuration-driven frameworks CDC Integration - Experience with Oracle GoldenGate, Kafka, or similar CDC tools Data Quality - DQ frameworks, reconciliation, exception handling patterns Version Control - Git, CI/CD pipelines, Databricks Asset Bundles (DAB) Good to Have: Power BI - Semantic layer design, report integration patterns Informatica - PowerCenter, IDR (for migration context) Control-M - Understanding for migration/replacement scenarios Terraform/IaC - Infrastructure provisioning for Databricks workspaces Telecommunications Domain Certifications (Preferred) Databricks Certified Data Engineer Professional Databricks Certified Solution Architect Professional Microsoft Azure Data Engineer Associate (DP-203) Microsoft Azure Solutions Architect Expert (AZ-305) Soft Skills & Competencies Strong analytical and problem-solving abilities Excellent communication skills - ability to articulate complex technical concepts to diverse audiences Experience working in distributed/remote teams across time zones Ability to mentor and guide junior team members Strong documentation skills - HLSD, LLD, design decisions Collaborative mindset - working with different teams, and customer stakeholders

Responsibilities

The Solution Architect will design and implement end-to-end data pipelines using Azure Databricks and lead the migration of legacy systems to a modern lakehouse architecture. They are responsible for defining governance strategies, ensuring operational excellence, and collaborating with stakeholders to align technical solutions with business requirements.