Senior Consultant - Tech Consulting - FS - CNS - TC - Platforms - Hyderabad at EY

Hyderabad, Telangana, India -

Full Time

Start Date

Immediate

Expiry Date

04 Jun, 26

Salary

0.0

Posted On

06 Mar, 26

Experience

5 year(s) or above

Remote Job

Yes

Telecommute

Yes

Sponsor Visa

Skills

Databricks, Spark, Delta Lake, Python, Pyspark, SQL, AWS, ETL/ELT, Medallion Architecture, Airflow, Databricks Workflows, Kimball Modelling, Terraform, Delta Live Tables, Unity Catalog, DataOps

Industry

Professional Services

Description

APAC Data Engineering Team Skill Requirements Data Engineer Role Build and maintain the future of Vanguard’s data platform on Databricks by designing and developing robust, scalable pipelines. Contribute to a validated, self-serve data ecosystem that supports analytics and business insights. Base Skillsets Hands-on experience with Databricks (Spark, Delta Lake). Strong Python, Pyspark and SQL skills. Building and maintaining scalable ETL/ELT pipelines in AWS. Understanding of Medallion Architecture and data validation. Familiarity with orchestration tools (e.g., Airflow, Databricks Workflows). Experience and comfort with modelling with Kimball-style modelling (Star schemas). Preferable Skillsets Experience with CI/CD and infrastructure-as-code (Terraform). Exposure to Delta Live Tables and Unity Catalog. Knowledge of DataOps practices and data quality frameworks. Responsibilities Design and deploy Databricks infrastructure using approved patterns. Develop and maintain ingestion pipelines via the Common Ingestion Framework. Collaborate with Product Managers and Business Analysts to validate data quality and ensure integrity. Define, document, and build tests within the Testing Framework defined within the program, including the Enterprise solution and the local, notebook-based process. Implement CI/CD for data pipelines and enforce best practices in testing and documentation. Monitor pipeline performance, troubleshoot issues, and optimize for cost and scalability. Document solutions and share knowledge with the engineering team. Support migration from existing infrastructure to Databricks. Requisition Id : 1637235 APAC Data Engineering Team Skill Requirements Data Engineer Exp level: 5 yrs Hyderabad location Role Build and maintain the future of Vanguard’s data platform on Databricks by designing and developing robust, scalable pipelines. Contribute to a validated, self-serve data ecosystem that supports analytics and business insights. Base Skillsets Hands-on experience with Databricks (Spark, Delta Lake). Strong Python, Pyspark and SQL skills. Building and maintaining scalable ETL/ELT pipelines in AWS. Understanding of Medallion Architecture and data validation. Familiarity with orchestration tools (e.g., Airflow, Databricks Workflows). Experience and comfort with modelling with Kimball-style modelling (Star schemas). Preferable Skillsets Experience with CI/CD and infrastructure-as-code (Terraform). Exposure to Delta Live Tables and Unity Catalog. Knowledge of DataOps practices and data quality frameworks. Responsibilities Design and deploy Databricks infrastructure using approved patterns. Develop and maintain ingestion pipelines via the Common Ingestion Framework. Collaborate with Product Managers and Business Analysts to validate data quality and ensure integrity. Define, document, and build tests within the Testing Framework defined within the program, including the Enterprise solution and the local, notebook-based process. Implement CI/CD for data pipelines and enforce best practices in testing and documentation. Monitor pipeline performance, troubleshoot issues, and optimize for cost and scalability. Document solutions and share knowledge with the engineering team. Support migration from existing infrastructure to Databricks.

How To Apply:

Incase you would like to apply to this job directly from the source, please click here

Responsibilities

The role involves designing and developing robust, scalable data pipelines on Databricks, building a self-serve data ecosystem for analytics, and deploying Databricks infrastructure using approved patterns. Responsibilities also include developing ingestion pipelines, collaborating on data quality validation, defining and building tests, and implementing CI/CD for data pipelines.