Lead Data Engineer at Cambridge Technology Enterprises Limited

Hyderabad, Telangana, India -

Full Time

Start Date

Immediate

Expiry Date

15 Jun, 26

Salary

0.0

Posted On

17 Mar, 26

Experience

5 year(s) or above

Remote Job

Yes

Telecommute

Yes

Sponsor Visa

Skills

Data Pipelines, ETL/ELT, ADF, Airflow, DBT, SQL, Python, Spark, PySpark, Databricks, Azure, AWS, GCP, Data Lakes, Data Warehouses, Dimensional Modelling

Industry

IT Services and IT Consulting

Description

Job Overview Lead Data Engineer is responsible for designing, building, and maintaining the data infrastructure that enables organizations to collect, store, process, and analyze large volumes of data. They create the data pipelines and systems that ensure data flows smoothly and reliably from various sources to databases, analytics platforms, and applications. Key Responsibilities Build and maintain data pipelines for ingesting, transforming, and delivering data. Design and manage data lakes, warehouses, and storage systems for analytics. Develop optimized data models that support reporting and business insights. Ensure data quality, data validation, and governance across all data systems. Collaborate with analysts, data scientists, and business teams to deliver reliable datasets. Use cloud and big data technologies (Azure/AWS/GCP, Spark, Databricks) for scalable processing. Monitor, troubleshoot, and optimize pipelines for performance and cost efficiency. Implement DevOps practices like CI/CD, version control, and automation. Apply strong security practices including access control and compliance. Document data flows, models, and processes to maintain clarity and best practices. Key Skill Technical Skills Data pipelines & ETL/ELT: Building automated data workflows using tools like ADF, Airflow, dbt. Programming: Strong SQL and Python (often Spark/PySpark). Big Data: Experience with Spark, Databricks, and distributed processing. Cloud Platforms: Azure, AWS, or GCP data services (Data Lake, Redshift, BigQuery, etc.). Data Storage & Modelling: Data lakes, warehouses, Dimensional modelling, Delta Lake. Databases: SQL (PostgreSQL, SQL Server) + NoSQL (MongoDB, DynamoDB). DevOps & Automation CI/CD, Git, Docker, Terraform Monitoring and pipeline reliability Data Governance Data quality, metadata management, security, and compliance Soft Skills Problem solving, cross-team collaboration, clear communication, documentation.

Responsibilities

The Lead Data Engineer is responsible for designing, building, and maintaining the data infrastructure, creating pipelines to ensure reliable data flow from various sources to analytics platforms. Key duties include building and managing data pipelines, designing storage systems, developing optimized data models, and ensuring data quality and governance.