Sign up with

Already have an account? Log in here

Need some help?
Talk to us at +91 7670800001

Big Data Lead at HEXAWARE

, , United States -

Full Time

Start Date

Immediate

Expiry Date

05 Oct, 26

Salary

0.0

Posted On

07 Jul, 26

Experience

5 year(s) or above

Remote Job

Yes

Telecommute

Yes

Sponsor Visa

Skills

Python, Pyspark, SQL, Hadoop, Spark, AWS, Azure, GCP, Bitbucket, Github, CI/CD, Database Design, Data Modelling, Data Warehousing, ETL/ELT Pipelines, Data Validation

Industry

IT Services and IT Consulting

Description

Responsibilities: • Development and Maintain Data Pipelines: Design, implement, and optimize end-to-end ETL/ELT pipelines for ingesting, processing, and transforming large volumes of structured and unstructured data. • Utilize Python and Pyspark: Write efficient, scalable and maintainable code in Python and leverage Pyspark for large-scale data processing in distributed computing environments. Also be able to review existing code and identify areas of improvement. • Ensure Data Quality and Integrity: Implement data validation, cleansing, transformation and reconciliation processes to ensure data accuracy and consistency throughout the data lifecycle. • Collaborate with Stakeholders: Work closely with IT teams and business stakeholders to gather data requirements and translate them to technical solutions. • Troubleshoot and Optimize: Monitor job performance, troubleshoot complex data issues and fine-tune for performance and scalability. • Adhere to Best Practices: Participate in code reviews, establish coding standards, and implement CI/CD pipelines for automated testing and deployment. Skills: • Strong hands-on coding proficiency in Python, Pyspark and SQL (Microsoft SQL Server preferred) • Experience with big data frameworks (Hadoop, Spark). • Experience with cloud platforms ( AWS, Azure or GCP) • Experience with Code versioning tools ( Bitbucket, Github ) • Experience with CI/CD and setting up pipelines. • Solid understanding of database design principles, data modelling, schemas and data warehousing solutions. • Excellent problem-solving and analytical skills to troubleshoot complex data issues independently.

Responsibilities

Design and optimize end-to-end ETL/ELT pipelines for processing large volumes of structured and unstructured data. Collaborate with stakeholders to translate business requirements into technical solutions while ensuring data quality and integrity.