Senior Data Engineer - Python & Pyspark at Citi

Chennai, tamil nadu, India -

Full Time

Start Date

Immediate

Expiry Date

18 Mar, 26

Salary

0.0

Posted On

18 Dec, 25

Experience

5 year(s) or above

Remote Job

Yes

Telecommute

Yes

Sponsor Visa

Skills

Python, PySpark, Data Architecture, ETL, Data Quality, Data Governance, Performance Optimization, Cloud Platforms, SQL, NoSQL, Apache Spark, Data Warehousing, Containerization, Git, MLOps, Data Visualization, Real-Time Data Streaming

Industry

Financial Services

Description

Data Architecture & Design: Design, develop, and optimize data architectures, pipelines, and data models to support various business needs, including analytics, reporting, and machine learning. ETL/ELT Development (Python/PySpark Focus): Build, test, and deploy highly scalable and efficient ETL/ELT processes using Python and PySpark to ingest, transform, and load data from diverse sources into data warehouses and data lakes. Develop and optimize complex data transformations using PySpark. Data Quality & Governance: Implement best practices for data quality, data governance, and data security to ensure the integrity, reliability, and privacy of our data assets. Performance Optimization: Monitor, troubleshoot, and optimize data pipeline performance, ensuring data availability and timely delivery, particularly for PySpark jobs. Infrastructure Management: Collaborate with DevOps and MLOps teams to manage and optimize data infrastructure, including cloud resources (AWS, Azure, GCP), databases, and data processing frameworks, ensuring efficient operation of PySpark clusters. Mentorship & Leadership: Provide technical guidance, mentorship, and code reviews to junior data engineers, particularly in Python and PySpark best practices, fostering a culture of excellence and continuous improvement. Collaboration: Work closely with data scientists, analysts, product managers, and other stakeholders to understand data requirements and deliver solutions that meet business objectives. Innovation: Research and evaluate new data technologies, tools, and methodologies to enhance our data capabilities and stay ahead of industry trends. Documentation: Create and maintain comprehensive documentation for data pipelines, data models, and data infrastructure. Bachelor's or Master's degree in Computer Science, Software Engineering, Data Science, or a related quantitative field. 5+ years of professional experience in data engineering, with a strong emphasis on building and maintaining large-scale data systems. Extensive hands-on experience with Python for data engineering tasks. Proven experience with PySpark for big data processing and transformation. Proven experience with cloud data platforms (e.g., AWS Redshift, S3, EMR, Glue; Azure Data Lake, Databricks, Synapse; Google BigQuery, Dataflow). Strong experience with SQL and NoSQL databases (e.g., PostgreSQL, MySQL, MongoDB, Cassandra). Extensive experience with distributed data processing frameworks, especially Apache Spark. Programming Languages: Expert proficiency in Python is mandatory. Strong SQL mastery is essential. Familiarity with Scala or Java is a plus. Big Data Technologies: In-depth knowledge and hands-on experience with Apache Spark (PySpark) for data processing, including Spark SQL, Spark Streaming, and DataFrame API. Experience with Apache Kafka, Apache Airflow, Delta Lake, or similar technologies. Data Warehousing: In-depth knowledge of data warehousing concepts, dimensional modeling, and ETL/ELT processes. Cloud Platforms: Hands-on experience with at least one major cloud provider (AWS, Azure, GCP) and their data services, particularly those supporting Spark/PySpark workloads. Containerization: Familiarity with Docker and Kubernetes is a plus. Version Control: Proficient with Git and CI/CD pipelines. Excellent problem-solving and analytical abilities. Strong communication and interpersonal skills, with the ability to explain complex technical concepts to non-technical stakeholders. Ability to work effectively in a fast-paced, agile environment. Proactive and self-motivated with a strong sense of ownership. Experience with real-time data streaming and processing using PySpark Structured Streaming. Knowledge of machine learning concepts and MLOps practices, especially integrating ML workflows with PySpark. Familiarity with data visualization tools (e.g., Tableau, Power BI). Contributions to open-source data projects. ------------------------------------------------------ For complementary skills, please see above and/or contact the recruiter. ------------------------------------------------------

Responsibilities

Design, develop, and optimize data architectures and pipelines to support business needs. Collaborate with teams to manage data infrastructure and provide mentorship to junior engineers.