Data Engineer (Databricks)

at  Capgemini

YOUR SKILLS AND EXPERIENCE

• Minimum 3+ years of experience as a Data Engineer or similar role.
• Proven expertise in Databricks, Apache Spark, and data pipeline development and strong understanding of data warehousing concepts and practices.
• Experience with Microsoft Azure cloud platform, including Azure Data Lake Storage, Databricks, and Azure Data Factory.
• Knowledge of SQL and scripting languages like Python or Scala and hands-on experience with AI/ML concepts and tools.
• Excellent problem-solving and analytical skills and strong communication and teamwork skills.
• Passion for data and a thirst for learning.

YOUR ROLE

• Design and build high-performance data pipelines: Utilize Databricks and Apache Spark to extract, transform, and load data into Azure Data Lake Storage and other Azure services.
• Develop and maintain secure data warehouses and data lakehouses: Implement data models, data quality checks, and governance practices to ensure reliable and accurate data.
• Build and deploy AI/ML models: Integrate Machine Learning into data pipelines, leverage Databricks ML and Azure ML to develop predictive models and drive business insights.
• Monitor and optimize data pipelines and infrastructure: Analyze performance metrics, identify bottlenecks, and implement optimizations for efficiency and scalability.
• Collaborate with cross-functional teams: Work closely with business analysts, data scientists, and DevOps engineers to ensure successful data platform implementations.
• Stay ahead of the curve: Continuously learn and adapt to the evolving landscape of big data technologies and best practices.