Data Engineer

at  Commercial Services Group

HYBRID/KINGS HILL, KENT

Commercial Services Group are looking for an experienced Data Engineer with strong analytical and problem-solving skills to design, build and maintain data pipelines using the Microsoft Azure toolset.
The successful candidate will be responsible for data ingestion, transformation, cleansing and quality assurance and governance, as well as collaborating with other data professionals and stakeholders to deliver data-driven solutions.

WHAT WE NEED FROM YOU -

• At least 3 years’ experience in Data Engineering
• Experience in using Microsoft Azure services for data engineering
• Proficient in SQL and Spark, and familiar with Python, Scala or R
• Experience with Azure data tools and frameworks
• Knowledge of data modelling, data warehousing and data quality concept and best practices
• Experience in using Azure DevOps or data pipeline project management

• Implement data quality, security and governance policies and procedures to ensure data integrity and compliance
• Manage data pipeline projects using Azure DevOps, including version control, code review, testing, and deployment
• Implement data ingestion strategies from various sources such as files, databases, APIs and streaming platforms
• Perform data transformation and cleansing using SQL, Python, Spark and other tools
• Ensure data quality and reliability using data validation, testing and monitoring techniques
• Optimise data pipeline performance, scalability and security
• Document data pipeline design, architecture and metadata