Senior Data Engineer

at  Retail Trust

REQUIREMENTS:

We are looking for individuals with strong problem-solving skills, attention to detail, and excellent communication and collaboration abilities. An analytical mindset is essential, along with the ability to adapt to changing requirements and work in a small team in a fast-paced environment.

• Proven experience as a Senior Data Engineer, with a focus on Databricks and Azure, is essential. Proficiency in Python and SQL, strong knowledge of Spark, and experience with automated deployment tools like Azure DevOps and Terraform are required.
• Familiarity with Azure Data Lake Storage is crucial. We are looking for a T-shaped data engineer who is happy to work on all aspects of the data platform. A passion for using technology and data to impact others is highly valued.
• You will work with Databricks, including Workflows/Jobs, DLT Pipelines, Delta Live Tables (DLT), automated deployment, architecture, and security.
• Programming languages and tools such as Python, PySpark, Pandas, Pytest, and SQL will be used.
• You will be involved in data modelling concepts like schema design and ETL/ELT processes.
• The role involves working with cloud technologies, specifically Azure, and using version control systems like Git. CI/CD pipelines and collaboration tools such as Azure ADO, Slack, and Teams.
• Experience with Dynamic CRM along with advanced analytics and machine learning in Databricks are all advantageous.

ARE YOU LOOKING TO WORK FOR AN ORGANISATION WITH A REAL PURPOSE – ONE WHERE THERE IS NO LIMIT TO FRESH IDEAS AND DISCOVERING WHAT’S NEXT?

As the charity for everyone in retail, our cause is to create, hope and health and happiness for everyone in retail. Yes, that’s right, everyone. That’s more than three million retail colleagues and their families. And you can help to achieve that mission.
We’re revolutionising the way data drives business insights and innovation to support more people to be happy at work. Our cutting-edge data platform, built on Databricks and Azure, is the backbone of our data-driven decision-making. This platform already leverages AI to generate insights for retailers and is built to be carbon-efficient, supporting our commitment to sustainability.
Join us and be part of a team that’s pushing the boundaries of technology to deliver real-world impact and make the retail industry a happy and healthy place to work.

As a Senior Data Engineer, you’ll be at the forefront of our Databricks-based data platform. You’ll manage automated deployments across multiple workspaces, pipelines, and workflows using Databricks SQL, Python notebooks, and Delta Live Tables (DLT). Your expertise will ensure smooth data operations by tracking down and resolving issues within Databricks notebooks and codebase.

• You’ll streamline the ingestion of new data sources and build pipelines to integrate additional data into existing models. This will likely include delta live sharing with other organisations and leading a platform migration to Unity Catalogue. Comprehensive documentation of data pipelines, workflows, and processes will be maintained by you, ensuring clarity and continuity.
• You’ll implement top-notch data governance and security measures, including data quality checks, lineage tracking, and compliance with relevant standards. Monitoring, managing, and optimising data storage and compute resources for peak performance, cost-efficiency, and carbon impact will be a key part of your role.
• You will carry out day-to-day activities to run the operational data platform, including performing daily checks to ensure the platform is running smoothly, onboarding new clients and offboarding existing clients, fixing bugs and addressing issues as they arise and over time, automate manual tasks for business efficiency