bout the job

Senior Data Engineer

Duration: Contract

Workload: Full time hours

Setup: Day rate contract Inside IR35 / PAYE

Sponsorship: Not Provided

Location: London (3 days per week onsite)

Our client is seeking an experienced Senior Data Engineer to design and deliver scalable, cloud-based data solutions that support both real-time and batch analytics across the organisation.

Key Responsibilities:

• Design and build robust data pipelines using Databricks, Delta Lake, and Delta Live Tables (DLT).
• Develop and maintain streaming data architectures using technologies such as Spark Structured Streaming and Autoloader.
• Implement data ingestion and replication solutions using Azure Data Factory (ADF) and Qlik Replicate, including integration with SAP data sources.
• Create scalable data models and optimise data structures for reporting and analytics.
• Collaborate with stakeholders to gather requirements and deliver high-quality data solutions.
• Monitor, troubleshoot, and optimise data platforms for reliability, performance, and scalability.
• Contribute to technical documentation and data engineering best practices.

Must-Have Skills:

• Extensive experience with Databricks, including DLT, Delta Lake, and Unity Catalog governance.
• Strong expertise in streaming data solutions and real-time analytics.
• Advanced programming skills in Python, SQL, or Scala.
• Proven experience with ADF, Qlik Replicate, and SAP data integration.
• Strong knowledge of data modelling, ETL/ELT processes, and data architecture.
• Experience working within Azure cloud environments.
• Understanding of performance optimisation techniques including watermarking, checkpointing, parallelism, and late data handling.
• Experience with Microsoft Copilot, AI Foundry, and Databricks AI capabilities such as Mosaic AI, MLflow, Agent Bricks, and Genie.

Nice-to-Have:

• Experience working within large enterprise environments.
• Oil & Gas industry experience.
• Relevant Azure or Databricks certifications.
• Master's degree in Computer Science, Engineering, or a related discipline.