• Key ResponsibilitiesBuild and maintain ETL/ELT pipelines using Databricks, Apache Spark and Delta Lake
• Develop and support bronze, silver and gold layer data models within a Lakehouse architecture
• Create and manage jobs and workflows using Lakeflow Jobs and Spark Declarative Pipelines (formerly Delta Live Tables)
• Write clean, efficient PySpark and SQL code for data transformation and processing
• Optimise Spark job performance through partitioning, caching and cluster configuration
• Implement data quality checks and basic monitoring/alerting for pipeline health
• Support data governance and access management using Unity Catalog, including permissions, lineage and catalog/schema structure
• Participate in code reviews, follow CI/CD practices and manage deployments using Declarative Automation Bundles (formerly Databricks Asset Bundles)
• Troubleshoot pipeline failures and data quality issues, escalating complex problems as needed
• Document pipelines, data models and processes for team and stakeholder reference
• Collaborate with analysts and data scientists to understand data requirements and deliver fit for purpose datasets

Technical Skills Required

• 5 to 7 years of experience in data engineering, including at least 2 to 4 years working directly with Databricks
• Strong practical PySpark and SQL skills for data transformation
• Understanding of Delta Lake and medallion (bronze/silver/gold) architecture
• Familiarity with the Databricks platform, including Lakeflow, declarative pipeline based ETL tooling, job orchestration and Unity Catalog
• Experience with at least one cloud platform (AWS, Azure or GCP)
• Experience with Git based version control and CI/CD workflows using Databricks Asset Bundles
• Understanding of data modelling and ETL/ELT design principles
• Exposure to or interest in AI assisted workflows on the platform, such as setting up Genie spaces or building simple agents with Agent Bricks

• Key ResponsibilitiesBuild and maintain ETL/ELT pipelines using Databricks, Apache Spark and Delta Lake
• Develop and support bronze, silver and gold layer data models within a Lakehouse architecture
• Create and manage jobs and workflows using Lakeflow Jobs and Spark Declarative Pipelines (formerly Delta Live Tables)
• Write clean, efficient PySpark and SQL code for data transformation and processing
• Optimise Spark job performance through partitioning, caching and cluster configuration
• Implement data quality checks and basic monitoring/alerting for pipeline health
• Support data governance and access management using Unity Catalog, including permissions, lineage and catalog/schema structure
• Participate in code reviews, follow CI/CD practices and manage deployments using Declarative Automation Bundles (formerly Databricks Asset Bundles)
• Troubleshoot pipeline failures and data quality issues, escalating complex problems as needed
• Document pipelines, data models and processes for team and stakeholder reference
• Collaborate with analysts and data scientists to understand data requirements and deliver fit for purpose datasets

Technical Skills Required

• 5 to 7 years of experience in data engineering, including at least 2 to 4 years working directly with Databricks
• Strong practical PySpark and SQL skills for data transformation
• Understanding of Delta Lake and medallion (bronze/silver/gold) architecture
• Familiarity with the Databricks platform, including Lakeflow, declarative pipeline based ETL tooling, job orchestration and Unity Catalog
• Experience with at least one cloud platform (AWS, Azure or GCP)
• Experience with Git based version control and CI/CD workflows using Databricks Asset Bundles
• Understanding of data modelling and ETL/ELT design principles
• Exposure to or interest in AI assisted workflows on the platform, such as setting up Genie spaces or building simple agents with Agent Bricks