Lead Data Engineer

at  ITV Jobs

Skills you’ll need (minimum criteria)

• Experience designing and building Databricks data products
• Strong programming skills in languages such as Python (PySpark preferred), Scala, or SQL
• Experience in owning, designing and implementing data pipelines ingesting enterprise levels of data volume with strong knowledge of data engineering concepts, data integration, and ETL processes
• Ability to write production-grade code including automated testing and able to deploy code via CI/CD platforms (e.g. Github Actions, Jenkins)
• Experience in Proficiency in working with distributed computing frameworks, such as Apache Spark and data modelling, database systems, and SQL optimisation techniques
• Experience with cloud platforms (e.g., AWS, Azure, Google Cloud) and associated services (e.g., S3, Redshift, BigQuery

• Provide technical leadership by overseeing productivity, setting objectives and assessing the performance of a few data engineers.
• Design, develop and maintain scalable data pipelines and products using Databricks, including efficient data ingestion, processing and storage as well as prototyping new approaches
• Develop and optimise ELT processes for smooth data flow into the platform and perform data cleansing, validation and enrichments for accurate analysis
• Ensure adherence to data architecture principles and best practices by collaborating with data scientists to define requirements, design data models and optimise storage and partitioning for efficient retrieval and analysis
• Monitor and optimise data pipelines for high performance and reliability, resolving bottlenecks, data quality issues and interaction challenges
• Define data quality characteristics for your products and collaborate with the QA team to implement automated processes for measuring and reporting data correctness through our observability infrastructure

Skills you’ll need (minimum criteria)

• Experience designing and building Databricks data products
• Strong programming skills in languages such as Python (PySpark preferred), Scala, or SQL
• Experience in owning, designing and implementing data pipelines ingesting enterprise levels of data volume with strong knowledge of data engineering concepts, data integration, and ETL processes
• Ability to write production-grade code including automated testing and able to deploy code via CI/CD platforms (e.g. Github Actions, Jenkins)
• Experience in Proficiency in working with distributed computing frameworks, such as Apache Spark and data modelling, database systems, and SQL optimisation techniques
• Experience with cloud platforms (e.g., AWS, Azure, Google Cloud) and associated services (e.g., S3, Redshift, BigQuery)

Other things we’re looking for (key criteria)

• Knowledge of the UK broadcast industry & broadcast/OTT advertising market and digital marketing and advertising industry
• Familiar with: Data design techniques for classification and regression as well as knowledge of A/B testing, experimental design, and general statistical modelling
• Architecture disciplines such as Data Mesh Architecture, Data Architecture, BI Architecture and Enterprise Architecture
• Tools across other data management capabilities – data engineering, data governance, data quality, metadata management and master data management
• Designing and building out data visualisation solutions based on technology such as: Tableau, AWS QuickSight, Looker, ThoughtSpot