Lead Data Engineer

at  DL Remote

We are currently staffing a permanent, Lead Data Engineer role @ a leading energy solutions company in Hamburg.
In this role, you will join the Analytics & AI team. As part of this team, you will oversee the development and management of sophisticated data processing systems in the data platform on Azure Cloud. This role involves advancing data solutions on Databricks, optimizing data acquisition via Azure Data Factory, Logic Apps, Azure Functions, Spark and ensuring quality assurance for data pipelines.
You should have 5+ years of experience in Data Engineering, some (lateral) team leadership, and in-depth Azure Data Data Factory & Databricks experience.
Furthermore, you should have high-level Python and SQL skills and a comprehensive understanding of Big Data technologies like Hadoop, Spark or related distributed computing frameworks.

IN THIS ROLE, YOU WILL WORK ON THE FOLLOWING TOPICS:

Technical Leadership and Best Practices
Provide strong technical leadership and guidance to a team of data engineers, ensuring adherence to best practices in data engineering, cloud computing, and data security.
Streamlining Data Acquisition
Optimize data acquisition processes using Azure Data Factory, Logic Apps, and Azure Functions to improve efficiency and accuracy.
Quality Assurance
Implement robust validation, error handling, and performance consistency across various data sources and processes to ensure the highest quality of data pipelines.
Scalability and Performance
Oversee and enhance the scalability of data processing systems, ensuring they can manage increasing data volumes and complex processing needs while maintaining high performance.
Cost Optimization
Continuously monitor and optimize cloud resource usage and expenses, implementing cost-effective solutions without compromising performance and scalability.
Automation and CI/CD
Implement and maintain continuous integration and deployment (CI/CD) practices for data pipelines to improve efficiency, reduce manual errors, and accelerate deployment processes.