Principal Data Engineer

at  Ampersand

KEY ACCOUNTABILITIES INCLUDE:

• ETL Development: Create, optimize, and maintain Extract, Transform, Load (ETL) processes in Python (pySpark) and/or SQL to move and transform data from various sources to the target data warehouse or data lake, in Medallion architecture.
• Data Modeling: Design and implement data models to organize and structure data in the data warehouse or data lake. This includes defining tables, relationships, and ensuring data integrity.
• Cloud Platform Interaction: Work with cloud platforms like Azure, deploying and managing data solutions in the cloud. This may include setting up and configuring services like Azure Data Lake (Blob).
• Collaboration: Collaborate with data analysts, and other stakeholders to understand their data requirements and provide the necessary infrastructure and support.
• Troubleshooting: Diagnose and resolve issues related to data pipelines, database performance, or any other data-related challenges that may arise.
• Documentation: Maintain documentation for data processes, data models, and infrastructure configurations to facilitate knowledge sharing and future development.

Provide leadership to deliver expert advice and services in the design of data products and data-related services to support the achievement of business outcomes and strategic goals for the benefit of NSW citizens.
The role will also drive development of data processing frameworks and data architectural approaches scaled to current and future business demands.