EOI - Azure Databricks, Data Engineers & Data Architects

at  Hudson Australia

POSITION OVERVIEW:

We are inviting expressions of interest from experienced Data Engineers and Data Architects with expertise in Azure and Databricks. These expressions of interest will be considered for upcoming large projects of work that Hudson clients have engaged us for in 2025.
Roles include a variety of Permanent, Contract and Fixed term contract opportunities with varying timelines for hire over the next 6 months. All of these opportunities require varying skill levels of Azure technologies and Databricks.
All opportunities will require atleast permanent residency or citizenship. With some roles requiring eligibility for baseline clearance. Candidates with active baseline clearance will be looked upon favourably.

SKILLS AND QUALIFICATIONS:

• Proven experience in designing, building, and managing data solutions on Azure and Databricks.
• Strong proficiency in Azure services such as Azure Data Factory, Azure Synapse Analytics, Azure Data Lake, and Azure SQL.
• Expertise in Databricks, including experience with Spark, Delta Lake, and MLflow.
• Proficiency in programming languages such as Python, Scala, or SQL.
• Deep understanding of data modeling concepts, ETL/ELT pipelines, and data warehousing principles.
• Experience with CI/CD pipelines, version control, and infrastructure as code (e.g., Terraform, GitHub).
• Familiarity with data security, compliance standards and best practices.
• Strong problem-solving skills, with the ability to work in fast-paced environments and manage competing priorities.

PREFERRED QUALIFICATIONS:

• Certifications in Azure (e.g., Azure Data Engineer, Azure Solutions Architect) or Databricks.
• Understanding of Machine Learning and AI for system and process automation
• Experience in building batch and real-time data streaming solutions with technologies such as Kafka or Event Hubs.
• Knowledge of data visualization tools like Power BI or Tableau.
• Hands-on experience with DevOps practices and cloud cost optimization.
• Financial services experience desirable

ROLE TYPES:

Data Engineer
Senior Data Engineer
Data Migration specialist
Senior Databricks Specialist
Data Architect
Data/Solutions Architect
Senior Data Architect

DATA ENGINEER RESPONSIBILITIES:

• Design, build, and maintain scalable data pipelines and ETL processes on Azure and Databricks.
• Collaborate with cross-functional teams to understand data requirements and deliver optimized solutions.
• Implement data integration and transformation solutions to support analytics and reporting.
• Develop and maintain high-performance data models and queries.
• Monitor, troubleshoot, and improve existing data workflows to ensure efficiency and reliability.
• Ensure data quality, security, and compliance with organizational and regulatory standards.

DATA ARCHITECTURE RESPONSIBILITIES:

• Design and oversee the implementation of end-to-end data architecture solutions on Azure and Databricks.
• Define best practices, guidelines, and standards for data modeling, storage, and integration.
• Lead the evaluation and selection of appropriate technologies and tools to meet business needs.
• Collaborate with stakeholders to design solutions that align with strategic goals and performance requirements.
• Establish governance frameworks to ensure data consistency, accuracy, and security.
• Mentor and guide engineering teams to adhere to architectural principles and best practices.