Data Engineer at Top Consulting Group Inc
Victoria, Victoria, Australia -
Full Time


Start Date

Immediate

Expiry Date

12 Feb, 27

Salary

50000.0

Posted On

14 Nov, 26

Experience

2 year(s) or above

Remote Job

Yes

Telecommute

Yes

Sponsor Visa

No

Skills

Industry

Information Technology & Services

Description


Key Responsibilities

Design, develop, and maintain scalable ETL pipelines using AWS Glue, EMR (PySpark), and Lambda functions

Implement real-time data streaming solutions using Confluent Kafka with source and sink connectors

Develop and optimize PySpark jobs on EMR clusters for large-scale data transformations

Build serverless data processing workflows using AWS Lambda

Design and optimize queries in Amazon Athena for analytical workloads

Manage data warehouse solutions in Amazon Redshift including data modeling and performance tuning

Implement monitoring and alerting using CloudWatch for data pipeline health and performance

Provision and manage AWS infrastructure using Terraform

Collaborate with data analysts, data scientists, and business stakeholders to understand data requirements

Implement data governance and security best practices using Lake Formation, IAM, and KMS


Additional Technical Experience

Knowledge of AWS EventBridge and Step Functions for workflow orchestration

Familiarity with EMR runtime roles and execution roles for secure data access

Experience with DynamoDB for NoSQL data storage

Knowledge of API Gateway for building data APIs


Preferred Qualifications

Bachelor's degree in Computer Science, Data Engineering, Information Technology, or related field

Strong understanding of distributed computing and big data processing frameworks

Experience with Agile development methodologies

Excellent problem-solving and analytical skills

Strong communication and collaboration abilities

How To Apply:

Incase you would like to apply to this job directly from the source, please click here

Responsibilities


Key Responsibilities

Design, develop, and maintain scalable ETL pipelines using AWS Glue, EMR (PySpark), and Lambda functions

Implement real-time data streaming solutions using Confluent Kafka with source and sink connectors

Develop and optimize PySpark jobs on EMR clusters for large-scale data transformations

Build serverless data processing workflows using AWS Lambda

Design and optimize queries in Amazon Athena for analytical workloads

Manage data warehouse solutions in Amazon Redshift including data modeling and performance tuning

Implement monitoring and alerting using CloudWatch for data pipeline health and performance

Provision and manage AWS infrastructure using Terraform

Collaborate with data analysts, data scientists, and business stakeholders to understand data requirements

Implement data governance and security best practices using Lake Formation, IAM, and KMS


Additional Technical Experience

Knowledge of AWS EventBridge and Step Functions for workflow orchestration

Familiarity with EMR runtime roles and execution roles for secure data access

Experience with DynamoDB for NoSQL data storage

Knowledge of API Gateway for building data APIs


Preferred Qualifications

Bachelor's degree in Computer Science, Data Engineering, Information Technology, or related field

Strong understanding of distributed computing and big data processing frameworks

Experience with Agile development methodologies

Excellent problem-solving and analytical skills

Strong communication and collaboration abilities

Loading...