Data Engineer at Thrive IT Systems
Dublin, Dublin County, Ireland -
Full Time


Start Date

Immediate

Expiry Date

02 Feb, 27

Salary

55000000.0

Posted On

04 Nov, 26

Experience

1 year(s) or above

Remote Job

Yes

Telecommute

Yes

Sponsor Visa

No

Skills

Industry

Information Technology & Services

Description

About the job

  • We are looking for Data Engineer who can connect technical fundamentals with real project delivery. The candidate should be able to write PySpark and SQL code independently, explain Spark execution and performance concepts, describe an end-to-end AWS-based ETL pipeline, and troubleshoot hypothetical production failures in a structured manner.Must-Have SkillsETL and end-to-end data pipeline understanding
  • Strong hands-on PySpark and independent coding ability
  • Strong SQL fundamentals and independent query writing
  • Apache Spark architecture, execution and optimization concepts
  • AWS understanding for data engineering workloads
  • Hands-on AWS Glue Job knowledge
  • Python fundamentals
  • Relevant past Data Engineering project experience
  • Structured troubleshooting and scenario-based problem solving
  • Good-to-Have SkillsDatabricks
  • Hadoop ecosystem, including HDFS and Hive


  • 2. PySpark - Must HaveCandidates should be able to receive a problem statement and produce working DataFrame-based PySpark code with minimal prompting. Assessment should cover both coding correctness and awareness of execution cost.Core Coding TopicsDataFrame API: select, filter, withColumn, groupBy and aggregations
  • Joins: inner, left, right, full outer, left semi and left anti
  • Window functions and ranking logic
  • NULL handling and duplicate removal
  • union and unionByName
  • Reading and writing common formats, especially Parquet
  • Partitioning, repartition and coalesce
  • Caching and persistence
  • Transformations and actions
  • Lazy evaluation
  • Performance and Optimization TopicsNarrow vs wide transformations
  • Shuffle: why it happens and how to reduce unnecessary shuffle
  • Data skew and salting
  • Broadcast join and when it is appropriate
  • Join strategy selection
  • Partition sizing and data distribution
  • Executor/resource awareness
  • Identifying slow stages and high-cost operations


  • Spark Architecture - Must HaveCandidates should understand what happens behind PySpark code and should be able to connect architecture to troubleshooting and performance decisions.Architecture and Execution TopicsDriver and executors
  • Cluster manager
  • SparkSession and SparkContext awareness
  • Jobs, stages and tasks
  • DAG and lineage
  • Lazy evaluation
  • Transformations vs actions
  • Narrow vs wide transformations
  • Partitions and parallelism
  • Shuffle operations
  • Executor memory and cores
  • Caching and persistence
  • Serialization awareness
  • Spark UI and execution metrics


  • . SQL - Must HaveCandidates should have strong SQL fundamentals and should be able to write complete queries independently during the interview.SQL FundamentalsSELECT, WHERE, ORDER BY and DISTINCT
  • GROUP BY, HAVING and aggregate functions
  • CASE WHEN and NULL handling
  • Inner, left, right, full and self joins
  • Subqueries and Common Table Expressions (CTEs)
  • Window functions: ROW_NUMBER, RANK, DENSE_RANK, LEAD and LAG
  • UNION vs UNION ALL
  • Date functions
  • Duplicate identification and removal
  • Primary and foreign key fundamentals
  • Basic query optimization awareness
  • Sample Coding ChecksFind the latest transaction for each customer.
  • Identify duplicates using multiple business-key columns.
  • Return the second-highest value with a clear tie-handling rule.
  • Calculate a running total by customer and transaction date.
  • Compare source and target tables to identify inserts and updates.



  • 5. ETL and End-to-End Pipeline - Must HaveThe candidate should be able to explain a real production pipeline, including their own contribution, operating model, failure handling, and business purpose.ETL TopicsETL vs ELT
  • Source ingestion and target loading
  • Full load vs incremental load
  • Batch processing
  • Data validation and data quality
  • NULL and duplicate handling
  • Schema management and schema changes
  • Upsert logic and change handling
  • Error/reject handling
  • Pipeline dependencies and orchestration
  • Logging, monitoring and alerting
  • Restart, recovery and reprocessing
  • Idempotent pipeline design
  1. Expected End-to-End ExplanationBusiness problem and expected output
  2. Source systems and file/table formats
  3. Data volume and arrival pattern
  4. Ingestion mechanism
  5. Storage layers and S3 paths
  6. PySpark and SQL transformations
  7. Data quality controls
  8. AWS Glue Job configuration and scheduling/orchestration
  9. Output target and downstream consumers
  10. Monitoring, logging, alerts and operational support
  11. Failure recovery and safe rerun approach
  12. Candidate's personal responsibilities and design decisions


  • AWS and AWS Glue, Lamda, EMR and other - Must HaveAWS knowledge should be assessed from a Data Engineering perspective. The candidate should understand how AWS services participate in the pipeline and should be able to diagnose Glue Job failures.AWS UnderstandingAmazon S3 storage and data organization
  • Lamda, Limitations of lambda
  • AWS Glue and PySpark processing
  • AWS Glue Data Catalog awareness
  • Glue Crawlers awareness
  • IAM roles, permissions and access fundamentals
  • Amazon CloudWatch logs and monitoring
  • How S3, Glue, IAM and catalog metadata interact within a pipeline
  • AWS Glue Job TopicsReading input data from Amazon S3
  • Applying PySpark transformations
  • Writing processed data to the target location
  • Job arguments and runtime configuration
  • Job bookmarks / incremental-processing awareness
  • Glue Data Catalog integration
  • Logging and monitoring
  • Worker/resource awareness
  • Handling schema or data-format changes
  • Investigating failed and slow Glue Jobs
  • AWS Glue Scenario QuestionsA Glue Job that succeeded yesterday fails today. What would you check first?
  • The job cannot read from or write to an S3 path. How would you isolate the issue?
  • Input volume increases significantly and the job exceeds its normal runtime. What would you investigate?
  • A source column changes data type and the transformation fails. How would you handle the incident and prevent recurrence?
  • The job completes but expected output is missing or incomplete. How would you validate each pipeline stage?
  1. Expected Troubleshooting ApproachRead the exact error and review CloudWatch/Glue logs
  2. Confirm input arrival, path, format and schema
  3. Validate IAM role and S3 permissions
  4. Check code, arguments, catalog metadata and recent deployment changes
  5. Review data volume, partitioning, skew, shuffle and resource pressure
  6. Validate output paths, write mode and partial writes
  7. Explain recovery, rerun safety and prevention measures


  • Python - Must HavePython assessment should focus on practical fundamentals needed to produce maintainable data engineering code. Easy-to-medium coding questions are appropriate.TopicsData types and variables
  • Lists, tuples, sets and dictionaries
  • Loops and comprehensions
  • Functions and lambda functions
  • String manipulation
  • Exception handling
  • File-handling basics
  • Modules and packages basics
  • OOP: classes, objects, inheritance, encapsulation and polymorphism
  • Example Coding ChecksCount the frequency of each item in a list.
  • Remove duplicates while preserving input order.
  • Find the second-largest distinct number.
  • Group records by a key using a dictionary.
  • Create a class that demonstrates simple inheritance and method overriding.


How To Apply:

Incase you would like to apply to this job directly from the source, please click here

Responsibilities
Loading...