Senior Data Engineer

at  XTY AI PTE LTD

JOB SUMMARY:

We are seeking an experienced Senior Solutions Consultant to lead the design, development, and implementation of large-scale data pipelines and architectures. The successful candidate will have 8+ years of hands-on experience in data engineering, with expertise in Spark, Python, SQL, and cloud-based data integration services.

Requirements :

• 8+ years of hands-on experience in data engineering with expertise in Spark, Python, SQL, and AWS/Azure/GCP Cloud Data Services.
• Familiarity with cloud-based data services such as Databricks, Azure Synapse, Azure Data Factory, Amazon Redshift, AWS Glue etc.
• Familiarity with data visualisation tools such as Power BI, Tableau, or Databricks.
• Experience in developing and deploying large-scale data processing systems using technologies like Apache Spark, Hadoop, or similar distributed computing frameworks.
• Strong understanding of database concepts, including schema design, indexing, caching, and query optimization.
• Proficiency in programming languages such as Python, Java, or Scala.
• Excellent problem-solving skills with ability to work under pressure to meet deadlines.
• Strong communication and collaboration skills to work effectively with cross-functional teams.
• Experience with cloud-based security, governance, and compliance tools

• Design, develop, and implement scalable and efficient data processing systems using technologies such as Databricks, Spark and Azure Synapse.
• Collaborate with cross-functional teams to gather requirements, define data models, and ensure data quality and integrity.
• Develop and maintain complex data pipelines that handle large volumes of data from various sources, including relational databases, NoSQL databases, and cloud storage.
• Implement data integration processes using cloud-based data integration services.
• Design and implement real-time streaming data processing systems using Apache Kafka, Apache Flink, or similar technologies.
• Develop and deploy data quality and validation processes to ensure accuracy and consistency of data.
• Optimize data storage and retrieval performance by designing efficient schema, indexing, and caching strategies.
• Troubleshoot complex data engineering issues and provide technical leadership in resolving them.

Requirements :

• 8+ years of hands-on experience in data engineering with expertise in Spark, Python, SQL, and AWS/Azure/GCP Cloud Data Services.
• Familiarity with cloud-based data services such as Databricks, Azure Synapse, Azure Data Factory, Amazon Redshift, AWS Glue etc.
• Familiarity with data visualisation tools such as Power BI, Tableau, or Databricks.
• Experience in developing and deploying large-scale data processing systems using technologies like Apache Spark, Hadoop, or similar distributed computing frameworks.
• Strong understanding of database concepts, including schema design, indexing, caching, and query optimization.
• Proficiency in programming languages such as Python, Java, or Scala.
• Excellent problem-solving skills with ability to work under pressure to meet deadlines.
• Strong communication and collaboration skills to work effectively with cross-functional teams.
• Experience with cloud-based security, governance, and compliance tools.