Senior/Lead Data Engineer (Databricks)

at  VERSENT

Senior Engineer is responsible for driving the analysis, design, development and testing of a unified data infrastructure including activities such as data discovery; ingestion and orchestration; storage; query processing; transformation; observability; and analytics outputs.
Key Responsibilities:
Databricks environment set up:
Configure and optimise Databricks clusters to ensure optimal performance and scalability
Implement best practices for Databricks workspace organisation and resource management
Data integration and ETL:
Develop and optimise ETL pipelines using Databricks to ingest, transform, and load large volumes of data
Integrate Databricks with various data sources and destinations to support data stores and API integrations
Data processing and analysis:
Implement data transformations, aggregations, and statistical analyses as needed
Execute queries and data models against stored data including use of distributed compute
Performance tuning:
Identify and address performance bottlenecks in Databricks workflows and queries
Optimize Spark jobs and clusters for maximum efficiencies
Data quality and governance:
Implement data quality checks and ensure adherence to data governance policies
Write unit/integration tests along with relevant documentation of data standards
Collaboration and documentation:
Work closely with stakeholders, architects, analysts, and other engineers to understand data requirements, implement architectural recommendations and maintain release and deployment processes
Document Databricks workflows, processes, and best practices
Actively contribute to positive, goal-orientated delivery
Experience:
5+ years working on both AWS & Azure platforms
Direct customer engagement on Databricks environment
Skills & Requirements:
Proven experience as a Data Engineer with a focus on Databricks
Strong proficiency in Apache Spark, Spark SQL and PySpark
Experience with cloud platforms (especially Azure) and integration with Databricks
Expertise in designing and developing distributed data pipelines using big data technologies on large scale data sets including:
Collection and workflow of data
Integration of disparate data sources
Event streaming / stream processing
Data warehouse / Lakehouse architecture and design patterns
Application of CI/CD orchestration
Familiarity with data quality management and governance principles related to Databricks including Unity Catalog
Excellent problem-solving skills and attention to detail
Strong communication and collaboration skills
Qualifications
Information Systems, Computer Science Bachelor’s Degree or higher is preferred
Certifications (desirable)
Databricks Data Engineer Associate or Professional preferred
Our Culture
No cool cliques, just motivated passionate humans generating ideas and devising solutions in a creative, collaborative workspace. We’re building a team and looking for people who can think outside the square and deliver what others might consider impossible.
At Versent, we embrace the notion of Family: working together to achieve outstanding success.
What we offer you

At Versent we invest in our people, with benefits such as:

• Competitive salary package
• Working in a flexible, outcome-driven environment. Outside of client obligations, you can work from the office or home as needed. Also, leave the suit and tie behind and put on a Versent T-shirt instead.
• Weekly internal tech talks; all streamed and recorded so those who can’t make it to the office can still participate. For those who do make it, there’s pizza, beer, wine and soft drinks on tap
• Regular social events, meetups and guest speakers organised and facilitated by our team
• A supportive family culture - If you need help, just ask
• Celebration of the wins – big or small, because we know how hard you work.

And finally, we believe that every human is equal, across every dimension, and we function as an unbiased meritocracy. We strive to have Inclusion front and centre in how we operate and embed the fundamentals of tolerance and understanding in everything we do. This core belief is woven into all our policies.
Curious?
but honestly, we are more interested in you!

Please refer the Job description for details