Databricks Lead Data Engineer

at  K2 Insurance Services LLC

QUALIFICATIONS:

• Proven experience (5+ years) in data engineering, with 2+ years as a team lead or equivalent role.
• Expertise in Databricks, including Spark, Delta Lake, and MLFlow.
• Proficiency in Python, SQL, and at least one cloud platform (Azure, AWS, or GCP).
• Hands-on experience with data lakes, data warehouses, and big data technologies.
• Strong understanding of CI/CD pipelines and infrastructure as code (IaC).
• Excellent problem-solving and communication skills.
• Prior insurance industry experience is a strong plus.

Preferred Skills:

• Familiarity with Databricks Unity Catalog and Lakehouse architecture.
• Experience with Airflow or other workflow orchestration tools.
• Knowledge of machine learning pipelines and integrations.
• Certifications in Databricks, Azure, AWS, or GCP.
• Previous IT product owner role.

• Leadership & Strategy:
• Lead and mentor a team of data engineers in implementing Databricks-based solutions.
• Define and drive the data engineering strategy, ensuring alignment with business goals.
• IT Product Owner role for organizational data warehouse project.
• Databricks Expertise:
• Design and develop scalable data pipelines and ETL processes using Databricks.
• Optimize and tune Spark jobs for performance and efficiency.
• Develop and enforce best practices for Databricks cluster management and data security.
• Data Architecture:
• Build and maintain robust data models to support analytics and reporting needs.
• Integrate Databricks with various data sources (cloud storage, databases, APIs).
• Implement Delta Lake for reliable, scalable, and performant data lakes.
• Collaboration:
• Work closely with data scientists, analysts, and stakeholders to deliver actionable insights.
• Act as the technical liaison between the data engineering team and other stakeholders.
• Innovation & Optimization:
• Stay updated on the latest Databricks and Azure features to drive continuous improvements.
• Automate repetitive processes and streamline data workflows