REQUIRED QUALIFICATIONS

• Bachelor’s or Master’s degree in Computer Science, Engineering, or related field.
• 5+ years of experience in data engineering with at least 2 years on Databricks.
• Proficiency in Python, Scala, SQL, and Spark.
• Hands-on experience with Azure Data Services (ADF, ADLS, Synapse).
• Strong understanding of ETL, data warehousing, and data modeling concepts.
• Experience with Power BI, including DAX and advanced visualizations.
• Familiarity with MLflow, LangChain, and LLM integration is a plus.

ROLE OVERVIEW

We are seeking a highly skilled and experienced Senior Data Engineer to lead the design, development, and optimization of scalable data pipelines using Databricks on Azure. This role will be instrumental in driving data architecture initiatives, enhancing data quality, and enabling advanced analytics across the enterprise.

KEY RESPONSIBILITIES

• Databricks Development & Optimization: Build and optimize distributed data processing jobs using Apache Spark on Databricks. Implement Delta Lake, DLT pipelines, and Medallion architecture for scalable and reliable data workflows.
• ETL & Data Integration: Design and automate ETL pipelines using Azure Data Factory, Databricks, and Synapse Analytics. Integrate data from diverse sources including Salesforce, Workday, Duckcreek, and external APIs.
• Data Modeling & Warehousing: Develop dimensional models (Star/Snowflake schemas), stored procedures, and views for data warehouses. Ensure efficient querying and transformation using SQL, T-SQL, and PySpark.
• Cloud & DevOps Integration: Leverage Azure DevOps, CI/CD pipelines, and GitHub for version control and deployment. Utilize Azure Logic Apps and ML Flow for workflow automation and model training.
• Security & Governance: Implement role-based access control (RBAC), data encryption, and auditing mechanisms. Ensure compliance with enterprise data governance policies.
• Collaboration & Leadership: Work closely with data scientists, analysts, and business stakeholders to deliver high-quality data solutions. Mentor junior engineers and contribute to code reviews and architectural decisions.