JOB SUMMARY:

We are seeking an experienced Databricks Data Engineer to join our analytics team for a 6-month contract. You will architect, develop, and optimize scalable data pipelines and data architectures on the Databricks Lakehouse Platform to support advanced analytics and AI-driven business initiatives. This role demands deep expertise in big data processing, cloud data engineering, and emerging technologies including generative AI applications.

REQUIRED TECHNICAL SKILLS:

• Minimum 5 years hands-on experience as a Data Engineer with strong focus on Databricks platform and Apache Spark ecosystem.
• Proficiency in Python (PySpark), Scala or Java for large-scale data processing.
• Expert knowledge of SQL including complex query optimization in EDM and Data Lake environments.
• Experience integrating cloud data storage and services (Azure Blob Storage, ADLS, AWS S3, Snowflake).
• Solid understanding of data warehousing, lakehouse architecture, and ETL/ELT methodologies.
• Expertise in data governance and data catalog tools, Unity Catalog preferred.
• Familiarity with workflow orchestration tools such as Apache Airflow or Prefect.
• Experience with CI/CD pipeline concepts and tools (Git, Jenkins, Azure DevOps).
• Strong background in distributed systems and scalable data architecture design.

PREFERRED BONUS SKILLS:

• Working knowledge of Generative AI frameworks and applications (e.g., OpenAI GPT, Azure OpenAI Service).
• Experience deploying AI/ML models on Databricks and integrating AI workflows within data pipelines.
• Knowledge of AI data labeling, data augmentation, and responsible AI practices.
• Familiarity with Kubernetes and containerization to support scalable AI infrastructure.
• Hands-on experience with modern AI platforms and frameworks (TensorFlow, PyTorch, MLflow).

QUALIFICATIONS:

• Bachelor’s degree in Computer Science, Engineering, Data Science, or related field. Advanced degrees preferred.
• Relevant certifications such as Databricks Certified Data Engineer Associate or Azure Data Engineer are a plus.
• Proven ability to work independently and collaboratively in Agile environments with remote/distributed teams.
  Job Types: Full-time, Freelance
  Pay: $90.00-$100.00 per hour

Education:

• Bachelor’s Degree (preferred)

Experience:

• Databricks: 3 years (required)

Licence/Certification:

• Databricks Data engineer (required)

Work Location: Hybrid remote in Laval, QC H7T 1C7
Application deadline: 2025-08-2

• Design, build, and maintain robust, scalable ETL/ELT data pipelines leveraging Databricks and Apache Spark frameworks.
• Collaborate closely with Data Scientists, AI Engineers, and business analysts to understand data requirements and enable AI/ML model deployment.
• Implement data ingestion, processing, transformation, and integration from diverse cloud-based sources (Azure, AWS, GCP) using Python, SQL, and PySpark.
• Optimize data pipelines for performance, cost-efficiency, and reliability; proactively troubleshoot and resolve bottlenecks.
• Enforce data governance, security, and compliance best practices implementing tools like Unity Catalog for data cataloging and lineage.
• Develop automation for workflow orchestration (e.g., Apache Airflow, Databricks Jobs API) and CI/CD pipeline integration for continuous deployment.
• Document architecture, system design, and data pipeline specifications to support knowledge sharing and compliance.
• Stay abreast of the latest advancements in data engineering, big data platforms, and generative AI technologies to drive innovation within D2D Analytics.