US
ID asp-002-0825
Full Time
04/08/2025
Hourly rate: $70/hour to $80/hour on W2
Location: St. Louis, MO (Onsite)
Key Skills: Azure, Databricks, ADF

REQUIRED SKILLS AND QUALIFICATIONS:

• 8+ years of experience in data engineering.
• Strong hands-on experience with Azure Databricks, Apache Spark, and Delta Lake.
• Proven experience integrating and managing streaming data using Apache Kafka.
• Experience using Azure Data Factory for orchestrating and scheduling ETL/ELT processes.
• Familiarity or experience with Stelo for metadata-driven data movement (or similar tools).
• Proficiency in Python, SQL, and optionally Scala.
• Good understanding of data warehousing concepts, data lakes, and big data architecture.
• Knowledge of Azure ecosystem including ADLS, Synapse, Key Vault, etc.
• Excellent analytical, problem-solving, and communication skills.

• Lead the design, development, and deployment of scalable data pipelines using Azure Databricks.
• Integrate real-time and batch data sources using Apache Kafka and ADF.
• Implement data transformations, enrichment, and quality checks using PySpark/Scala within the Databricks environment.
• Design and develop integrations with Stelo for structured data loading and transformation (if applicable).
• Collaborate with data scientists, analysts, and other engineering teams to deliver business-critical insights.
• Optimize pipeline performance, cost, and scalability in the Azure cloud environment.
• Define best practices for data ingestion, processing, storage, and governance.
• Lead and mentor a team of data engineers, conducting code reviews and ensuring development standards.
• Support troubleshooting and incident management for data-related issues in production.