· Create business models, logical specifications and/or user requirements to develop solutions for the application environment–10%
· Created reusable, parameterized Databricks notebooks for ingestion, enrichment, and validation stages across multiple data domains.
· Created and maintained dimensional models (Star and Snowflake schemas) in Synapse and Snowflake to support analytical workloads.
· Built custom connectors and ingestion logic for sources such as PostgreSQL, MySQL, REST APIs, and Blob Storage.
· Design software applications, create system procedures and ensure that the developed applications function normally- 10%
· Designed and implemented scalable ETL pipelines using Apache Spark with Scala on Azure Databricks to process high-volume structured and unstructured data.
· Built and orchestrated complex data workflows in Azure Data Factory (ADF) for ingesting and transforming data from various on-prem and cloud sources.
· Collaborated with cross-functional DevOps, QA, and architecture teams to ensure scalable, resilient, and maintainable pipeline design.
· Develop and create business models, logical specifications and/or user requirement solutions for the application environment-15%
· Developed real-time ingestion pipelines using Apache Kafka and Spark Structured Streaming to support streaming data requirements.
· Developed CI/CD pipelines in Azure DevOps for seamless deployment and version control of ETL and Spark jobs.
· Developed and deployed Java-based microservices using Spring Boot to process real-time messages from Kafka into ADLS and Snowflake.
· Implement and modify programs; make approved changes by amending flow charts, develop detailed programming logic, and coding changes-15%
· Implemented historical and incremental data loading strategies using ADF and Spark for large-scale data warehouse migrations.
· Implemented data validation logic and automated quality checks using PySpark, ensuring pipeline reliability.
· Implemented logging and monitoring using Azure Monitor, Log Analytics, and custom logging frameworks for proactive issue resolution.
· Participate in scrum meetings and coordinate with Business Analysts to understand the business needs and implement the same into a functional design-20%
· Participated in Agile delivery processes, including sprint planning, daily standups, retrospectives, and task estimation using JIRA.
· Utilized Azure Data Lake Storage Gen2 (ADLS) for secure, high-performance storage, supporting curated data layers for analytics.
· Configured role-based access control, secure credential handling, and integration with Azure Key Vault across all pipeline components.
· Write source code, prepare test data, tests and debug programs; revise and refine programs to improve performance of the application software-15%
· Configured and managed Snowflake pipelines using SnowSQL and Snowpipe for curated data processing and downstream BI access.
· Engineered Delta Lake solutions on Databricks to support ACID transactions, time travel, and efficient batch updates.
· Migrated legacy batch ETL processes from on-prem Hadoop systems using tools like Hive, Sqoop, and HDFS to cloud-native alternatives.
· Perform execution of functional test plan, validate test results, prepare documentation & data for analysis-15%
· Performed complex data transformations using Spark SQL, optimizing partitioning and caching strategies to improve performance and reduce compute costs.
· Deployed big data solutions across multiple clouds including Azure, AWS (S3, Glue), and GCP (BigQuery, Cloud Storage).
· Tuned Spark jobs for performance by optimizing memory, broadcast joins, and data shuffling across large distributed clusters.