Technical Specialist -Databricks at Datavail Career Site

Mumbai, maharashtra, India -

Full Time

Start Date

Immediate

Expiry Date

15 Jun, 26

Salary

0.0

Posted On

17 Mar, 26

Experience

5 year(s) or above

Remote Job

Yes

Telecommute

Yes

Sponsor Visa

Skills

Databricks, PySpark, Spark SQL, Delta Lake, Unity Catalog, Spark Streaming, Python, SQL, ETL, ELT, Data Governance, CI/CD, Azure DevOps, AWS, Azure, Data Modeling

Industry

IT Services and IT Consulting

Description

Title: Technical Specialist Location: Mumbai Education: Bachelor’s Degree Job Description: * Build scalable ETL/ELT pipelines using Databricks (PySpark, SQL, Spark Streaming). * Develop and optimize Delta Lake tables, ACID transactions, schema evolution, and time travel. * Implement Unity Catalog, data governance, and access control.Optimize cluster configurations, job workflows, and performance tuning in Databricks. * Design and implement batch and streaming pipelines using Spark Structured Streaming. * Integrate Databricks with multiple data sources (RDBMS, APIs, cloud storage, message queues).Develop reusable, modular, and automated data processing frameworks. * Implement CI/CD pipelines for Databricks using GitHub Actions / Azure DevOps / GitLab.Automate cluster management and job orchestration using Databricks REST APIs. * Maintain code quality, unit tests, and documentation. * Write and optimize complex SQL queries and statements to ensure high performance and efficient data retrieval. * Strong database design including normalization, data modelling, and relational schema creation. * Conduct performance analysis, troubleshoot database issues like slow queries or deadlocks and implement solutions * Design and implement database structures, including tables, schemas, views, stored procedures, functions, and triggers. * Optimize database performance through query tuning, indexing, and performance analysis. * Ensure data integrity, security, and compliance standards * Need strong Python skills combined with expertise in Apache Spark for large scale data processing. Core abilities include building efficient ETL pipelines, optimizing distributed jobs, and handling large-scale data transformations * Expertise in Python programming, Spark APIs, and parallel processing. * Proficiency in Python (including Pandas, NumPy) for data manipulation and scripting * Deep knowledge of PySpark APIs like DataFrames, RDDs, Spark SQL for querying and processing. * Familiarity with RESTful APIs, batch processing, CI/CD, and monitoring data jobs. * Optimize Spark jobs for performance, troubleshoot issues, and ensure data quality across systems. * Collaborate with data engineers and scientists to implement workflows, conduct code reviews, and integrate with cloud platforms like AWS or Azure. * Design, develop, and maintain scalable data pipelines and ETL processes using Azure Databricks * Build data transformation workflows using Python or Scala. * Work with data lakes using Delta Lake. * Integrate data from multiple sources such as APIs, databases, and cloud storage. * Monitor and optimize data workflows for performance and reliability. * Collaborate with data scientists, analysts, and business teams Datavail is a leading provider of data management, application development, analytics, and cloud services, with more than 1,000 professionals helping clients build and manage applications and data via a world-class tech-enabled delivery platform and software solutions across all leading technologies. For more than 17 years, Datavail has worked with thousands of companies spanning different industries and sizes, and is an AWS Advanced Tier Consulting Partner, a Microsoft Solutions Partner for Data & AI and Digital & App Innovation (Azure), an Oracle Partner, and a MySQL Partner.

Responsibilities

The role involves building scalable ETL/ELT pipelines using Databricks, PySpark, and Spark Streaming, while developing and optimizing Delta Lake tables with features like ACID transactions and schema evolution. Responsibilities also include implementing Unity Catalog for data governance and optimizing cluster configurations and job workflows for performance.