Python Lead at Weekday AI

Pune, maharashtra, India -

Full Time

Start Date

Immediate

Expiry Date

19 Apr, 26

Salary

2000000.0

Posted On

19 Jan, 26

Experience

5 year(s) or above

Remote Job

Yes

Telecommute

Yes

Sponsor Visa

Skills

Python, Web Scraping, Data Extraction, Scrapy, Beautiful Soup, HTML, CSS, XPath, Regular Expressions, Databases, JSON, CSV, XML, Debugging, Problem Solving, Performance Optimization

Industry

technology;Information and Internet

Description

This role is for one of the Weekday's clients Salary range: Rs 1000000 - Rs 2000000 (ie INR 10-20 LPA) Min Experience: 5 years Location: Pune JobType: full-time We are looking for an experienced Python Lead with deep expertise in web scraping and data extraction to lead the design, development, and optimization of scalable scraping solutions. This role requires strong hands-on skills with Scrapy, Beautiful Soup (BeautifulSoup), and web scraping frameworks, along with the ability to guide a team, define best practices, and deliver reliable data pipelines at scale. As a Python Lead, you will take ownership of complex scraping architectures, ensure high data quality, handle anti-bot challenges, and mentor developers while working closely with product, data, and business stakeholders. Key Responsibilities Lead the end-to-end design and development of web and data scraping solutions using Python. Architect and maintain scalable scraping systems using Scrapy framework, Beautiful Soup / BeautifulSoup, and custom Python solutions. Handle complex scraping scenarios including dynamic websites, pagination, authentication flows, JavaScript-rendered content, and APIs. Implement techniques to manage rate limiting, IP rotation, proxy management, CAPTCHA handling, and bot detection mechanisms. Ensure data accuracy, consistency, and reliability, including validation, cleaning, and structured storage of scraped data. Optimize scraping performance for speed, scalability, and fault tolerance. Lead code reviews, enforce coding standards and best practices, and ensure maintainable and testable codebases. Mentor and guide junior developers, providing technical leadership and knowledge sharing. Collaborate with cross-functional teams to understand data requirements and translate them into robust scraping solutions. Ensure compliance with legal, ethical, and website terms of service related to web scraping activities. Required Skills & Qualifications 5–10 years of hands-on Python development experience, with a strong focus on web scraping and data scraping. Extensive experience with Scrapy, Scrapy Framework, and Beautiful Soup / BeautifulSoup. Strong understanding of HTML, CSS, XPath, and regular expressions. Experience scraping large-scale and high-volume websites. Knowledge of handling dynamic content using headless browsers or JavaScript rendering techniques. Solid experience with databases (SQL/NoSQL) and data formats such as JSON, CSV, and XML. Strong debugging, problem-solving, and performance optimization skills. Experience leading or mentoring teams in a technical capacity. Nice to Have Experience with cloud platforms, distributed systems, or task queues. Familiarity with data pipelines, ETL processes, or analytics workflows. Exposure to monitoring, logging, and alerting for scraping systems.

Responsibilities

Lead the design and development of web and data scraping solutions using Python. Ensure high data quality and optimize scraping performance for speed and scalability.