Lead Software and Data Engineer at William Blair

Chicago, Illinois, United States -

Full Time

Start Date

Immediate

Expiry Date

16 Jul, 26

Salary

200000.0

Posted On

17 Apr, 26

Experience

5 year(s) or above

Remote Job

Yes

Telecommute

Yes

Sponsor Visa

Skills

Python, Data engineering, Dagster, Azure Synapse, PySpark, Kubernetes, Data modeling, ETL/ELT design, Microservices architecture, SQL, Data lake architecture, CI/CD, Cloud infrastructure, API development, Data governance, Machine learning

Industry

Financial Services

Description

Solutions for Today’s Challenges. Vision for Tomorrow’s Opportunities. Join William Blair, the Premier Global Partnership. The Investment Banking division has built a differentiated AI foundation: proprietary ML models integrated into CRM workflows, an Azure-based analytics stack, and generative AI solutions deployed across 650+ bankers globally. We are now scaling our Investment Banking AI & Technology team to accelerate the integration of next-generation AI capabilities—anchored by frontier LLMs as the central reasoning engine, augmented by best-in-class point solutions for research and deal execution—into every stage of the banking workflow. We are looking for a Lead Software & Data Engineer to join the Investment Banking AI & Technology team. You will own the reliability and evolution of our microservices architecture, data pipelines, and data models—ensuring the systems that underpin investment banking workflows are robust, scalable, and fit for purpose. Data engineering and data modeling are central to this role. You will design and maintain the pipelines, schemas, and orchestration patterns that move and shape data across the platform—working closely with stakeholders to ensure data is accurate, accessible, and structured to support downstream analytics and reporting. You will bring rigorous engineering standards to everything you ship: clean code, tested pipelines, and systems built for longevity, not just the immediate need. Responsibilities may include but are not limited to: Design, build, and maintain production-grade data pipelines in Python—writing clean, modular, well-tested pipeline code that can be owned and extended by the broader team. Lead the migration of existing data processes from Azure Synapse to Dagster, including Synapse Pipelines, Notebooks, Linked Services, and Integration Runtimes, ensuring continuity of data flows throughout the transition. Own and evolve Dagster-based orchestration: asset-based pipelines, sensors, schedules, and Dagster+ Cloud deployments as the primary orchestration standard going forward. Architect and maintain a scalable data lake on ADLS Gen2, including data modeling, ETL/ELT design, and schema governance appropriate for confidential deal information. Support PySpark workloads where required—DataFrames, Spark SQL, and performance tuning—with the majority of pipeline development delivered in Python. Develop Kubernetes-based microservices for data and API workloads, ensuring reliable, scalable deployment of pipeline and application components. Set engineering standards for the Innovation Team: code review practices, CI/CD pipelines, testing frameworks, and documentation norms that enable speed without sacrificing reliability. Leverage AI-assisted development tools—including agentic coding environments such as Claude Code—to accelerate prototyping, code generation, and pipeline development; champion their effective adoption across the team. Work directly with deal teams and industry/sector groups to understand workflows, identify automation opportunities, and iterate on deployed tools based on real-world banker feedback. Perform rapid analysis and prototyping—translate a banker's pain point into a working proof of concept within days, not weeks. Implement security and data governance protocols appropriate for confidential deal information. Qualifications: Bachelor’s degree in Computer Science, Software Engineering, or a related field. 5+ years of software and data engineering experience with a strong foundation in production systems that serve demanding end users. Strong Python proficiency for data engineering: writing clean, production-grade pipeline code, scripting, and package management—Python is the primary development language for this role. Hands-on experience with Dagster for pipeline orchestration: asset-based pipelines, sensors, schedules, and ideally Dagster+ Cloud deployments. Hands-on Azure Synapse experience: Synapse Pipelines, Notebooks, Linked Services, and Integration Runtimes—migration experience strongly preferred. Solid general data engineering fundamentals: data modeling, ETL/ELT design, pipeline orchestration, data lake architecture (ADLS Gen2), and SQL. Working knowledge of PySpark—DataFrames, Spark SQL—sufficient to support and maintain existing distributed workloads. Experience deploying and managing containerized services on Kubernetes, including API microservice development patterns. Comfort working with AI-assisted development tools (e.g. Claude Code, GitHub Copilot) and a track record of using them to ship higher-quality work faster. Rigorous engineering practices: you write tested, reviewed, well-documented code and build systems designed for maintainability, not just demos. Familiarity with capital markets, and preferably direct experience in or adjacent to investment banking, private equity, venture capital, or hedge funds. Experience with Azure cloud infrastructure and Databricks/Spark platform. Outcome orientation—you measure success by business impact delivered, not features shipped. Preferred Qualifications: Familiarity with Dagster asset-based orchestration, sensors, schedules, and ideally Dagster+ Cloud deployments. Prior work with Salesforce APIs, SOQL, or CRM integration patterns. Contributions to engineering culture: mentoring, establishing best practices, or leading technical design reviews in a small-team environment. #LI-CH #LI-HYBRID A reasonable estimate of the current base salary range at time of posting is below. Base salary does not include other forms of compensation or benefits. Actual base salary within the specified range is based on several factors, including but not limited to applicant's skills, prior relevant experience, specific degrees and certifications, job responsibilities, market considerations and, if applicable, the location of the position. This role is eligible for either a discretionary annual bonus (based on company, business unit and individual performance) and/or commission-based incentives. Our featured benefit offerings include medical, dental and vision coverage, employer paid short & long-term disability and life insurance, 401(k), profit sharing, paid time off, Maven family & fertility benefit, parental leave (including adoption, surrogacy, and foster placement), as well as other voluntary benefits. Salary Range $155,000—$200,000 USD About Our Firm William Blair strives to attract qualified candidates who specialize in investment banking, investment management, private wealth management, and other strategic resource groups. We are committed to empowering our colleagues to deliver client success and engage in our communities. Our firm has delivered trusted advice for nine decades, continuing to deepen our expertise and relationships across asset classes and markets throughout North America, Europe, and Asia. We provide advisory services, strategies, and solutions to meet clients’ evolving needs amid dynamic market conditions and varying industries. What sets us apart is that we are an independent partnership, with colleagues who have unique experiences, perspectives, and backgrounds. We empower our people to bring their best thinking so we can deliver the tailored, thoughtful work and problem-solving abilities that our clients expect. We invite you to learn more about us by visiting williamblair.com. William Blair is an equal opportunity employer. It complies with all laws and regulations that prohibit discrimination in employment practice because of race, color, religion, creed, ancestry, marital status, gender, age, national origin, sexual orientation, unfavorable discharge from the military service or on the basis of a physical or mental disability that is unrelated to the employee's ability to perform the duties of the job applied for. EOE m/f/d/v Be aware of hiring scams: William Blair has clear processes and guidelines with regards to recruiting. We do not request personal financial information in connection with an employment application nor does William Blair extend any employment offers without first conducting an interview through one of its registered offices. William Blair does not use instant messaging services such as WhatsApp, Telegram, or iMessage as part of the recruiting or interviewing process. Note to External Recruiters / Search Firms: William Blair does not accept unsolicited resumes and will not pay for any placement resulting from the receipt of an unsolicited resume. Any unsolicited resumes received will not be considered as a valid submission. Download William Blair's privacy policies for job applicants: California Consumer Privacy Act Privacy Notice (CCPA) General Data Protection Regulation Privacy Notice (GDPR) Contact us should you have any questions or concerns.

Responsibilities

You will design, build, and maintain robust data pipelines and microservices to support investment banking workflows. Additionally, you will lead the migration of data processes to Dagster and establish engineering standards for the team.