Senior Data Engineer - Dallas/California, United States at Photon

, , United States -

Full Time

Start Date

Immediate

Expiry Date

01 Sep, 26

Salary

203000.0

Posted On

03 Jun, 26

Experience

10 year(s) or above

Remote Job

Yes

Telecommute

Yes

Sponsor Visa

Skills

ETL Pipelines, SQL, PySpark, Scala, Airflow, Google Cloud Platform, BigQuery, Data Lakehouse, GitOps, GDPR, Data Modeling, CI/CD, Data Orchestration, Python, Data Engineering, Analytics Systems

Industry

IT Services and IT Consulting

Description

Greetings Everyone Who are we? For the past 20 years, we have powered many Digital Experiences for the Fortune 500. Since 1999, we have grown from a few people to more than 4000 team members across the globe that are engaged in various Digital Modernization. For a brief 1 minute video about us, you can check https://youtu.be/uJWBWQZEA6o [https://youtu.be/uJWBWQZEA6o]. Senior Data Engineer: As part of the Mail Analytics Data Engineering team, you will be working on large-scale batch pipelines, data serving, data lakehouse, and analytics systems, enabling mission critical decision making, downstream, AI-powered capabilities, and more. If you're passionate about building data infrastructure and platforms that power modern Data- and AI-driven business at scale, we want to hear from you! Your Day ● Partner with Data Science, Product, and Engineering to collect requirements to define the data ontology for Mail Data & Analytics ● Lead and mentor junior Data Engineers to support Yahoo Mail’s ever-evolving data needs ● Design, build, and maintain efficient and reliable batch data pipelines to populate core data sets ● Develop scalable frameworks and tooling to automate analytics workflows and streamline users interactions with data products ● Establish and promote standard methodologies for data operations and lifecycle management ● Develop new or improve and maintain existing large-scale data infrastructures and systems for data processing or serving, optimizing complex code through advanced algorithmic concepts and in-depth understanding of underlying data system stacks ● Create and contribute to frameworks that improve the efficacy of the management and deployment of data platforms and systems, while working with data infrastructure to triage and resolve issues ● Prototype new metrics or data systems ● Define and manage Service Level Agreements for all data sets in allocated areas of ownership ● Develop complex queries, very large volume data pipelines, and analytics applications to solve analytics and data engineering problems ● Collaborate with engineers, data scientists, and product managers to understand business problems, technical requirements to deliver data solutions ● Engineering consulting on large and complex data lakehouse data You Must Have ● BS in Computer Science/Engineering, relevant technical field, or equivalent practical experience, with specialization in Data Engineering ● 8+ years of experience building scalable ETL pipelines on industry standard ETL orchestration tools (Airflow, Composer, Oozie) with deep expertise in SQL, PySpark, or scala. ● 3+ years leading data engineering development directly with business or data science partners ● Built, scaled, and maintained Multi-Terabyte data sets and having an expansive toolbox for debugging and unblocking large scale analytics challenges (skew mitigation, sampling strategies, accumulation patterns, data sketches, etc.) ● Experience with at least one major cloud's suite of offerings (AWS, GCP, Azure). ● Developed or enhanced ETL orchestrations tools or frameworks ● Worked within standard GitOps workflow (branch and merge, PRs, CI / CD systems) ● Experience working with GDPR ● Self-driven, challenge-loving, detail oriented, teamwork spirit, excellent communication skills, ability to multitask and manage expectations Preferred ● MS/PhD in Computer Science/Engineering or relevant technical field, with specialization in Data Engineering ● 3 years experience in Google Cloud Platform technologies (BiqQuery, Dataproc, Dataflow, Composer, Looker) Compensation, Benefits and Duration Minimum Compensation: USD 58,000 Maximum Compensation: USD 203,000 Compensation is based on actual experience and qualifications of the candidate. The above is a reasonable and a good faith estimate for the role. Medical, vision, and dental benefits, 401k retirement plan, variable pay/incentives, paid time off, and paid holidays are available for full time employees. This position is available for independent contractors No applications will be considered if received more than 120 days after the date of this post

Responsibilities

Design, build, and maintain large-scale batch data pipelines and lakehouse infrastructures to support AI-powered capabilities and mission-critical decision making. Lead and mentor junior engineers while collaborating with Data Science and Product teams to define data ontologies and automate analytics workflows.