Senior Data Engineer - Copilot Notebooks at Microsoft

Redmond, Washington, United States -

Full Time

Start Date

Immediate

Expiry Date

02 Mar, 26

Salary

0.0

Posted On

02 Dec, 25

Experience

5 year(s) or above

Remote Job

Yes

Telecommute

Yes

Sponsor Visa

Skills

Data Engineering, Data Science, Software Development, Data Modeling, C#, Python, SQL, Scala, Kusto Databases, Kusto Query Language, GDPR, DORA, CCPA, Spark, Hadoop, Azure, AWS, GCP

Industry

Software Development

Description

You will coordinate with Data Scientists, Product Managers and Software Engineers to ensure that the production telemetry contains the information required to generate insights and understanding of how customers are using the product. You will build code to extract raw data, validate its quality, and apply your deep data knowledge to ensure the correct data is ingested across the pipeline. You'll also create techniques to transform raw data into forms compatible with downstream data sources. You will identify opportunities to leverage and contribute to the development of data tools that are used to transform, manage, and access data. You'll also write and validate code to test the storage and availability of data platforms so that they're more resilient. You will write code to implement performance monitoring protocols across data pipelines, building visualizations and aggregations to monitor pipeline health. You'll also implement solutions and self-healing processes that minimize points of failure across multiple product features. You will anticipate data governance needs, designing data modeling and handling procedures to ensure compliance with all applicable laws and policies. You'll also govern data accessibility within your assigned pipelines. Master's Degree in Computer Science, Math, Software Engineering, Computer Engineering, or related field AND 3+ years experience in business analytics, data science, software development, data modeling, or data engineering OR Bachelor's Degree in Computer Science, Math, Software Engineering, Computer Engineering, or related field AND 4+ years experience in business analytics, data science, software development, data modeling, or data engineering OR equivalent experience. These requirements include but are not limited to the following specialized security screenings: 2+ years experience handling data under strict compliance, laws and regulations. 2+ years experience with large data sets spanning multiple architectures. 4+ years experience with C#, Python, SQL, Scala or other scripting languages for data processing and automation. Experience with Kusto Databases and Kusto Query Language. Experience with operating under GDPR, DORA, CCPA and other regulations. Experience with Data Engineering for AI and Data Science. Expertise in distributed computing frameworks like Spark, Hadoop, or other cloud-based data solutions. Experience with on-premises enterprise data warehouse (EDW) appliance such as: Teradata, Netezza, or Exadata. Experience developing solutions in distributed cloud platforms such as Azure, AWS, or GCP, with a focus on data storage, processing, and security.

Responsibilities

You will coordinate with Data Scientists, Product Managers, and Software Engineers to ensure that production telemetry contains the necessary information for insights into customer usage. You will build code to extract and validate data quality, transform raw data, and implement performance monitoring protocols across data pipelines.