Data Engineer II - Getting Customers Ready for AI

Overview Security represents the most critical priorities for our customers in a world awash in digital threats, regulatory scrutiny, and estate complexity. Microsoft Security aspires to make the world a safer place for all. We want to reshape security and empower every user, customer, and developer with a security cloud that protects them with end to end, simplified solutions.

The Microsoft Security organization accelerates Microsoft’s mission and bold ambitions to ensure that our company and industry is securing digital technology platforms, devices, and clouds in our customers’ heterogeneous environments, as well as ensuring the security of our own internal estate.

Our culture is centered on embracing a growth mindset, a theme of inspiring excellence, and encouraging teams and leaders to bring their best each day. In doing so, we create life-changing innovations that impact billions of lives around the world.

Microsoft Security's Getting Customers Ready for AI team is seeking a Data Engineer II to help build the data foundations that power AI-native security systems. In this role, you will design and develop scalable data platforms, pipelines, and telemetry systems that transform raw signals across identity, devices, data, applications, and infrastructure into high-quality, trusted datasets for analytics and AI-driven insights.

You will work in a collaborative engineering environment to deliver reliable, secure, and performant data systems, enabling downstream AI models, security detections, and customer-facing experiences. This role is ideal for engineers looking to deepen their expertise in large-scale data systems, distributed processing, and modern data platform architectures for AI and Agents.

Microsoft’s mission is to empower every person and every organization on the planet to achieve more. As employees we come together with a growth mindset, innovate to empower others, and collaborate to realize our shared goals. Each day we build on our values of respect, integrity, and accountability to create a culture of inclusion where everyone can thrive at work and beyond.

Responsibilities Data Platform & Pipeline Engineering Design and build scalable data pipelines (batch and streaming) to process large volumes of security and operational data. Develop and optimize ETL/ELT workflows that transform raw telemetry into structured, consumable datasets.

Implement data ingestion frameworks to integrate multi-source data from services, APIs, and event streams. Improve pipeline performance, reliability, and efficiency through partitioning, indexing, and optimization techniques. Data Modeling & Storage Systems Design and evolve data models, schemas, and storage strategies for analytics and AI use cases.

Work with distributed storage systems (e.g., data lakes, warehouses) to ensure scalability and cost efficiency. Maintain data partitioning, retention, and lifecycle strategies aligned to business and compliance needs. Ensure data is structured for downstream ML pipelines, feature engineering, and analytics workloads.

Data Quality, Governance & Security Implement data validation, quality checks, and observability frameworks to ensure data accuracy and reliability. Apply best practices for data governance, lineage, and auditing across pipelines and datasets.

Ensure compliance with security, privacy, and regulatory requirements when handling sensitive data. Contribute to standardization of data contracts and schemas across services. AI Readiness & Intelligence Enablement Enable high-quality datasets for AI/ML teams, including support for feature pipelines and training data preparation.

Collaborate with AI engineers to ensure data is optimized for RAG pipelines, model training, and evaluation workflows. Build and maintain telemetry pipelines and metrics systems that provide insights into AI system performance and usage. Support development of data-driven signals and insights that improve customer security posture.

Collaboration & Execution Partner with engineering, data science, and product teams to deliver end-to-end data solutions. Contribute to system design discussions, architecture reviews, and cross-team integration efforts. Work across teams to ensure consistent data definitions and interoperability across platforms.

Engineering Excellence Write clean, maintainable, and well-tested code for data pipelines and platform services. Build monitoring and alerting mechanisms for pipeline health, latency, and data quality issues. Document data architecture, pipeline designs, and operational practices for knowledge sharing and scalability.

Follow best practices for performance optimization, cost management, and reliability engineering. Embody our culture and values. Qualifications Required Qualifications Bachelor's Degree in Computer Science or related technical field AND 2+ years of software, data, or related engineering experience with coding in languages including, but not limited to, C, C++, C#, Java, JavaScript, or Python OR equivalent experience Other Requirements: Ability to meet Microsoft, customer, and/or government security screening requirements are required for this role.

These requirements include but are not limited to the following specialized security screenings: Microsoft Cloud Background Check : This position will be required to pass the Microsoft Cloud background check upon hire/transfer and every two years