Senior Lead Data & AI Engineer
Vanguard•1d ago
CanadaHybridFull-timeSenior Level9+ yrs exp
Top focus
Ai EngineerSenior Data EngineerData EngineerVp DataSenior Data Analyst
Key Responsibilities
- Architect and build reusable, metadata-driven data and AI engineering frameworks that standardize ingestion, transformation, feature engineering, and AI workflow deployment.
- Leverage Databricks, lakehouse architecture, declarative pipelines, and cloud-native services to enable scalable, governed, and reusable data products across the organization.
- Architect and deploy Delta Live Tables and Lakeflow jobs on Databricks to automate data processing, AI pipelines, and agent data refresh cycles.
- Leverage Databricks Workflows and Job Orchestration to schedule and monitor AI agent deployments across multiple business workflows.
- Integrate Lakeflow for real-time data stream processing, ensuring AI agents are updated and responsive to live data.
- Ensure seamless orchestration between AI models and data pipelines, using event-driven architectures for real-time inference and deployment.
- Implement and orchestrate AI agents using frameworks such as Agentic systems, AgentOps tooling, and solutions like Agents on Databricks (Agent-bricks).
- Hands-on experience deploying AI agents using RAG, Graph RAG, MCP-enabled integrations, and agent orchestration frameworks such as AgentOps, AgentBricks, LangGraph, or cloud-native orchestration services.
- Manage AI agent lifecycles, monitoring, and scaling using tools like SageMaker, Bedrock, or AI orchestration frameworks on AWS.
- Ensure robust data governance, metadata management, and AI observability through Unity Catalog, AWS Glue, or custom metadata layers.
- Design for scalability and modularity, ensuring AI agents are reusable across multiple business processes
Qualifications
- 9–12+ years in data engineering, specializing in AI deployment within cloud ecosystems (Databricks, AWS).
- Hands-on experience deploying AI agents using frameworks like RAG, graph RAG, and orchestrating agents (AgentOps, Agent-bricks, etc.).
- Proficient in AWS AI/ML services (SageMaker, Bedrock) and orchestration tools (MWAA, Step Functions).
- Strong knowledge of lakehouse architecture, Unity Catalog, and data modeling best practices.
- Deep experience in data orchestration, monitoring, and scalable AI-driven workflows.
- Special Factors Sponsorship Vanguard is not offering visa sponsorship for this position.
- About Vanguard At Vanguard, we don't just have a mission—we're on a mission.
- To work for the long-term financial wellbeing of our clients.
- To lead through product and services that transform our clients' lives.
- To learn and develop our skills as individuals and as a team.
- From Malvern to Melbourne, our mission drives us forward and inspires us to be our best.
- How We Work Vanguard has implemented a hybrid working model for the majority of our crew members, designed to capture the benefits of enhanced flexibility while enabling in-person learning, collaboration, and connection.
- We believe our mission-driven and highly collaborative culture is a critical enabler to support long-term client outcomes and enrich the employee experience.
Required skills
DatabricksAWSSageMakerBedrockAgentOpsAgent-bricksUnity CatalogAWS Gluedata engineeringAI deploymentlakehouse architecturedata orchestrationfeature engineeringAI workflowsmetadata management