Position Details

Type Full-Time

Experience senior

Exp. Years 10+ years

Education Bachelor's degree with 10 years of related experience; Master's with 6+ years; PhD with 3+ years

Category AI & Machine Learning

About this role

Principal AI Software Engineer at Palo Alto Networks leads the design and deployment of GenAI-powered, enterprise-grade AI applications with emphasis on observability, testing, and agentic workflows.

Key Responsibilities

Lead architecture and implementation of a Proxy-First 'AI Gateway' and unified API to abstract LLM complexities
Design GenAI observability pipelines for trace data, prompts, and latency
Engineer cost-aware model routing across models
Advance agentic AI and Multi-Agent Systems
Build security guardrails for PII and content safety

Technical Overview

Hands-on Python with GenAI/LLMs (GPT-4, Llama 3, Anthropic); building observable GenAI systems using LangSmith/Arize/HoneyHive/OpenTelemetry; architecting Proxy-First AI Gateway and A2A workflows.

Ideal Candidate

The ideal candidate is a senior principal AI engineer with deep GenAI experience, expert in LLMs like GPT-4 and Llama 3, and strong observability tooling (LangSmith, Arize, HoneyHive, OpenTelemetry). They can architect agentic GenAI workflows and drive GTM automation within an enterprise cybersecurity context.

Must-Have Skills

Bachelor's degree with 10 years of related experienceor Master's with 6 yearsor PhD with 3 years in Computer Science or related fieldHands-on Python expertExpert-level proficiency with GenAI and LLMs (e.g.GPT-4Llama 3Anthropic)Experience building highly observable GenAI systems with tracing/evaluation toolsExperience architecting GenAI workflows for enterprise applications

Nice-to-Have Skills

Experience with Multi-Agent systems and Agent-to-Agent (A2A) orchestrationJava or Node.js for enterprise integrationFamiliarity with Golden Datasets and ground-truth testingSecurity and governance for AI systems

Tools & Platforms

LangSmithArizeHoneyHiveOpenTelemetry

Required Skills

Bachelor's degree with 10 years of related experience; hands-on Python; GenAI and LLMs; LangSmith; Arize; HoneyHive; OpenTelemetry; agentic/multi-agent workflows; AI observability; testing methodologies for GenAI

Hard Skills

PythonLarge Language ModelsGenAILangSmithArizeHoneyHiveOpenTelemetryAgent-to-AgentMulti-Agent SystemsLLM testingAI observability

Soft Skills

Technical leadershipArchitectureCollaborationProblem solvingMentorship

Industry & Role

Industry Cybersecurity

Job Function Lead GenAI engine design and governance for enterprise security-focused applications

Role Subtype AI Engineer

Tech Domains Python, Cybersecurity

Keywords for Your Resume

principal ai software engineerpalo alto networksgenailarge language modelsllmsllmpythonlangsmitharizehoneyhiveopentelemetrymulti-agent systemsagent-to-agentai gatewayproxy-firstgenai observabilitygTMllm testingentpriseai securityGenAILarge Language ModelsPythonLangSmithArizeHoneyHiveOpenTelemetryMulti-Agent SystemsAgent-to-AgentLLMs

Deal Breakers

Lack of hands-on Python experience, Inability to work onsite in California, No experience with GenAI/LLMs

Apply for this Position →

Get matched to jobs like this

Luna finds roles that fit your skills and career goals — no endless scrolling required.

Create a Free Profile