About this role
Leads the Advanced Analytics and Enterprise Insights engineering team to deliver AI-ready data products on the enterprise data platform, building and operating data pipelines with modern tools, and guiding governance, security, and DevOps practices.
Key Responsibilities
- Lead the AI Data Engineering team
- Oversee delivery outcomes
- Collaborate on end-to-end designs
- Define pipelines-as-code standards
- Implement CI/CD across environments
Technical Overview
Stack includes Databricks, Snowflake, Palantir, CI/CD, Git, automated testing, and IaC (Terraform/CloudFormation); focuses on AI-ready data governance, embeddings/vector indexing, and hybrid retrieval patterns.
Ideal Candidate
The ideal candidate is a senior data engineering manager with 8+ years delivering production-grade data pipelines on Databricks and Snowflake, leading AI-ready data products and embedding/patterns for enterprise retrieval. You have strong governance, security, and DevOps discipline, and can mentor teams across architecture, governance, security, and AI partnerships.
Must-Have Skills
Bachelor's degree in Computer ScienceInformation Systemsor related field3+ years of engineering leadership8+ years in data engineeringplatform engineeringor similar roles delivering production-grade pipelinesHands-on experience with DatabricksHands-on experience with SnowflakeGit workflowsCI/CD automationAutomated testingEnvironment promotionObservabilityExperience preparing data for AI/ML including GenAI readinessExperience with unstructured data processing patternsExperience implementing governance and security controls including permissionaware accessPII handlingand auditabilityStrong problem-solving mindsetExcellent communication and collaboration skills
Nice-to-Have Skills
Vector search or retrieval technologies (Azure AI SearchOpenSearch/ElasticDatabricks Vector Search)Databricks WorkflowsAirflowInfrastructure as Code such as Terraform or CloudFormationEvaluation signals for retrieval qualityHybrid retrieval and relevance tuning approaches
Required Skills
Bachelor's degree in Computer ScienceInformation Systemsor related field; 3+ years of engineering leadership; 8+ years in data engineering/platform engineering; Databricks; Snowflake; Palantir; Git workflows; CI/CD automation; Automated testing; Observability; GenAI readiness; unstructured data patterns; governance and security controls; permission-aware access; PII handling; auditability
Hard Skills
DatabricksSnowflakePalantirDatabricks WorkflowsAirflowTerraformCloudFormationGitCI/CDContinuous Integration/Continuous DeploymentAutomated testingObservabilityInfrastructure as CodeVersion controlEmbeddingsVector indexingContent versioningMetadata enrichmentRetrieval patternsPermissionaware accessPII handlingAuditability
Soft Skills
leadershipmentoringcommunicationstakeholder managementcollaborationproblem-solvingrisk managementdelivery accountabilitycross-functional collaboration
Keywords for Your Resume
DatabricksSnowflakePalantirGenAILLMEmbeddingsVector indexingCI/CDContinuous Integration/Continuous DeploymentGitObservabilitydata pipelinesAI-readyAIreadydata contractssemantic layersconformed dimensionscontent versioningretrievalpermissionawarePII handlingauditabilityembedding pipelines
Deal Breakers
Lack of Databricks or Snowflake experience, No experience leading data engineering teams, No exposure to governance and security controls (PII, auditability), No hands-on CI/CD or automated testing experience, Bachelor's degree not in required fields
Get matched to jobs like this
Luna finds roles that fit your skills and career goals — no endless scrolling required.
Create a Free Profile