✦ Luna Orbit — Software Engineering

Engineering Manager - Pipelines Engine

at Databricks

📍 Mountain View, California; San Francisco, California Unknown Posted March 18, 2026
Apply Now → ← Browse All Jobs

Position Details

Type Full-Time
Experience lead
Exp. Years 3+ years
Education Not specified
Category Software Engineering

About this role

Databricks is seeking an Engineering Manager to lead the Pipelines Engine team responsible for building next-generation ETL features and ensuring high performance of data pipelines. The role involves managing a talented engineering team, setting product vision, and overseeing technical architecture.

Key Responsibilities

  • Lead ETL feature development
  • Manage engineering team recruitment and upskilling
  • Oversee technical debt and architecture
  • Align product strategy with organizational goals
  • Ensure high-quality, reliable infrastructure

Technical Overview

The role involves building and managing data infrastructure systems, with a focus on Apache Spark, distributed systems, performance optimization, and scalable ETL pipelines within the Databricks Lakeflow platform.

Ideal Candidate

The ideal candidate is a senior engineering leader with at least 3 years of experience managing top-tier data engineering teams, with deep expertise in building scalable data infrastructure using Apache Spark or similar systems. They possess strong leadership, strategic planning, and technical architecture skills, and are passionate about performance optimization and customer-centric product development.

Must-Have Skills

Managing top-tier engineering teamsBuilding data infrastructure systemsExperience with Apache Spark or database internalsCustomer interactionTechnical architecture decisions

Nice-to-Have Skills

Performance optimizationDistributed systemsProduct strategyTechnical debt management

Tools & Platforms

Apache SparkDatabricks ETLGenie Code

Required Skills

Managing top-tier engineering teamsBuilding data infrastructure systemsApache SparkDistributed SystemsPerformance OptimizationCustomer InteractionTechnical ArchitectureTalent Recruitment

Hard Skills

Apache SparkSparkDatabase InternalsDistributed SystemsPerformance OptimizationProduct ManagementCustomer InteractionTechnical Debt Management

Soft Skills

LeadershipTeam BuildingStrategic PlanningCommunicationProblem-solvingTalent RecruitmentUpskilling

Industry & Role

Industry Technology / Data Engineering
Job Function Lead data engineering infrastructure development and team management
Role Subtype Systems Architect
Tech Domains Apache Spark, Databricks ETL

Keywords for Your Resume

engineering managerdata infrastructureApache Sparkdistributed systemsETLtechnical architecturetechnical debtproduct managementcustomer needstalent recruitmentperformance optimizationDatabricksLakeflowpipelines enginesoftware developmentcustomer interaction

Deal Breakers

Less than 3 years managing engineering teams, No experience with Apache Spark or database internals, Lack of leadership or strategic planning skills, Inability to manage technical debt

Apply for this Position →

Get matched to jobs like this

Luna finds roles that fit your skills and career goals — no endless scrolling required.

Create a Free Profile