✦ Luna Orbit — Software Engineering

Engineering Manager - Notebook Dataplane

at Databricks

📍 San Francisco, California Unknown Posted March 10, 2026
Apply Now → ← Browse All Jobs

Position Details

Type Not Specified
Experience senior
Exp. Years 5+ years
Education Not specified
Category Software Engineering

About this role

Lead the Spark Structured Streaming team at Databricks, focusing on developing advanced stream processing features, optimizing engine performance, and managing technical architecture for high-throughput, low-latency data streaming.

Key Responsibilities

  • Develop streaming engine features
  • Optimize performance
  • Lead team in OSS and product development
  • Manage technical debt
  • Drive adoption of streaming solutions

Technical Overview

Specializes in distributed data systems, stream processing, and performance optimization within cloud environments, primarily using Apache Spark and Kubernetes.

Ideal Candidate

The ideal candidate is a senior engineering leader with extensive experience in distributed data systems, streaming, and database internals. They have a strong background in building scalable SaaS platforms and leading technical teams in complex environments.

Must-Have Skills

5+ years experience in streamingquery processingor database internalsbuilding and leading teamsdistributed data systemsperformance optimizationresponsibility for SLAs

Nice-to-Have Skills

experience with open source projectscustomer engagementproduct strategylong-term architecture planning

Tools & Platforms

Apache SparkKubernetesAWSAzureGCP

Required Skills

distributed systemscloud platformsAWSAzureGCPKubernetesweb architecturesscalable SaaSquery processingApache Spark

Hard Skills

distributed systemscloud platformsAWSAzureGCPKubernetesweb application architecturesscalable SaaS platformssystem-level skillscontainersperformance optimizationquery processingquery optimizationApache Spark

Soft Skills

leadershipcollaborationproblem-solvingcommunicationteam buildingimpact-driven

Industry & Role

Industry SaaS
Job Function Lead development of Spark Structured Streaming and related data processing systems

Keywords for Your Resume

Engineering ManagerSpark Structured Streamingdistributed systemsquery processingquery optimizationApache Sparkcloud platformsKubernetesperformance optimizationSLAsteam leadershipdatabase internalsstreaming systemsbig dataOSS

Deal Breakers

Less than 5 years experience in streaming or database internals, No experience leading teams, Lack of expertise in distributed systems, No familiarity with Apache Spark or similar frameworks

Apply for this Position →

Get matched to jobs like this

Luna finds roles that fit your skills and career goals — no endless scrolling required.

Create a Free Profile