Position Details

Salary $99K – $225K USD / year

Type Not Specified

Experience lead

Exp. Years 8+ years

Education Bachelor's degree

Category Data & Analytics

About this role

This role involves leading data engineering activities, overseeing the development of scalable data pipelines, and mentoring a multidisciplinary team in a mission-driven environment.

Key Responsibilities

Lead data engineering projects
Oversee pipeline development
Mentor team members
Optimize data workflows
Ensure data security and compliance

Technical Overview

Focuses on building and maintaining data pipelines using Spark, Databricks, and streaming technologies, with cloud integration via AWS.

Ideal Candidate

The ideal candidate is a senior data engineer with over 8 years of experience in building and leading data pipeline projects, proficient in Spark, Databricks, and streaming technologies, with strong leadership skills in an agile environment.

Must-Have Skills

8+ years designingbuildingmaintaining data pipelines5+ years leading data engineering teams3+ years developing with SparkDatabricksor PySpark3+ years working with data lakehouse or warehouse platformsExperience with streaming data (KafkaKinesis)

Nice-to-Have Skills

Experience with ElasticSearchContainerization and orchestration (DockerKubernetesEKS)Experience with PythonSQLScalaJavaExperience with data analyticsExperience with AWS

Tools & Platforms

SparkDatabricksPySparkKafkaKinesisElasticSearchDockerKubernetesEKSAWS

Required Skills

Data pipelinesDistributed systemsSparkDatabricksPySparkData lakehouseSchema designQuery optimizationStreaming dataKafkaKinesisAWSPythonSQLScalaJava

Hard Skills

Data pipelinesDistributed systemsSparkDatabricksPySparkData lakehouseSchema designQuery optimizationStreaming dataKafkaKinesisParquetAvroJSONDeltaAWSPythonSQLScalaJava

Soft Skills

LeadershipMentoringAgile environmentAnalytical explorationTeam management

Industry & Role

Industry Government/Public Sector

Job Function Lead data engineering efforts to organize and process large-scale data for mission-critical applications.

Clearance & Visa

Clearance Required DHS Suitability

Keywords for Your Resume

Data Engineering Team LeadData pipelinesDistributed systemsSparkDatabricksPySparkData lakehouseSchema designQuery optimizationStreaming dataKafkaKinesisAWSPythonSQLScalaJavaElasticSearchDockerKubernetesEKS

Deal Breakers

Less than 8 years of experience in data engineering, No leadership experience, Lack of experience with Spark or Databricks

Apply for this Position →

Get matched to jobs like this

Luna finds roles that fit your skills and career goals — no endless scrolling required.

Create a Free Profile