Position Details

Salary $90K – $110K USD / year

Type Full-Time

Experience mid

Exp. Years 8+ years

Education Bachelor’s degree in Computer Science, Computer Engineering or related field

Category Data & Analytics

About this role

Senior data engineer building data pipelines on Databricks/ Spark in a federal civilian program context, with strong Python and AWS experience.

Key Responsibilities

Build and maintain PySpark data pipelines in the Databricks environment
Optimize Spark jobs performance and resource usage
Design, develop, and maintain backend software components and services
Research and build proofs of concepts
Implement security, governance, and compliance in Databricks/AWS

Technical Overview

Data engineering stack includes Python, PySpark, Databricks, and AWS services (S3/EC2/Glue/Lambda); focus on ETL, data modeling, and distributed computing.

Ideal Candidate

The ideal candidate is a senior data engineer with 8+ years of experience, expert in PySpark and Databricks, strong SQL/data modeling, and hands-on AWS data platform experience for federal civilian programs.

Must-Have Skills

Strong experience with Python / Apache SparkSolid understanding of data modelingETL processand distributed computingBachelor’s degree in Computer ScienceComputer Engineering or related field8 years of experience in data engineering / analyticsExperience with Agile development methodologies

Nice-to-Have Skills

AWS Experience (S3EC2GlueLambda)R experienceProfessional Databricks/Apache Spark Certification(s)SAS experience

Tools & Platforms

DatabricksPythonSparkAWS

Required Skills

PythonApache SparkDatabricksPySparkDatabricksAWSS3EC2GlueLambdaETLData pipelinesSQLData modeling

Hard Skills

PythonApache SparkDatabricksPySparkDatabricksAWSS3EC2GlueLambdaETLData pipelinesSQLData modeling

Soft Skills

Strong problem-solvingAnalytical thinkingCommunicationTeam collaboration

Certifications

Preferred

Professional Databricks/Apache Spark Certification

Industry & Role

Industry Government/Public Sector

Job Function Develop and maintain data pipelines and platforms on Databricks/Spark in AWS

Role Subtype Data Engineer

Tech Domains Python, SQL / PostgreSQL, Amazon Web Services

Keywords for Your Resume

data engineer iipysparkdatabrickspySparkapache sparkawss3ec2gluelambdaetldata pipelinesagileresearchsecuritygovernancepythondata modelingsqldatabricks certification

Deal Breakers

Less than 8 years of experience, Lack of PySpark/Databricks experience, No AWS data platform exposure

Apply for this Position →

Get matched to jobs like this

Luna finds roles that fit your skills and career goals — no endless scrolling required.

Create a Free Profile

Data Engineer II, PySpark/Databricks

Get matched to jobs like this