About this role
This role involves leading the development and deployment of large-scale data applications and pipelines for government clients, ensuring data integrity, scalability, and security.
Key Responsibilities
- Lead data application development
- Oversee data pipeline deployment
- Mentor data engineering teams
- Ensure data security and compliance
- Collaborate with stakeholders on data solutions
Technical Overview
The technical environment includes Databricks, PySpark, AWS cloud services, Hadoop, Hive, EMR, Kafka, and NoSQL databases like MongoDB and Cassandra.
Ideal Candidate
The ideal candidate is a senior data engineer with over 7 years of experience in application development, specializing in Databricks, PySpark, and AWS cloud services. They should have leadership experience and a strong background in building scalable data applications for government or enterprise projects.
Must-Have Skills
7+ years of experience in application development using DatabricksPySparkand AWS7+ years of experience designingdevelopingoperationalizingand maintaining complex data applications5+ years of experience creating software for data parsing and processing5+ years of experience developing scalable ETL/ELT workflowsExperience supervising others and leading projectsAbility to obtain and maintain a Public Trust or Suitability/Fitness determinationBachelor's degree
Nice-to-Have Skills
Experience with cloud platforms such as AzureGCPor AWSExperience with distributed data tools like HadoopHiveKafkaExperience with real-time data and streaming applicationsExperience with NoSQL databases like MongoDB or CassandraExperience with data warehousing solutions like RedshiftSnowflakeUNIX/Linux scriptingAgile engineering practices
Tools & Platforms
DatabricksPySparkAWSHadoopHiveEMRKafkaMongoDBCassandraRedshiftSnowflake
Required Skills
DatabricksPySparkAWSHadoopHiveEMRKafkaData engineeringData pipelinesData analysisData examinationData management
Hard Skills
DatabricksPySparkAWSHadoopHiveEMRKafkaData engineeringData pipelinesStructured dataUnstructured dataData applicationsScalable platformsData analysisData examinationData integrationData management
Soft Skills
leadershipmentoringcollaborationproject managementcommunicationproblem-solving
Keywords for Your Resume
DatabricksPySparkAWSHadoopHiveEMRKafkaData engineeringData pipelinesScalable data applicationsData parsingData processingData analysisData examinationETL workflowsData applicationsData managementData integrationData platforms
Deal Breakers
Lack of extensive experience with Databricks and PySpark, No leadership or supervisory experience, Inability to obtain security clearance, No Bachelor's degree
Get matched to jobs like this
Luna finds roles that fit your skills and career goals — no endless scrolling required.
Create a Free Profile