About this role
Lead data engineering work at Capital One by designing, developing, testing, and supporting cloud-based data solutions. The role emphasizes Python/Spark-based engineering, big data tooling, and delivering robust analytics platforms in AWS and other public clouds.
Key Responsibilities
- Collaborate across Agile teams to build and support data engineering solutions
- Develop using Python, Java, Scala and database technologies
- Use cloud data warehousing like Redshift and Snowflake
- Apply distributed data/computing frameworks such as Spark and Kafka
- Perform unit tests and optimize for performance
Technical Overview
You will work across Agile teams to build data engineering systems using Python and Spark, leveraging big data frameworks (MapReduce, Hadoop, Hive, EMR, Kafka) and cloud warehousing (Redshift, Snowflake). Responsibilities include unit testing, performance tuning, and working on real-time/streaming data when applicable.
Ideal Candidate
The ideal candidate is a lead data engineer with 4+ years of application development experience and 2+ years in big data technologies, plus hands-on cloud computing experience on Amazon Web Services. They have delivered real-time or streaming data solutions and are comfortable working across Agile teams using Python, Spark, and cloud data warehousing like Redshift or Snowflake.
Must-Have Skills
Bachelor's DegreeAt least 4 years of experience in application development (Internship experience does not apply)At least 2 years of experience in big data technologiesAt least 1 year experience with cloud computing (AWSMicrosoft AzureGoogle Cloud)collaborate with and across Agile teams to designdeveloptestimplementand support technical solutions
Nice-to-Have Skills
7+ years of experience in application development including PythonSQLScalaor Java4+ years of experience with a public cloud (AWSMicrosoft AzureGoogle Cloud)4+ years experience with Distributed data/computing tools (MapReduceHadoopHiveEMRKafkaSparkGurobior MySQL)4+ year experience working on real-time data and streaming applications4+ years of experience with NoSQL implementation (MongoCassandra)4+ years of data warehousing experience (Redshift or Snowflake)4+ years of experience with UNIX/Linux including basic commands and shell scripting2+ years of experience with Agile engineering practicesmentoring other members of the engineering community
Tools & Platforms
PythonJavaScalaRedshiftSnowflakeAmazon Web ServicesMicrosoft AzureGoogle CloudMapReduceHadoopHiveEMRKafkaSparkGurobiMySQLMongoCassandraUNIX/Linuxshell scripting
Required Skills
PythonSparkGlueAWSAmazon Web ServicesJavaScalaSQLRedshiftSnowflakebig data technologiesNoSQL databasesKafkaHadoopHiveEMRMapReducereal-time datastreaming applicationsUNIX/Linuxshell scriptingAgile
Hard Skills
application developmentprogramming languages: Javaprogramming languages: Scalaprogramming languages: PythonOpen Source RDBMSNoSQL databasesdata warehousing services: Redshiftdata warehousing services: Snowflakebig data technologiescloud computingAmazon Web ServicesMicrosoft AzureGoogle CloudDistributed data/computing toolsMapReduceHadoopHiveEMRKafkaSparkGurobiMySQLreal-time datastreaming applicationsNoSQL implementationMongoCassandraUNIX/Linuxshell scriptingunit testsperformance tuningAgile engineering practicesAgile teamsfull-stack development tools
Soft Skills
collaborationmentoringcommunicationiterationexperimentationlearning new technologiesinclusive mindsetstaying on top of tech trends
Keywords for Your Resume
Lead Data EngineerData EngineerPythonSparkGlueAWSAmazon Web ServicesSQLScalaJavaOpen Source RDBMSNoSQL databasesNoSQL implementationRedshiftSnowflakebig data technologiescloud computingMicrosoft AzureGoogle CloudAgileAgile teamsunit testsperformance tuningdistributed dataMapReduceHadoopHiveEMRKafkaGurobiMySQLreal-time datastreaming applications
Deal Breakers
Must have at least 4 years application development experience, Must have at least 2 years big data technologies experience, Must have at least 1 year cloud computing experience (AWS, Microsoft Azure, Google Cloud)
Get matched to jobs like this
Luna finds roles that fit your skills and career goals — no endless scrolling required.
Create a Free Profile