About this role
Lead data engineering for card technology, collaborating with Agile teams to design, build, test, and support cloud-based data solutions. The role emphasizes data platform development with big data technologies, warehousing, and real-time/streaming capabilities.
Key Responsibilities
- Collaborate with Agile teams to design, develop, test, implement, and support solutions
- Utilize Java, Scala, Python and RDBMS/NoSQL databases with cloud data warehousing (Redshift, Snowflake)
- Work with machine learning, distributed microservices, and full stack systems
- Perform unit tests, code reviews, and performance tuning
- Collaborate with digital product managers and mentor the engineering community
Technical Overview
Core stack includes Java, Scala, Python, Open Source RDBMS, NoSQL databases, and cloud data warehousing with Amazon Redshift and Snowflake. It also expects experience with distributed data/computing tools (Hadoop ecosystem, Kafka, Spark) and real-time streaming workloads, plus UNIX/Linux and Agile development practices.
Ideal Candidate
The ideal candidate is a Lead Data Engineer with 4+ years of application development experience and at least 2 years of big data technologies experience. They have hands-on cloud experience (AWS, Microsoft Azure, or Google Cloud) and strong data engineering skills spanning data warehousing (Redshift or Snowflake) and streaming/real-time systems.
Must-Have Skills
Bachelor's DegreeAt least 4 years of experience in application development (Internship experience does not apply)At least 2 years of experience in big data technologiesAt least 1 year experience with cloud computing (AWSMicrosoft AzureGoogle Cloud)
Nice-to-Have Skills
7+ years of experience in application development including PythonSQLScalaor Java4+ years of experience with a public cloud (AWSMicrosoft AzureGoogle Cloud)4+ years experience with Distributed data/computing tools (MapReduceHadoopHiveEMRKafkaSparkGurobior MySQL)4+ year experience working on real-time data and streaming applications4+ years of experience with NoSQL implementation (MongoCassandra)4+ years of data warehousing experience (Redshift or Snowflake)4+ years of experience with UNIX/Linux including basic commands and shell scripting2+ years of experience with Agile engineering practices
Tools & Platforms
Amazon Web Services (AWS)Microsoft AzureGoogle CloudAmazon RedshiftSnowflakeMapReduceHadoopHiveEMRKafkaSparkGurobiMySQLMongoCassandra
Required Skills
application developmentbig data technologiescloud computing (AWSMicrosoft AzureGoogle Cloud)JavaScalaPythonOpen Source RDBMSNoSQL databasesAmazon RedshiftSnowflakeunit testscode reviewsperformance tuningdistributed microservicesmachine learningMapReduceHadoopHiveEMRKafkaSparkGurobiMySQLreal-time datastreaming applicationsMongoCassandraUNIX/Linuxshell scriptingAgile engineering practices
Hard Skills
application developmentbig data technologiescloud computing (AWSMicrosoft AzureGoogle Cloud)full-stack development tools and technologiesJavaScalaPythonOpen Source RDBMSNoSQL databasesAmazon RedshiftSnowflakeunit testsperformance tuningdistributed microservicesmachine learningreal-time datastreaming applicationsMapReduceHadoopHiveEMRKafkaSparkGurobiMySQLMongoCassandraNoSQL implementationdata warehousing (Redshift or Snowflake)UNIX/Linux including basic commands and shell scriptingAgile engineering practices
Soft Skills
collaborationiterative deliverymentoringcode review participationstaying on top of tech trendsexperimenting and learning new technologiesteamworkcommunicationinclusive environmentdigital product manager collaboration
Keywords for Your Resume
Lead Data EngineerLead Data Engineer - Card TechData Engineersfull-stack development toolsAgile teamsAgile engineering practicesapplication developmentbig data technologiescloud computingAWSAmazon Web ServicesMicrosoft AzureGoogle CloudJavaScalaPythonOpen Source RDBMSNoSQL databasesRedshiftAmazon RedshiftSnowflakeunit testscode reviewsperformance tuningdistributed microservicesmachine learningMapReduceHadoopHiveEMRKafkaSparkUNIX/Linuxshell scriptingreal-time datastreaming applicationsMongoCassandradata warehousingdata warehousing experience (Redshift or Snowflake)
Deal Breakers
Bachelor's Degree, At least 4 years of experience in application development (Internship experience does not apply), At least 2 years of experience in big data technologies, At least 1 year experience with cloud computing (AWS, Microsoft Azure, Google Cloud), No immigration sponsorship (Capital One will not sponsor)
Get matched to jobs like this
Luna finds roles that fit your skills and career goals — no endless scrolling required.
Create a Free Profile