✦ Luna Orbit — Data & Analytics

Senior Cloud Data Engineer - Emerging Lead

at State Street

Hybrid Posted March 22, 2026
Apply Now → ← Browse All Jobs

Position Details

Type Not Specified
Experience lead
Exp. Years 5-8 years
Education Not specified
Category Data & Analytics

About this role

This role involves designing, developing, and optimizing scalable data pipelines on cloud platforms using Databricks and Spark, supporting enterprise data needs in the financial sector.

Key Responsibilities

  • Design data pipelines
  • Implement medallion architecture
  • Ensure data quality
  • Optimize performance
  • Collaborate with teams

Technical Overview

The technical environment includes Databricks, Spark, PySpark, SQL, cloud platforms (AWS, Azure), CI/CD tools, and enterprise job scheduling systems, with a focus on performance and data quality.

Ideal Candidate

The ideal candidate is a lead data engineer with at least 5 years of experience in designing scalable data pipelines using Databricks and Spark, with strong skills in Python, SQL, and cloud platforms like AWS or Azure, preferably within financial services.

Must-Have Skills

DatabricksSparkPySparkSQLData pipelinesMedallion architectureData qualityData integrityPerformance tuningPython

Nice-to-Have Skills

AWSAzureData warehousingRegulatory knowledgeFinancial domain

Tools & Platforms

DatabricksAWSAzureGitJenkinsAutosysAirflow

Required Skills

DatabricksSparkPySparkSQLData pipelinesMedallion architectureData qualityData integrityPerformance tuningPythonCloud platformsAWSAzureCI/CDGitJenkinsAutosysAirflow

Hard Skills

DatabricksSparkPySparkSQLData pipelinesMedallion architectureData qualityData integrityPerformance tuningPythonSQLCloud platformsAWSAzureCI/CDGitJenkinsEnterprise job schedulingAutosysAirflow

Soft Skills

OwnershipDelivery focusCollaborationTeamworkLearning mindset

Industry & Role

Industry Financial Services
Job Function Cloud-based data pipeline engineering and optimization
Role Subtype Data & Analytics
Tech Domains Amazon Web Services, Microsoft Azure, Databricks, ETL, Big data processing

Keywords for Your Resume

data pipelinesdatabrickssparkpysparksqlmedallion architecturedata qualitydata integrityperformance tuningpythoncloud platformsawsazureci/cdgitjenkinsautosysairflowenterprise job scheduling

Deal Breakers

Lack of experience with Databricks or Spark, No cloud platform experience, Insufficient years of experience, No background in financial services

Apply for this Position →

Get matched to jobs like this

Luna finds roles that fit your skills and career goals — no endless scrolling required.

Create a Free Profile