Position Details
About this role
This role involves designing and developing scalable data pipelines for financial data, supporting real-time analytics, and ensuring data governance using Databricks, PySpark, and Airflow.
Key Responsibilities
- Design and develop data pipelines
- Implement ETL processes
- Support real-time analytics
- Collaborate with data scientists and analysts
- Ensure data quality and compliance
Technical Overview
The environment includes Databricks for big data processing, PySpark for data engineering, Airflow for workflow orchestration, and Kafka for streaming data.
Ideal Candidate
The ideal candidate is a mid-level data engineer with at least 3 years of experience in building and managing large-scale data pipelines using PySpark, Databricks, and Airflow, preferably in a financial services environment. Strong collaboration and problem-solving skills are essential.
Must-Have Skills
Nice-to-Have Skills
Tools & Platforms
Required Skills
Hard Skills
Soft Skills
Industry & Role
Keywords for Your Resume
Deal Breakers
No experience with PySpark or Databricks, Lack of SQL or Python skills, No experience with data pipelines or ETL, Location outside New York
Get matched to jobs like this
Luna finds roles that fit your skills and career goals — no endless scrolling required.
Create a Free Profile