Position Details

Salary $99K – $162K USD / year

Type Full-Time

Experience senior

Exp. Years 5+ years

Education Not specified

Category Data & Analytics

About this role

This role involves developing and maintaining scalable data pipelines using Databricks and PySpark, with a focus on real-time processing and data quality.

Key Responsibilities

Build scalable data pipelines
Monitor and troubleshoot data workflows
Collaborate with data teams
Optimize data transformation processes
Support real-time data solutions

Technical Overview

The technical environment includes Databricks, PySpark, Python, SQL, and OpenSearch, emphasizing big data processing, ETL/ELT pipelines, and troubleshooting.

Ideal Candidate

The ideal candidate is a senior data engineer with extensive experience in Databricks, PySpark, and Python, capable of designing scalable data pipelines and troubleshooting large data environments.

Must-Have Skills

Experience with DatabricksProficiency in Python and PySparkDesigning scalable data pipelinesExperience with ETL/ELT processesMonitoring and troubleshooting data pipelines

Nice-to-Have Skills

OpenSearchReal-time data pipelinesAWS services

Tools & Platforms

DatabricksPySparkOpenSearch

Required Skills

DatabricksPySparkPythonSQLETLELTdata pipelinesbig datadata transformationopenSearch

Hard Skills

DatabricksPySparkPythonSQLData PipelinesBig DataData TransformationData ModelingETL/ELTOpenSearch

Soft Skills

collaborationproblem-solvingcommunicationmentorshipteamwork

Industry & Role

Industry Insurance

Job Function Design and maintain enterprise data pipelines

Role Subtype Data Engineer

Tech Domains Databricks, Python, PySpark, OpenSearch

Clearance & Visa

Clearance Required None

Visa Sponsorship No

Keywords for Your Resume

Senior Data EngineerDatabricksPySparkPythonSQLETLELTData pipelinesBig dataData transformationData modelingOpenSearchreal-time data pipelinescloud datadata engineeringdata solutions

Deal Breakers

Lack of experience with Databricks or PySpark, No experience with ETL/ELT pipelines, Unwillingness to work in a hybrid or flexible environment, No experience with data pipeline monitoring

Apply for this Position →

Get matched to jobs like this

Luna finds roles that fit your skills and career goals — no endless scrolling required.

Create a Free Profile