Position Details

Salary $98K – $146K USD / year

Type Not Specified

Experience senior

Exp. Years Not specified

Education Not specified

Category Cloud & Infrastructure

About this role

This role designs and develops scalable data pipelines to support business analytics and reporting. You will use Databricks with Spark Scala and Azure Data Factory to build ETL workflows, ensure data quality and governance, and contribute to architecture decisions.

Key Responsibilities

Design and develop scalable data pipelines using Databricks Spark Scala and Azure Data Factory
Develop and implement robust ETL processes for ingestion, transformation, and preparation
Monitor, troubleshoot, and resolve data pipeline issues
Ensure data quality, consistency, and governance across sources and environments
Document workflows, pipeline configurations, and technical specifications

Technical Overview

Hands-on development of ETL/data ingestion, transformation, and preparation pipelines using Databricks, Spark Scala, and Azure Data Factory (ADF). Includes monitoring/troubleshooting, data architecture improvement recommendations, documentation of workflows, and implementing data governance/security compliance and data modeling standards.

Ideal Candidate

The ideal candidate is a senior data engineering professional with strong experience building scalable data pipelines using Databricks and Spark Scala. They have hands-on experience with Azure Data Factory (ADF) for ETL, including monitoring, troubleshooting, and implementing data governance, quality, and modeling standards.

Must-Have Skills

DatabricksSpark ScalaAzure Data Factory (ADF)ETL processesData pipelinesData governance

Tools & Platforms

DatabricksApache SparkSpark ScalaAzure Data Factory (ADF)Scala

Required Skills

DatabricksSpark ScalaAzure Data Factory ADFETL processesdata pipelinesdata qualitydata governancesecurity compliancedata modeling

Hard Skills

DatabricksApache SparkSpark ScalaScalaAzure Data Factory (ADF)Data pipelinesETLETL processesData ingestionData transformationData preparationData pipeline monitoringData pipeline troubleshootingData architecture analysisData governanceData qualityData consistencySecurity complianceData modelingData integrationData transformation solutionsScalable data pipelinesBusiness analytics and reporting

Soft Skills

Cross-functional collaborationCollaboration with crossfunctional teamsGathering and translating data requirementsParticipate in technical discussionsContribute to architectural decisionsKnowledge sharingMentor and guide junior team membersContinuous learningAttention to detail

Industry & Role

Industry SaaS

Job Function Develop and govern scalable ETL data pipelines using Databricks Spark Scala and Azure Data Factory (ADF) for analytics and reporting.

Role Subtype Cloud Engineer

Tech Domains Azure, Amazon Web Services, Data & Analytics, Cloud & Infrastructure, SQL / PostgreSQL

Keywords for Your Resume

Senior Specialist - Cloud EngineeringData Bricks developerDatabricksSpark ScalaScalaAzure Data Factory ADFETLETL processesdata pipelinesdata ingestiontransformationdata preparationdata qualitydata governancesecurity compliancedata modelingdata integrationarchitectural decisionsMonitortroubleshootDocument data workflowspipeline configurationsscalable

Deal Breakers

Must have experience with Databricks, Must have experience with Spark Scala, Must have experience with Azure Data Factory (ADF) for ETL/data pipelines

Apply for this Position →

Get matched to jobs like this

Luna finds roles that fit your skills and career goals — no endless scrolling required.

Create a Free Profile