Position Details

Type Full-Time

Experience mid

Exp. Years 5+ years

Education B.Sc. or M.Sc. in Computer Science, Computer Engineering, or a related field

Category Data & Analytics

About this role

This role involves developing and maintaining scalable data pipelines for telemetry and performance analytics, leveraging Databricks, Spark, and cloud platforms to ensure data quality and reliability.

Key Responsibilities

Build data ingestion pipelines
Develop ETL/ELT workflows
Ensure data quality and schema evolution
Collaborate with R&D and data scientists
Maintain data reliability standards

Technical Overview

The position requires expertise in data engineering, cloud platforms, Spark, Databricks, schema management, and real-time data processing frameworks.

Ideal Candidate

The ideal candidate is a senior data engineer with over 5 years of experience in building scalable data pipelines using Databricks and Apache Spark. They should be proficient in Python and SQL, with strong knowledge of cloud platforms and data validation frameworks.

Must-Have Skills

5+ years in data engineeringExperience with Databricks and Apache SparkProficiency in Python and SQLExperience with schema evolution and data validation frameworksExperience working with cloud platforms (AWSGCPAzure)

Nice-to-Have Skills

Experience with hardwarefirmwareor embedded telemetryKnowledge of real-time analytics frameworksData governance toolsCI/CD for data pipelinesData cataloging

Tools & Platforms

DatabricksApache SparkPySparkScalaAWSGCPAzureAirflowPrefectDagsterKafkaDelta LakeGreat ExpectationsIceberg

Required Skills

DatabricksApache SparkPySparkScalaPythonSQLSchema evolutionData validationDelta LakeAWSGCPAzureStreamingKafka

Hard Skills

DatabricksApache SparkPySparkScalaPythonSQLSchema evolutionData versioningData validationDelta LakeGreat ExpectationsIcebergAWSGCPAzureAirflowPrefectDagsterTelemetryStreamingKafka

Soft Skills

collaborationadaptabilityproblem-solvinganalytical thinkingcommunication

Industry & Role

Industry Technology

Job Function Design and implement scalable data pipelines for telemetry and analytics

Role Subtype Data Engineer

Tech Domains Apache Spark, Python, SQL / PostgreSQL, AWS, Kafka, Databricks

Keywords for Your Resume

Data EngineerDatabricksApache SparkPySparkScalaPythonSQLSchema evolutionData validationDelta LakeAWSGCPAzureStreamingKafkaTelemetryData pipelines

Deal Breakers

Less than 5 years of data engineering experience, No experience with Databricks or Apache Spark, Lack of proficiency in Python or SQL, No experience with cloud platforms

Apply for this Position →

Get matched to jobs like this

Luna finds roles that fit your skills and career goals — no endless scrolling required.

Create a Free Profile

Senior Data Engineer, Network Clustering

Get matched to jobs like this