About this role
This role involves designing and supporting scalable big data solutions with a focus on AI integration and data governance.
Key Responsibilities
- Design data pipelines
- Implement data governance
- Optimize data lakehouse
- Support AI/ML models
- Troubleshoot data systems
Technical Overview
Technical environment includes Databricks, Snowflake, GCP, Spark, PySpark, Delta Lake, and AI/ML tools like RAG and LLMs, emphasizing data pipelines, security, and performance optimization.
Ideal Candidate
The ideal candidate is a senior data engineer with over 7 years of experience in big data platforms, especially Databricks, Snowflake, and GCP. They have expertise in building scalable data pipelines, implementing data governance, and supporting AI/ML integrations like RAG and LLMs.
Must-Have Skills
DatabricksSnowflakeSparkPySparkPythonSQLData lakehouseDelta LakeETLData pipelinesData governanceData securityData qualityBig data platformsStreaming technologiesKafkaVector databases
Nice-to-Have Skills
Retrieval-Augmented GenerationRAGSemantic searchLLMData workflowsData governanceData securityData qualityMonitoringOptimization
Tools & Platforms
DatabricksSnowflakeGoogle Cloud PlatformKafkaGitDevOpsSource controlBig data platformsStreaming technologiesVector databases
Required Skills
DatabricksSnowflakeSparkPySparkPythonSQLData lakehouseDelta LakeRAGSemantic searchLLMData pipelinesData governanceData securityData qualityETLData workflowsGitDevOpsSource controlKafkaVector databases
Hard Skills
DatabricksSnowflakeGoogle Cloud PlatformSparkPySparkPythonSQLData lakehouseDelta LakeRetrieval-Augmented GenerationRAGSemantic searchLLMData pipelinesData governanceData securityData qualityETLData workflowsGitDevOpsSource controlBig data platformsStreaming technologiesKafkaVector databases
Soft Skills
CollaborationProblem-solvingCommunicationTeamworkTroubleshootingMonitoringOptimizationContinuous improvement
Keywords for Your Resume
Senior Data Platform EngineerDatabricksSnowflakeGoogle Cloud PlatformSparkPySparkPythonSQLData lakehouseDelta LakeRetrieval-Augmented GenerationRAGSemantic searchLLMData pipelinesData governanceData securityData qualityETLData workflowsGitDevOpsSource controlBig data platformsStreaming technologiesKafkaVector databases
Deal Breakers
Less than 7 years of experience, Lack of experience with Databricks or Snowflake, No background in big data or data pipelines, Unable to work in hybrid mode in Elk Grove, CA
Get matched to jobs like this
Luna finds roles that fit your skills and career goals — no endless scrolling required.
Create a Free Profile