Senior Data Engineer.

Senior Data Engineer - Hybrid in London - Outside IR35

Overview

Our client is expanding their core data engineering function and hiring two Senior Data Engineers to strengthen a modern, scalable data platform. Both will be Databricks specialists, with one role focused on deep technical delivery and the other combining hands-on engineering with team leadership and upskilling responsibilities.

Key Responsibilities

• Build and maintain Databricks Delta Live Tables (DLT) pipelines across Bronze → Silver → Gold layers, ensuring quality, scalability, and reliability.

• Develop and optimise Spark (PySpark) jobs for large-scale distributed processing.

• Design and implement streaming data pipelines with Kafka/MSK, applying best practices for late event handling and throughput.

• Use Terraform and CI/CD pipelines (GitHub Actions or similar) to manage infrastructure and automate deployments.

• (For leadership-focused role) Mentor and upskill engineers, define coding standards, and embed engineering excellence across the team.

What's Expected

• Proven experience delivering end-to-end data pipelines in Databricks and Spark environments.

• Strong understanding of data modelling, schema evolution, and data contract management.

• Hands-on experience with Kafka, streaming architectures, and real-time processing principles.

• Proficiency with Docker, Terraform, and cloud platforms (AWS, GCP, or Azure) for scalable data infrastructure.

• Demonstrated ability to lead, coach, and elevate team capability, fostering best practices and knowledge sharing.