Senior Data Engineer

The role 

We are seeking a skilled Senior Data Engineer to design, build and maintain scalable data pipelines and infrastructure within a fast-paced agile data engineering team. You will play a crucial role in modernizing our data platform to enhance product offerings, closely collaborating with stakeholders across the organization to maximize the value of data for reporting, analytics, and decision-making processes.

In this role, you'll help transform our multi-cloud data landscape, working with streaming technologies, cloud data warehouses, and Lakehouse architectures to unlock new insights and capabilities for our business units. Additionally, you will leverage automation and generative AI to increase operational efficiency and enhance self-service capabilities across the organization.

What you will do 

Pipeline Development & Engineering

• Design, develop, and maintain production ETL/ELT pipelines in Python and SQL

• Build and optimize large-scale data processing jobs using Apache Spark/PySpark balancing performance, cost and maintainability

• Build and optimize complex Airflow DAGs for workflow orchestration (scheduling, monitoring, alerting, error handling)

• Develop and manage data transformation logic using dbt for version control, testing, and documentation

• Process and transform datasets ranging from GBs to TBs efficiently

 

Lakehouse Architecture & Design

• Design and implement scalable data lake medallion-style architectures using S3 and modern table formats (Apache Iceberg, Delta Lake)

• Design schema evolution & Implement partitioning and optimize table layouts (partitioning, clustering, compaction) for both batch and streaming workloads

 

Data Platform Infrastructure

• Design and implement reusable data pipeline frameworks and libraries

• Implement schema registries and enforce data contracts between pipeline stages

• Build and maintain CI/CD pipelines for data workflows (Git-based deployments, testing, validation)

• Establish observability practices: logging, monitoring, alerting, anomaly detection, pipeline health, data freshness, and quality metrics

 

Data Quality & Reliability

• Implement comprehensive data quality frameworks and validation checks within pipelines

• Establish data contracts that define expected schema, quality, and delivery guarantees

• Design and implement SLIs/SLOs for critical data pipelines (latency, throughput, accuracy)

 

Your experience 

Required

Core Data Engineering (5+ years)

• 5+ years of professional data engineering experience

• Demonstrated expertise designing and implementing scalable data pipelines

• Production experience building Airflow DAGs at scale

• Strong proficiency in Python (data processing, libraries like pandas, sqlalchemy)

• Advanced SQL skills (complex queries, window functions, performance tuning) and data modelling

• Experience with dbt or similar data transformation tools

• Hands-on experience with Apache Spark/PySpark optimization

• Proficiency with CI/CD practices for data pipelines (Git workflows, testing, deployment automation)

 

Data Architecture & Platforms

• Experience with modern table formats (Apache Iceberg, Delta Lake) for Lakehouse architectures

• Experience designing dimensional models and/or star schemas for analytics

• Understanding of medallion architecture (bronze/silver/gold layers)

• Practical experience with at least one cloud data warehouse (e.g. S3, Athena, Redshift)

 

Data Quality, Contracts & Observability

• Experience building data quality frameworks and comprehensive validation testing

• Understanding of SLIs/SLOs and their application to data pipelines

• Experience designing data contracts and schema evolution strategies

• Demonstrated experience implementing monitoring and alerting for production data pipelines

 

Advantageous

• • Working experience with streaming data platforms (Kafka, Kinesis, Spark Streaming)

  • Experience using AI coding assistants (Cursor, Claude Code, GitHub Copilot) to accelerate development

  • Infrastructure as Code (Terraform) for data platform deployment

  • Experience with Kubernetes or Docker containerization

  • Familiarity with data catalogs and metadata management platforms

  • Nice to have : JAVA experience