Senior Data Engineer (Spark Developer)

+ Job Description

We are looking for a Senior Data Engineer (Spark Developer) who can translate our current Python ETL Pipeline to Spark to deal with an increasing amount of transactional data from our streaming platform and create dimensional data models from events in our data warehouse.

You will clean, transform, and analyse vast amounts of raw data from Kafka using Spark to provide ready-to-use data models in Redshift to feed our developers and business analysts.

This involves both ad-hoc requests as well as data pipelines that are embedded in our production environment and feed our BI tool (Qlik Sense Saas)

+ Responsibilities;

As a Senior Data Engineer you take ownership of the data technology stack, its development and operation at Livespins. You’ll coordinate with developers and business analysts to maintain and develop data models in our data warehouse and feed our business intelligence application. In particular some of the tasks you will be responsible for:

• Create Spark jobs for data transformation and aggregation
• Coordinate with developers to define events schema and file formats
• Design batch data processing pipelines using EMR and Airflow
• Produce unit tests for Spark transformations and methods to handle schema changes
• Design dimensional data models in Redshift to feed our BI tool
• Manage our serverless data infrastructure (Airflow, Redshift and EMR) and code base

+ Skills;

• Experience consuming avro files from Kafka (Azure Event Hubs) on Apache Spark (AWS EMR)
• Spark query tuning and performance optimization
• Expert Python (i.e. PySpark, pandas) and SQL programming skills
• Experience using AWS services for data processing (i.e. EC2, S3, EMR, Redshift)
• Extensive experience in building DAGs using Apache Airflow
• Data warehousing and modelling experience in Redshift
• DevOps skills, and the ability to manage and optimise a data infrastructure on AWS