You need to enable JavaScript in order to use the AI chatbot tool powered by ChatBot

Data Engineer - Big Data

London

Data Engineer - Big Data

£350 - £480 per day
JTDE0001

London

The details

This vacancy has now expired.

Start your job search

Data Engineer

A chance to join a glabal energy provider in their South West London headquarters as a Data Engineer. This role will be an initial 9 month contract with a chance to go perm after.
We are looking for a Data Engineer to work with our Big Data platform team. The role will be responsible for supporting all aspects data engineering delivery from scoping, design, development and test support, solution acceptance, production implementation and support. The role will be within the platform team working closely with use case delivery teams. The role will combine elements of traditional SDLC execution as well as hands on data engineering facing off to business stakeholders, other IT teams and supporting the platform DevOps team. Responsibilities include:

* Develop data engineering pipelines and apply best practice in big data engineering
* Maintain and enhance the data engineering SDLC and framework.
* Production of design artefacts to describe the pipelines, build and deployment processes.
* Support, maintain and enhance the build and deploy framework for SQL SDLC.
* Support of development and test activity of use case delivery teams.
* Support performance testing and tuning examples.
* Support maintain and enhance reference implementations for key design patterns
* Demonstration and documentation of evidence to demonstrate Acceptance Criteria.
* Participation in daily scrum to update progress on tasks.
* Participation in sprint planning to decompose stories/PBIs into tasks and provide estimates.
* Support of End to End testing with external systems and components.

Essential
* Experience of big data engineering environment delivering data pipelines using Python, Scala or SparkSQL on Spark
* Experience of support and diagnosis in a data pipeline environment under AWS, Cloudera, Hortonworks or Microsoft Apache Hadoop implementations (such as EMR)
* In depth understanding of map/reduce and spark patterns to deliver data pipelines including evidence of best practice implementation.
* Strong data analysis skills including Entity Relationship Diagrams, Object Graph Mapping.
* Strong SQL in Oracle, SQL Server for data analysis.
* Strong Data Visualisation experience in PowerBI, Tableau.
* Strong Python development skills or analysis experience in Jupyter with R or Python
* Exposure to Scrum/Agile delivery methodologies.
* Experience of continuous integration/deployment and development techniques

Desirable
* Any Experience of the Palantir Foundry Platform
Solid grounding in Hadoop Big Data technologies: HDFS, YARN, SQOOP, Hive, Impala, Spark and general cluster management
* Exposure to streaming data applications such as Kafka or Kinesis
* Data migration delivery experience.
* Exposure to Python development and BDD frameworks.
* Demonstrable experience of requirements elicitation and documentation carried through to design and implementation.
* Excel, Access as data analysis, cataloguing and specification tools.
* Familiarity with ETL concepts and solutions particularly for big data engineering.
* Familiarity with scheduling and orchestration tools
* Certified scrum master qualification.

jobs

Related jobs

Group Reporting Manager

Salary:

£75,000 - £85,000 per annum + Excellent Benefits Package

Location:

Manchester, Greater Manchester

Industry

Consumer & Retail

Qualification

Fully qualified

Market

Commerce & Industry

Salary

£80,000 - £100,000

Job Discipline

Qualified Finance

Contract Type:

Permanent

Description

Exciting Group Reporting Finance Manager for a leading retail organisation based in Manchester offering hybrid working paying circa £80,000.

Reference

BBBH184314

Expiry Date

01/01/01

Nathan  Jones

Author

Nathan Jones
Find out more
Technical Reporting Manager

Salary:

£82,000 - £98,000 per annum + Excellent Benefits Package

Location:

Manchester, Greater Manchester

Industry

Insurance

Qualification

Fully qualified

Market

Financial Services

Salary

£80,000 - £100,000

Job Discipline

Qualified Finance

Contract Type:

Permanent

Description

Technical Reporting Manager role for an Insurance business based remotely with occasional travel to Manchester and London paying up to £98,000.

Reference

BBBH187917

Expiry Date

01/01/01

Nathan  Jones

Author

Nathan Jones
Find out more
Senior Commercial Finance Manager

Salary:

£75,000 - £85,000 per annum + Additional Benefits Package

Location:

Bolton, Greater Manchester

Industry

Consumer & Retail

Qualification

Fully qualified

Market

Commerce & Industry

Salary

£80,000 - £100,000

Job Discipline

Qualified Finance

Contract Type:

Permanent

Description

Excellent Senior Commercial Finance Manager role based in North Manchester paying up to £85,000 ideally with digital/ecommerce experience.

Reference

BBBH189991

Expiry Date

01/01/01

Nathan  Jones

Author

Nathan Jones
Find out more
Financial Controller

Salary:

£65,000 - £75,000 per annum + Additional Benefits Package

Location:

Stockport, Greater Manchester

Industry

Education

Qualification

Fully qualified

Market

Commerce & Industry

Salary

£70,000 - £80,000

Job Discipline

Qualified Finance

Contract Type:

Permanent

Description

Hands On Financial Controller role for an SME education business based in Stockport paying between £65,000 and £75,000 full or part time hours.

Reference

BBBH190012

Expiry Date

01/01/01

Nathan  Jones

Author

Nathan Jones
Find out more
View all jobs
posts

Related articles

Business Confidence Report 2025 | Your strategic compass for H2
Business Confidence Report 2025 | Your strategic compass for H2

Teaser

General

Content Type

Market Insight Reports

10/07/25

Summary

UK & Ireland Finance Job Market determined to outperform 2024 Despite mounting economic headwinds and a barrage of uncertainty both domestically and globally, 50% of finance, tech and legal

Teaser

Discover business confidence trends for H2 in our market research report

Read full article
Matthew Wilcox

by

Matthew Wilcox

Matthew Wilcox

by

Matthew Wilcox

Key trends shaping the 2025 European Private Capital Market
Key trends shaping the 2025 European Private Capital Market

Teaser

Executive Search

Content Type

General

08/07/25

Summary

In December 2024, pitchbook shared six key predictions in our 2025 EMEA Private Capital Outlook. Now, as we reach the halfway point of the year, this article offers a brief overview of how tho

Teaser

What’s ahead for private investing in Europe this year?

Read full article
Tracey Alper

by

Tracey Alper

Tracey Alper

by

Tracey Alper

The rise of fractional working in the UK & Ireland
The rise of fractional working in the UK & Ireland

Teaser

General

Content Type

General

01/07/25

Summary

Fractional working, the practice of engaging talent on a part-time, flexible basis across multiple organisations, is gaining momentum in the UK and Ireland. This trend has been catalysed by th

Teaser

Discover the emergence of fractional working in the UK and Ireland

Read full article
Alastair Paterson

by

Alastair Paterson

Alastair Paterson

by

Alastair Paterson