Data Engineer - Big Data
London
Data Engineer - Big Data
London
The details
This vacancy has now expired.
Start your job searchData Engineer
A chance to join a glabal energy provider in their South West London headquarters as a Data Engineer. This role will be an initial 9 month contract with a chance to go perm after.
We are looking for a Data Engineer to work with our Big Data platform team. The role will be responsible for supporting all aspects data engineering delivery from scoping, design, development and test support, solution acceptance, production implementation and support. The role will be within the platform team working closely with use case delivery teams. The role will combine elements of traditional SDLC execution as well as hands on data engineering facing off to business stakeholders, other IT teams and supporting the platform DevOps team. Responsibilities include:
* Develop data engineering pipelines and apply best practice in big data engineering
* Maintain and enhance the data engineering SDLC and framework.
* Production of design artefacts to describe the pipelines, build and deployment processes.
* Support, maintain and enhance the build and deploy framework for SQL SDLC.
* Support of development and test activity of use case delivery teams.
* Support performance testing and tuning examples.
* Support maintain and enhance reference implementations for key design patterns
* Demonstration and documentation of evidence to demonstrate Acceptance Criteria.
* Participation in daily scrum to update progress on tasks.
* Participation in sprint planning to decompose stories/PBIs into tasks and provide estimates.
* Support of End to End testing with external systems and components.
Essential
* Experience of big data engineering environment delivering data pipelines using Python, Scala or SparkSQL on Spark
* Experience of support and diagnosis in a data pipeline environment under AWS, Cloudera, Hortonworks or Microsoft Apache Hadoop implementations (such as EMR)
* In depth understanding of map/reduce and spark patterns to deliver data pipelines including evidence of best practice implementation.
* Strong data analysis skills including Entity Relationship Diagrams, Object Graph Mapping.
* Strong SQL in Oracle, SQL Server for data analysis.
* Strong Data Visualisation experience in PowerBI, Tableau.
* Strong Python development skills or analysis experience in Jupyter with R or Python
* Exposure to Scrum/Agile delivery methodologies.
* Experience of continuous integration/deployment and development techniques
Desirable
* Any Experience of the Palantir Foundry Platform
Solid grounding in Hadoop Big Data technologies: HDFS, YARN, SQOOP, Hive, Impala, Spark and general cluster management
* Exposure to streaming data applications such as Kafka or Kinesis
* Data migration delivery experience.
* Exposure to Python development and BDD frameworks.
* Demonstrable experience of requirements elicitation and documentation carried through to design and implementation.
* Excel, Access as data analysis, cataloguing and specification tools.
* Familiarity with ETL concepts and solutions particularly for big data engineering.
* Familiarity with scheduling and orchestration tools
* Certified scrum master qualification.
Related jobs
Salary:
+ Benefits
Location:
London
Industry
Private Equity
Qualification
Fully qualified
Market
Financial Services
Salary
£100,000 - £125,000
Job Discipline
Qualified Finance
Contract Type:
Contract
Description
Senior Fund Controller (Private Debt Funds) - 12m Contract
Reference
BBBH177293
Expiry Date
01/01/01
Author
Paul RocheAuthor
Paul RocheSalary:
£150,000 - £175,000 per annum + Equity, Bonus and Benefits
Location:
West Midlands
Industry
Business Services
Qualification
Fully qualified
Market
Executive Search
Salary
£175,000 - £250,000
Job Discipline
Private Equity Executive Search
Contract Type:
Permanent
Description
Marks Sattin executive search is working with a PE-Backed Environmental services business
Reference
TA 3255
Expiry Date
01/01/01
Author
Executive SearchAuthor
Executive SearchSalary:
£150,000 - £170,000 per annum + Equity, bonus and benefits
Location:
Essex
Industry
Business Services
Qualification
Fully qualified
Market
Executive Search
Salary
£125,000 - £175,000
Job Discipline
Private Equity Executive Search
Contract Type:
Permanent
Description
Marks Sattin Executive Search is working with a PE-backed multisite product-led distribution business
Reference
TA 3252
Expiry Date
01/01/01
Author
Executive SearchAuthor
Executive SearchSalary:
£150,000 - £175,000 per annum + Equity, Bonus and Benefits
Location:
Yorkshire and the Humber
Industry
Business Services
Qualification
Fully qualified
Market
Executive Search
Salary
£175,000 - £250,000
Job Discipline
Private Equity Executive Search
Contract Type:
Permanent
Description
Marks Sattin Executive Search has an exciting opportunity for a CFO to join a PE Backed infrastructure business
Reference
TA 3253
Expiry Date
01/01/01
Author
Executive SearchAuthor
Executive SearchRelated articles
Teaser
Finance & AccountingContent Type
General
25/04/24
Summary
Over the years, we have developed a strong reputation as a leading Senior Finance and Executive Search firm. We’re proud to have well-established teams of finance and accounting recruitment cons
by
Neil Burton
Teaser
GovernanceContent Type
Fintech
18/04/24
Summary
The role of risk and compliance in financial services As a sizeable, growing portion of the financial services sector, risk and compliance play a vital role in ensuring that firms conduct busine
by
David Clamp
Teaser
Executive SearchContent Type
General
18/04/24
Summary
Private equity (PE) is a growing industry that has always attracted ambitious top talent due to its high risk/high gain capital investment. Pre-covid, we saw the global private equity industry b
by
Tracey Alper