Data Engineer - Big Data
London
Data Engineer - Big Data
London
The details
This vacancy has now expired.
Start your job searchData Engineer
A chance to join a glabal energy provider in their South West London headquarters as a Data Engineer. This role will be an initial 9 month contract with a chance to go perm after.
We are looking for a Data Engineer to work with our Big Data platform team. The role will be responsible for supporting all aspects data engineering delivery from scoping, design, development and test support, solution acceptance, production implementation and support. The role will be within the platform team working closely with use case delivery teams. The role will combine elements of traditional SDLC execution as well as hands on data engineering facing off to business stakeholders, other IT teams and supporting the platform DevOps team. Responsibilities include:
* Develop data engineering pipelines and apply best practice in big data engineering
* Maintain and enhance the data engineering SDLC and framework.
* Production of design artefacts to describe the pipelines, build and deployment processes.
* Support, maintain and enhance the build and deploy framework for SQL SDLC.
* Support of development and test activity of use case delivery teams.
* Support performance testing and tuning examples.
* Support maintain and enhance reference implementations for key design patterns
* Demonstration and documentation of evidence to demonstrate Acceptance Criteria.
* Participation in daily scrum to update progress on tasks.
* Participation in sprint planning to decompose stories/PBIs into tasks and provide estimates.
* Support of End to End testing with external systems and components.
Essential
* Experience of big data engineering environment delivering data pipelines using Python, Scala or SparkSQL on Spark
* Experience of support and diagnosis in a data pipeline environment under AWS, Cloudera, Hortonworks or Microsoft Apache Hadoop implementations (such as EMR)
* In depth understanding of map/reduce and spark patterns to deliver data pipelines including evidence of best practice implementation.
* Strong data analysis skills including Entity Relationship Diagrams, Object Graph Mapping.
* Strong SQL in Oracle, SQL Server for data analysis.
* Strong Data Visualisation experience in PowerBI, Tableau.
* Strong Python development skills or analysis experience in Jupyter with R or Python
* Exposure to Scrum/Agile delivery methodologies.
* Experience of continuous integration/deployment and development techniques
Desirable
* Any Experience of the Palantir Foundry Platform
Solid grounding in Hadoop Big Data technologies: HDFS, YARN, SQOOP, Hive, Impala, Spark and general cluster management
* Exposure to streaming data applications such as Kafka or Kinesis
* Data migration delivery experience.
* Exposure to Python development and BDD frameworks.
* Demonstrable experience of requirements elicitation and documentation carried through to design and implementation.
* Excel, Access as data analysis, cataloguing and specification tools.
* Familiarity with ETL concepts and solutions particularly for big data engineering.
* Familiarity with scheduling and orchestration tools
* Certified scrum master qualification.
Related jobs
Salary:
£35,000 - £45,000 per annum
Location:
Greater Manchester
Industry
Professional Services
Qualification
None specified
Market
Professional Services
Salary
£40,000 - £50,000
Job Discipline
Internal Audit
Contract Type:
Permanent
Description
Marks Sattin is partnering with a consultancy firm who are recruiting for a Senior Internal Auditor.
Reference
BBBH177451
Expiry Date
01/01/01
Author
David ClampAuthor
David ClampSalary:
£55,000 - £65,000 per annum
Location:
Leeds, West Yorkshire
Industry
Manufacturing
Qualification
Fully qualified
Market
Commerce & Industry
Salary
£60,000 - £70,000
Job Discipline
Qualified Finance
Contract Type:
Permanent
Description
This acquisitive, PE backed business has more than doubled in size over the past 5 years whilst growing their team to 250 in the UK.
Reference
BBBH179803
Expiry Date
01/01/01
Author
Hannah BulmerAuthor
Hannah BulmerSalary:
£150,000 - £175,000 per annum + equity, bonus and benefits
Location:
London
Industry
Technology
Qualification
Fully qualified
Market
Executive Search
Salary
£175,000 - £250,000
Job Discipline
Private Equity Executive Search
Contract Type:
Permanent
Description
Marks Sattin executive is working with a PE Backed, International Software business, going through a period of rapid organic growth
Reference
TA 3225
Expiry Date
01/01/01
Author
Executive SearchAuthor
Executive SearchSalary:
£180,000 - £200,000 per annum + Equity, Bonus and Benefits
Location:
England
Industry
Professional Services
Qualification
Fully qualified
Market
Executive Search
Salary
£175,000 - £250,000
Job Discipline
Private Equity Executive Search
Contract Type:
Permanent
Description
Marks Sattin executive search is working with a PE Backed Financial Services business is seeking a Best-In-Class CFO
Reference
TA 3224
Expiry Date
01/01/01
Author
Executive SearchAuthor
Executive SearchRelated articles
Teaser
TechnologyContent Type
General
19/03/24
Summary
The year 2024 emphasises the crucial role of RegTech — a beacon guiding firms through the evolving global regulatory landscape, enabling them to connect the dots between short-term challenges and
by
Paul Roche
Teaser
TechnologyContent Type
General
19/03/24
Summary
The UK tech sector retains the number 1 spot in Europe and number 3 in the world as sector resilience brings continued growth. With this demand comes stiff competition. London offers a wide range
by
Ghazal Mayahi
Teaser
GeneralContent Type
Career Advice
19/03/24
Summary
Are you considering a change of scenery? Just look at London - an incredibly diverse city, with over 1,600 languages spoken and 300 nationalities represented. At the centre of the UK's 4.2 milli
by
Kelvin Murphy