Data Engineer. Purpose of the position. Organisational position / Virtual Team. Direct Reports: Date Created: July 2017

Transcription

1 Data Engineer Business Unit: Strategy and Growth Reporting to: Data Engineering Direct Reports: None Date Created: July 2017 Purpose of the position The purpose of the Data Engineer role is to design, build, test and automate data driver solutions using cutting edge Technologies. These data solutions and products must meet the needs of all stakeholders, both internal and external to loyalty NZ. The Data engineer ensures that all Data Architecture, standards and procedures are adhered and that the Technical solution is of high calibre and fit for purpose in terms of usability, performance, functionality and availability, and is delivered to time and budget. The role will have strong working relationships with Loyalty NZ s Lab360 Professional Services team, Data Architect, Information Security and Risk Manager and Product Managers/Owners to ensure that Data engineering is delivering to the needs of the Business in accordance to defined principles and standards, and in accordance with our strategic plan. Organisational position / Virtual Team The role of the Data Engineer reports directly to the Data Engineering and has a very strong working relationship with all the business units at Loyalty NZ. Chief Executive Officer Chief Strategy & Growth Officer Products and Innovation Customer Experience FlyBuys CI Data Engineering Insights Data Engineer Platform Administrators Platform Visualisation Platform Analysts

2 Key Responsibilities 1. Design, Develop, Test, Automate and maintain robust, secure and reliable Big Data Solutions. Work on complex Big data projects and Products with a focus on collecting, parsing, managing, analysing, aggregating and visualising large data sets of data, to turn information into insights, across multiple platforms. Be able to take models and automate them for re-use across multiple platforms. Develop prototypes and proof of concepts for selected data solutions. Create, configure, implement, document and automate ingestion, enrichment and analytic pipelines using distributed big data and cloud platforms. Design and own data models for a Product or group of products. Understand the end to end vision for data at Loyalty NZ. Build data expertise and own data quality in allocated areas of ownership. Design and manage SLAs for all data sets (existing and new) in allocated areas of ownership. Identify and understand risks that exist with our data and actively work to mitigate these risks Collaborate with team members, Product Owners, stakeholders and other key people to understand data and reporting needs to ensure the technical solution meets the Business needs. 2. Contribute to the tools and technologies used to support our data platforms and work with the Data Engineering team to ensure the optimal choice for our data products and solutions Identify, evaluate and implement big data tools and frameworks. Use prototyping to test out new open-source data processing frameworks. Working with the wider team, evaluate and implement machine learning algorithms at scale. Implement tools and pipelines for analysis of large data sets, including creation of spikes. 3. Provide support for delivered solutions and products. Provide support for data quality, performance, availability and any other data related issues, as required. Collaborate with other teams to successfully triage data related issues and drive to resolution. 4. Ensure all work is delivered to defined standards and processes and to a high quality. Adhere to processes, policies, standards and procedures and act as a strong advocate for these with both team and key stakeholders. Provide input into the development of new processes, policies, standards and procedures to support data platform development, Project development and Loyalty s Data programme. Deliver work to a high standard and of a high quality. Peer review the work of others to ensure it maintains a high level of quality. 2

3 Adhere to corporate governance legislation and requirements, together with company policies. Ensure all aspects of Loyalty NZ data is managed and used in accordance with Security policies and NZ law. 5. Continually work on improving your skills and knowledge Keep up to date with the latest open-source technologies and Big data platform solutions and trends. Under guidance from your leader, embark on proof of concepts and prototyping to test out new Technologies. As required, provide coaching support to team members to build their skills, knowledge and capability to perform effectively in their role. Invest time to ensure that your personal development is integrated into personal objectives/goals and that your personal development plan is both challenging and reviewed regularly. Actively promote and drive vision and values to assist building the desired culture and work environment. Health & Safety All of our people have a responsibility for their own and others safety and wellbeing. This includes following all safety and wellbeing procedures and instructions, including reporting hazards, incidents and accidents and participating in safety and wellbeing initiatives and programmes as required. The Loyalty Way 1. Focuses on and contributes towards continuous improvements within the workplace through improving activities and processes to make things Easier, Better, Faster and Cheaper. 2. Contributes towards, and promotes, The Loyalty Way, LNZL s values and the achievement of our desired work environment; specifically an environment that stimulates individual, team and organisational growth and achievements, and encourages our people to think and behave in ways that achieve their goals through co-operative efforts. 3. Establishes and maintains effective relationships: Develops and fosters good proactive working relationships with all internal and external contacts. Ensures suppliers of goods and services conform to the standards of business ethics adopted by LNZL. 4. Leads and/or contributes to specific projects. 5. Undertakes additional responsibilities and activities, as and when requested and as mutually agreed with your leader. 3

4 Physical demands of the role This is not a physically demanding role Most challenging parts of the role 1. The data engineering team will be involved in multiple projects to varying degrees most of the time which challenges the ability to effectively balance the workload across projects. This also creates pressure for ensuring the Data engineering team follow defined processes and adhere to standards and deliver to a high standard. 2. Ensuring the team, team processes, tools and technology do not stay static but instead are constantly being reviewed and improved upon. Key Functional Relationships Internal: External: Reports to the data engineering Collaborates and liaises with Lab360 Professional Services teams Collaborates and liaises with the Data Architect, Applications Architect, Systems Architect, Technical Delivery, Security and Risk and Product Teams. Must have a collaborate and constructive working relationship with every other LNZL Staff member Loyalty Clients Loyalty Business Partners Other external Contractors and Suppliers engaged from time to time, and other Businesses Working environment 1. Open plan layout and moderate amount of noise that goes with it 2. Very little if no travel required 3. Standard office equipment Delegations of Authority Capital Expenditure: $0.00 Operational Expenditure: $0.00 Authorisation to Hire: N/A Authorisation to sign Contracts: N/A Responsible for: Number of Staff: None Budget: $0.00 Revenue: $0.00 4

5 Appraisal and Performance Criteria Formal appraisal will occur at least annually or more frequently when performance plans are re-negotiated. Performance will be measured against the performance plan negotiated at the beginning of the reporting period and against the other responsibilities identified above. This job is being carried out successfully when all responsibilities are being met. Person specifications Qualifications Essential: Bachelor s or Master s degree in computer science or Software Engineering Preferred: Big Data Developer Certification Experience & Knowledge Essential: Technical knowledge in building large scale big data solutions, using AWS, Hadoop and various scripting languages 3+ years of hands on experience with AWS services 3+ years of working experience with distributed data platforms (eg: Hadoop, Spark, Cassandra, Elastic, Kafka) 6+ years of extensive working knowledge in different programming or scripting languages like Java, Linux, Shell, Perl, Python and/or R. Experience working with structured, semi-structured and unstructured data sets including Social, Web logs and real time data feeds. Proficient in designing and automating efficient, robust and performant data layers Ability to productionize and administer data models (eg; SAS, R or Python) provided by our Data Science team in a manner that is easily maintainable and scalable for on-going use Able to tune Big Data solutions to improve performance and end user experience Understanding of Data Architecture, Data Analysis, Data Integration, Data Modelling and Data Warehousing Understanding of various Analytical and Database technologies including emerging technologies like columnar and Nosql databases, predictive analysis, data visualisation and machine learning Understanding of Restful APIs Understanding of PII and Data Security Knowledge of data modelling, data access, data storage and data security techniques Strong Business and communication skills and ability to interface with and collaborate with a variety of people and roles Excellent organisational, prioritisation and time management skills Broad based information technology experience and foundation in Systems development (eg: languages, methods, Development lifecycle, Project management approaches etc) 5

6 Preferred: Knowledge of Bi and visualisation tools such as Tableau/Qlik is a plus Experience with Jenkins, GitHub and nodejs is a plus Experience in developing machine learning models at scale Understanding of Open source development, tools and technologies and Restful Integration. Understanding of Loyalty Business concepts Skills The ideal appointee should have a high level of the following skills: Interpersonal skills and the ability to negotiate and influence Passionate about data. Able to emphasise methodology, modelling and governance Articulate, persuasive and enthusiastic Project management skills and the ability to meet deadlines Ability to manage a heavy workload Analytical, conceptual, problem solving, decisiveness and strategic thinking Team-Oriented and collaborative approach Personal Attributes The ideal appointee should be able to demonstrate: A results oriented person who is passionate about data and new data technologies and working in a forward thinking team and organisation A passion for excellence and a commitment to high standards. Be a self-starter with the ability to accept responsibility and self-manage. Comfortable speaking with both technical and non-technical audiences, and able to adjust language to suit A proactive and open minded consultative approach Comfortable with sometimes ambiguous and unstructured environments and can seek clarity and bring structure to a problem. Ability to balance multiple projects, often with simultaneous and competing deadlines Excited by new technologies and a desire to learn new things Familiar with best practises and able to contribute to standards and processes accordingly The ability to work effectively with an eclectic group of people from a variety of backgrounds An enquiring mind. Be curious. Integrity, loyalty to the organisation and a commitment to organisational objectives. Initiative, judgement and ability to creatively solve problems. Be detail conscious. Ability to work under pressure with an achievement focus. Enthusiasm, a thirst for learning and self-development. 6