Data Management & Warehusing WHITE PAPER Data Warehuse Gvernance DAVID M WALKER Versin: 1.1 Date: 06/04/2007 Data Management & Warehusing 138 Finchampstead Rad, Wkingham, Berkshire, RG41 2NU, United Kingdm http://www.datamgmt.cm
White Paper - Data Warehuse Gvernance Table f Cntents Table f Cntents... 2 Synpsis... 3 Intended Audience... 3 Abut Data Management & Warehusing... 3 Intrductin... 4 What is gvernance?... 5 What affects the data warehuse?... 6 What is affected within the data warehuse?... 7 Hw ften will change ccur?... 9 What rganisatinal structures are needed?... 10 Executive Spnsr(s)... 10 Steering Cmmittee... 11 Prgramme Management... 11 Implementatin Teams... 12 Explitatin Teams... 12 User Frums... 12 Certificatin Cmmittee... 13 Prject Develpment Methdlgies... 14 Waterfall... 14 Iterative... 14 Agile... 15 Cwby Cding... 16 Best practices... 17 Mmentum... 17 Mnitr and Measure... 17 Priritise... 17 Light tuch... 17 Cmmunicatin... 17 Standards... 18 Track prcesses... 18 Understand cst and value... 18 Cntinuus Learning and Imprvement... 18 Pilt & Prttype... 18 Authrity t act... 18 Plan fr the lng term... 19 Gvernance and Success... 19 Summary... 20 Appendices... 21 Appendix 1 - Prgramme r Prject?... 21 Appendix 2 The "Declaratin f Interdependence"... 22 Appendix 3 - Team Values and Principles... 23 References... 24 Web resurces... 24 Acknwledgements... 24 Cpyright... 24 2007 Data Management & Warehusing Page 2
White Paper - Data Warehuse Gvernance Synpsis An rganisatin that is embarking n a data warehusing prject is undertaking a lng-term develpment and maintenance prgramme f a cmputer system. This system will be critical t the rganisatin and cst a significant amunt f mney, therefre cntrl f the system is vital. Gvernance defines the mdel the rganisatin will use t ensure ptimal use and reuse f the data warehuse and enfrcement f crprate plicies (e.g. business design, technical design and applicatin security) and ultimately derive value fr mney. This paper has identified five surces f change t the system and the aspects f the system that these surces f change will influence in rder t assist the rganisatin t develp standards and structures t supprt the develpment and maintenance f the slutin. These standards and structures must then evlve, as the prgramme develps t meet its changing needs. Dcumentatin is nt understanding, prcess is nt discipline, frmality is nt skill 1 The best gvernance must nly be an aid t the develpment and nt an end in itself. Data Warehuses are successful because f gd understanding, discipline and the skill f thse invlved. On the ther hand systems built t a template withut understanding, discipline and skill will inevitably deliver a system that fails t meet the users needs and sner rather than later will be left n the shelf, r maintained at a very high cst but with little real use. Intended Audience Reader Executive Business Users IT Management IT Strategy IT Prject Management IT Develpers Recmmended Reading Synpsis and Summary Entire Dcument Entire Dcument Entire Dcument Entire Dcument Entire Dcument Abut Data Management & Warehusing Data Management & Warehusing is a specialist cnsultancy in data warehusing based in Wkingham, Berkshire in the United Kingdm. Funded in 1995 by David M Walker, ur cnsultants have wrked fr majr crpratins arund the wrld including the US, Eurpe, Africa and the Middle East. Our clients are invariably large rganisatins with a pressing need fr business intelligence. We have wrked in many industry sectrs but have specialists in Telc s, manufacturing, retail, financial and transprt as well as technical expertise in many f the leading technlgies. Fr further infrmatin visit ur website at: http://www.datamgmt.cm 1 Agile Sftware Develpment, Cckburn, A., Addisn-Wesley, 2002 2007 Data Management & Warehusing Page 3
White Paper - Data Warehuse Gvernance Intrductin Data warehuse gvernance defines the mdel the rganisatin will use t ensure ptimal use and re-use f the data warehuse and enfrcement f crprate plicies (e.g. business design, technical design and applicatin security). Gvernance structures must be in place as early as pssible, ideally befre the prject starts and befre develpment begins. They are implemented in parallel with the initial develpment t ensure that they wrk tgether frm the utset. A data warehuse is, by definitin, a system that interacts with large parts f the rganisatin. At a technical level this invlves the number f systems feeding data t and being fed data frm the data warehuse. At a human level it relates t the peple wh interact with the system. These include thse querying the system, thse perating the technical envirnment and thse directing the business wh may nt use the system directly but will mandate that thers d s. The published literature n the subject falls int three categries: Hme grwn what a specific prject did within a specific envirnment Vendr specific what t d when using the vendrs tl set System Integratr prprietary hw an SI say they will d it if they are given the cntract The prblem is that the data warehuse interacts with and acrss s much f the rganisatin. A prgramme must deply standards and prcesses that integrate int the rganisatins existing plicies and prcedures. This paper lks at the areas that must be cvered by gvernance and discusses sme f the standards, ptins and issues. These ideas shuld be incrprated int the gvernance f the data warehuse. 2007 Data Management & Warehusing Page 4
White Paper - Data Warehuse Gvernance What is gvernance? In the intrductin gvernance is defined as the mdel the rganisatin will use t ensure ptimal use and re-use f the data warehuse and enfrcement f crprate plicies, but what des it mean? Data Warehuses are nt a single shrt-term prject. They are lng-term prgrammes f wrk that includes a number f prjects and a significant amunt f maintenance. The decisin t deply a data warehuse cmmits the business t a prgramme that needs t be cntrlled t ensure that it prvides business benefit and value fr mney. The rle f gvernance is t prvide the plicies, prcesses and prcedures necessary t ensure that the prgramme f wrk is effective. These must be clearly cmmunicated t everyne invlved. Gvernance cvers the integrated management f the prgramme frm its initial develpment, thrugh prductin running t end-f-life clse dwn. The gvernance structures must als be reviewed ver the lifetime f the prgramme. This is t ensure that the gvernance is neither t little t allw effective cntrl r s much such that prgramme wrk is stifled. Different rganisatins als have different requirements fr the level f detail and breadth f scpe that shuld be cvered by Data Warehuse gvernance. Fr example, des gvernance imply cding standards? Fr sme peple this is far t much detail, fr thers it is essential that the gvernance f the prgramme ensures that specific standards are develped and that staff adhered t them. The key t designing a suitable level f gvernance within an rganisatin is in understanding the fllwing: What events affect the data warehuse envirnment? What is affected in the data warehuse by the impacting event? What type f rganisatin is needed t supprt the data warehuse? What are the plicies and prcedures needed t cntrl the data warehuse? Frm this it is pssible t develp the standards, prcesses and cmmunicatins that will bth drive the prgramme frward and culturally fit the specific rganisatin. The prcesses, used in cnjunctin with the standards, are the way in which decisins are made. Gd prcesses enabling quick, effective and well-cmmunicated decisins and cnsequently effective management f the slutin. The cnverse is als true as pr standards and prcesses lead t delays and unreslved issues that discredit the data warehuse and create significant cst r verheads that eventually destry the prgramme. 2007 Data Management & Warehusing Page 5
White Paper - Data Warehuse Gvernance What affects the data warehuse? Data Management & Warehusing has identified five types f change that impact a data warehuse: Demand Demand is the change needed t the system t give the users what they want. This may include lading new data, pssibly frm new surces r restructuring existing infrmatin. It can als include adjusting system availability r the time at which data is laded r any ther user requirement. External Change A data warehuse has a large number f systems feeding it. The systems perate in a cmplex and integrated envirnment. Fr example, a data warehuse might have three surce systems and each surce system is nly allwed t perfrm upgrades and patches nce a quarter. As a result the data warehuse is faced with twelve changes a year r ne a mnth. In an ideal envirnment these changes will be develped and regressin tested in line with the change in the surce system. In practice mst data warehuses have many mre surces and mre frequent changes. When these changes are nt cmmunicated they becme issues. Managing external change prevents issues and therefre reduces emergency fixes and tactical slutins Maintenance All cmputer system have t manage rutine maintenance issues such as sftware patches, upgrades t sftware r hardware, special backups fr year end, etc. It is easy t ignre upgrades - the system is currently wrking s why bther? There cmes a pint when the sftware r hardware vendr mves the prduct ut f supprt. At that pint any system failure can have critical cnsequences. Rutine maintenance shuld be regarded as a critical part f business as usual and allwance made fr its impact. Risks Risks t the data warehuse can be identified by thse in cntrl. These risks need t be addressed befre they affect the system. Over recent years typical risks have included changes in regulatry plicies, Year 2000 issues, currency changes (e.g. revaluatins, mving t the Eur 2 r remving trailing zers 3 ). There are als risks frm crprate mergers and acquisitins and frm the bankruptcy f ther cmpanies that prvide data t supplement the data warehuse (e.g. with market share infrmatin). These risks will create significant wrk with tight deadlines. 2 Currently 13 cuntries are in the Eur zne (http://en.wikipedia.rg/wiki/eurzne#official_members) 3 49 cuntries including Turkey, Greece, Israel, Brazil and Argentina (http://www.allabutturkey.cm/ytl.htm) 2007 Data Management & Warehusing Page 6
White Paper - Data Warehuse Gvernance Issues In an ideal wrld all f the abve wuld ensure that the data warehuse wuld never have any issues. The reality is a cntinuus flw f lw-level prblems and ccasinally critical issues that need t be handled. In sme ways this can be a measure f the success f the slutin that peple care enugh t reprt issues and need supprt. 4 There are cst and resurce implicatins in handling these issues. What is affected within the data warehuse? 5 Once a data warehuse develpment has begun these areas can be affected: Requirements Catalgue All data warehuse prjects shuld have a cmprehensive catalgue f requirements. These will be develped at the start f the system and maintained when new r changed requirements emerge. Requirements are nt the same as the functinality f the system. This is because nt all requirements may be met, hwever they prvide a catalgue f wrk that the business needs t be delivered. This als prvides a check that new demands have nt already been met by anther means. Technical Infrastructure The technical infrastructure is the hardware, sftware and netwrk used t build the system. It can be affected by many different factrs including upgrades fr better perfrmance, platfrm re-hsting r a re-architecting exercise (e.g. mving frm desktp t web-based clients) Cnfiguratin Management Cnfiguratin management cvers all aspect f managing the develped cde and any system cnfiguratin files that exist utside the cde. Suitable tls must be used t ensure that the peple respnsible fr the maintenance f the system can lk at all the different versins f cde ver time. This enables the deplyment f new cde. It als can help when there is a need t rll back changes, fr example reverting t lder cde after a failed upgrade r t read an ld frmat surce file. It is als imprtant that the cde can be related t data mdels fr the data warehuse and t the cde versins and data mdels f the external systems that either supply r use data frm the data warehuse. 4 One Data Management & Warehusing client rganisatin with 2000 users and a 20Tb system raised 10,000 issues in ne year pst implementatin r abut 40 per day 5 This sectin refers t varius dcuments such as requirements etc. Depending n wh has develped the system they will have their wn names and templates fr these dcuments. Fr cnsistency this dcument refers t the templates and tls used in the Data Management & Warehusing (http://www.datamgmt.cm) Dcumentatin Radmap which can be fund at: http://www.datamgmt.cm/index.php?mdule=article&view=77 2007 Data Management & Warehusing Page 7
White Paper - Data Warehuse Gvernance Test Management Test management is abut asking questins t ensure that the impact f any change is well understd and that preventable errrs are avided. Questins like: After a change is made, hw is it tested? Is there sufficient data t be cnsidered a representative sample? Is the whle system regressin tested, r are changes islated and therefre separately testable? Hw can changes r testing be mdified s that tests can be islated? Hw lng des it take t test a change and will this impact emergency fixes t the system? Des the change affect the perfrmance r the data quality f the slutin? Can the figures be calculated by an alternative independent methd t ensure that the numbers prduced are crrect? Des the prject have sufficient resurces (e.g. size f machine, peple, time, etc.) t perfrm the tests? Data Stewardship Data stewardship is abut understanding wh is respnsible fr the data within the system. Again the rle is abut asking questins: Wh decides which data quality rules shuld be applied? Where is data cleansed? Wh is respnsible fr cleaning the data? Can systems r prcesses be changed t imprve the quality f the data befre it is sent t the data warehuse? Is data cleansed in the surce system and if s is this wrk als dne n histrical data? What is the impact if histrical data that is already in the data warehuse is cleansed in the surce? What is the relatinship between time and data e.g. is 95% f the data available after ne day gd enugh r must 100% f the data be there befre it can be used? Service Level Agreement The service level agreement defines much abut the perfrmance and availability f the system. It will include the times set aside fr tasks such as backups and maintenance. It als defines the times that the system will be available t the users. It shuld als include agreements n hw t respnd in case f emergencies and hw t manage systems failures. The Service Level Agreement als defines hw any change that affects the system (either psitively r negatively) needs t be cmmunicated t the user cmmunity. Supprt Mdel The supprt mdel describes the prcesses and escalatin paths fr any user supprt request. This mdel must be updated t reflect any changes t the system r any changes in the rganisatin. The supprt mdel shuld be mdified t imprve the respnsiveness f supprt by understanding the frequently asked questins and imprving the respnse time explicitly n these items. 2007 Data Management & Warehusing Page 8
White Paper - Data Warehuse Gvernance Training Plans Users and peratrs will need t be trained t use the system when it is first deplyed. There is need fr n-ging training as the system changes, when new staff arrive r when parts f the slutin are either cmmissined r de-cmmissined. Users will als need refresher curses t reduce the number and cst f calls t the supprt desk. Schedules The system will have an peratinal schedule that will determine nt nly when users can be n the system (als cvered in part by the service level agreement). The schedule defines which jbs t extract, transfrm r lad data can be run, when they are run and the dependencies between them. Mnitring The system must be mnitred and alerts directed t sme functin that can priritise them, act and if necessary escalate apprpriately. The prcess by which this is managed and its efficiency is critical t prviding a viable service t the users. Mnitring als ensures that service level agreements are met. Analysing the system events allws technical supprt t imprve the system in the same way that analysing supprt calls assists the supprt team imprve the supprt. The analysis must lk fr ways in which t bring abut significant imprvement t the system by either changing the system r imprving the respnse t frequently ccurring events. Hw ften will change ccur? The ten areas affected by change and the five types f change described abve result in the table belw alng with the likely frequency f any given type f change n a specific area f the data warehuse: Demand External Change Maintenance Risks Issues Requirements Catalgue 3 1 1 2 1 Technical Infrastructure 2 1 2 1 1 Cnfiguratin Management 1 3 3 1 3 Test Management 2 3 3 1 3 Data Stewardship 2 3 1 1 3 Service Level Agreement 2 2 2 2 1 Supprt Mdel 1 1 2 1 3 Training Plans 1 1 1 1 3 Schedules 2 3 2 1 2 Mnitring 1 3 3 1 1 1: Infrequently 2: Occasinally 3: Frequently This table can be used thrughut the life cycle f the prgramme t priritise gvernance develpment t ensure that the mst frequent changes are under the tightest cntrls. 2007 Data Management & Warehusing Page 9
White Paper - Data Warehuse Gvernance What rganisatinal structures are needed? This paper has identified a large amunt f ptential change frm different surces n different prcesses n a lng-term prgramme. It is necessary t have sufficient rganisatinal structure in place n cntrl the changes and the cmmunicatin f that change. The basic mdel rganisatin fr this can be described as fllws: Executive Spnsr(s) User Frums Steering Cmmittee Certificatin Cmmittee Explitatin Teams Prgramme Management Implementatin Teams Figure 1 - Data Warehuse Organisatinal Structure Executive Spnsr(s) An executive spnsr is a senir (executive) member f the rganisatin with an active interest and an understanding f the business uses f the data warehuse. A system f this magnitude and with such crss-functinal reach needs t be spnsred at the very highest level. It is likely that the executive spnsr fr the slutin as a whle will be cmmitted t the idea f develping better strategic infrmatin. A number f executive spnsrs fr individual functinal develpments will assist this persn. The spnsrs f individual functinal develpments are respnsible fr the delivery f the benefits prmised fr any given develpment. The executive spnsr(s) als need t ensure that the steering cmmittee is directing the develpment in line with the visin, strategic directin and pririties fr the rganisatin. 2007 Data Management & Warehusing Page 10
White Paper - Data Warehuse Gvernance Steering Cmmittee The steering cmmittee is the hub f the n-ging data warehuse slutin frm a management perspective. It is here that the develpment is aligned with the business bjectives. Mnitring ensures that the prgramme is delivering the right prjects at the right time and at fair value. By setting the principles and plicies the steering cmmittee can cntrl the directin that the develpment ges in. This maintains an enterprise wide business perspective fr the data warehuse. The steering cmmittee is als the centre f cmmunicatin. It takes input frm the user frums and the certificatin cmmittee as t what is needed. In return the cmmittee manages the expectatins f bth the business and IT departments as t what is pssible. The steering cmmittee is mst effective when it is empwered t make quick, effective, crss-functinal decisins and cmmunicate t all the stakehlders. Prgramme Management Prgramme management is the c-rdinated management f a prtfli f prjects t achieve a set f business bjectives. It delivers the c-rdinated supprt, planning, priritisatin and mnitring f prjects t meet changing business needs. T achieve the business bjectives the prgramme manager defines a series f prjects with quantifiable benefits that tgether will meet the lng-term bjectives f the rganisatin. These prjects may run simultaneusly r at least verlap with each ther and they may share resurces. Such prjects might have sme resurces devted t the prject fr a perid f time. In additin prjects will require a range f specialists available frm the prgramme team whse services are used fr shrter perids f time. Prjects within the prgramme can als be linked. Delays within ne prject will then cause knck n effects in ther prjects due t lgical links between tasks in bth f them. Resurce and timing cnflict reslutin is als an integral part f the functin f prgramme management. The prgramme is usually reflected in the management structure with a prgramme manager t whm the prject managers will reprt. The prgramme manager will be cncerned with recruiting and maintaining their prject management teams, allcatin f key, shared resurces and n the integratin f the deliverables frm each prject int the verall prgramme. In this envirnment every prject plays its part twards the rganisatin s ultimate aims and bjectives. Often, as prjects are cmpleted, this translates back int a revised set f crprate bjectives. 2007 Data Management & Warehusing Page 11
White Paper - Data Warehuse Gvernance Implementatin Teams The implementatin teams are thse that will actually d the wrk. They will cnsist f a team f peple (either fr sme r the entire prject) that will fulfil the fllwing rles: Prject Manager Technical Architect Systems Administratr Netwrk Administratr Database Administratr Metadata Administratr Data Mdeller ETL Develper Frnt End Tl / Reprt Develper Prduct Specialist Test Manager Many rganisatins may have utsurced sme r all f this functinality 6. The resurces may have the skills and the time t fulfil mre than ne rle, r fr sme prjects mre than ne resurce may be required t perfrm a given rle. The management and resurce levels must be determined by the prject scpe. Explitatin Teams Whilst the implementatin teams fcus n building the system the explitatin team are trying t ensure that the business is extracting the mst value frm the slutin. Explitatin teams wrk n the current versin f the system t help the business use the current system and develp new requirements t explit the system further. Typical rles fr the teams will include: Explitatin Team Leader Business Analyst Business Requirements Specialist Technical Authr / Dcumentatin Specialist Trainer End User Supprt Specialist Cmmunicatins Specialist Helpdesk Supprt Specialist As with the implementatin teams the resurce level is determined by the prject scpe. User Frums The prgramme shuld als cnsider setting up a number f user frums that invlve end users, subject matter specialists and staff frm the explitatin teams. These frums are useful t allw varius teams t express their issues and aspiratins fr the system and expand n the art f the pssible. These frums shuld als appint representatives n the steering cmmittee t represent their perspective n the system. 6 Data Warehuses benefit frm peple with detailed knwledge f the subject and utsurced develpment t cding shps with n experience is ften mre cstly than the initial perceived savings. When chsing resurces Data Management & Warehusing recmmend that rganisatins lk at the skills f named individuals n the prject 2007 Data Management & Warehusing Page 12
White Paper - Data Warehuse Gvernance Certificatin Cmmittee A number f grups within the rganisatin will als assess the data warehuse t ensure that it is fit fr purpse. These grups can either be cnsulted individually r brught tgether as a cmmittee t advise the prgramme. They include: Audit Mst rganisatins will have sme internal audit functin. The audit functin will need t ensure that the system meets the required standards and has sufficient checks and balances especially if the system is used fr any frm r statutry r fiscal reprting. The internal audit functin shuld als examine hw the system is managed. Where there is n (required) internal audit functin an individual r team representing thse with a stake in the quality f the infrmatin shuld perfrm the rle f auditr. Regulatry Cmpliance Depending n the rganisatin and industry there may be a need t cmply with requirements frm gvernment, lcal administratin r industry regulatrs ver the data that is held and wh has access t that data. Current examples include data prtectin law 7, financial management laws such as Sarbanes- Oxley Act 8 and Mifid 9 and telecms regulatrs such as Ofcm 10. IT Strategy & Architecture Many large rganisatins have a team respnsible fr the strategy and architecture f all IT systems. This team ensures inter-perability, re-use and cst reductin f IT systems. The team will need t review and assess any technical infrastructure and changes t the slutin that are prpsed. Security Security f infrmatin is a grwing cncern fr many rganisatins. Either a security team, if ne exists in the rganisatin, r a nminated individual shuld review the system peridically t ensure its security. 7 DPA Law website: http://www.dpalaw.inf/ 8 Sarbanes Oxley Act: http://en.wikipedia.rg/wiki/sarbanes-oxley_act 9 Mifid: http://www.fsa.gv.uk/pages/abut/what/internatinal/eu/fsap/mifid/index.shtml 10 Ofcm website: http://www.fcm.rg.uk/ 2007 Data Management & Warehusing Page 13
White Paper - Data Warehuse Gvernance Prject Develpment Methdlgies There are fur main methdlgies fr the develpment itself: Waterfall The waterfall mdel has the fllwing phases that are fllwed in rder: Requirements specificatin Design Build Integratin Testing Deplyment Maintenance T fllw the waterfall mdel, ne prceeds frm ne phase t the next in a purely sequential manner, starting each phase nce the previus ne has been cmpleted. Phases f develpment in the waterfall mdel are thus discrete and there is n jumping back and frth r verlap between them, hwever there are practical variatins n this mdel. Iterative 11 The basic idea behind iterative develpment is t develp a sftware system incrementally. This allws the develper t take advantage f what was being learned during the develpment f earlier, incremental, deliverable versins f the system. Learning cmes frm bth the develpment and use f the system where pssible. Key steps in the prcess are t start with a simple implementatin f a subset f the sftware requirements. The system is then iteratively enhanced in a sequence f evlving versins until fully implemented. At each iteratin design mdificatins are made and new functinal capabilities are added. The prcess itself cnsists f the initializatin step, the iteratin step and the prject cntrl list. The initializatin step creates a base versin f the system. The gal fr this initial implementatin is t create a prduct t which the user can react. It shuld ffer a sample f the key aspects f the prblem and prvide a slutin that is simple enugh t understand and implement easily. T guide the iteratin prcess, a prject cntrl list is created that cntains a recrd f all tasks that need t be perfrmed. It includes such items as new features t be implemented and areas f redesign f the existing slutin. The prject cntrl list is cnstantly being revised because f the analysis phase. The iteratin step invlves the redesign and implementatin f a task frm the prject cntrl list and the analysis f the current versin f the system. The gal fr the design and implementatin f any iteratin is t be simple, straightfrward and mdular, supprting redesign at that stage r as a task added t the prject cntrl list. 11 Descriptin is an edited frm f the text frm Wikipedia: http://en.wikipedia.rg/wiki/iterative_develpment 2007 Data Management & Warehusing Page 14
White Paper - Data Warehuse Gvernance The cde can represent the majr surce f dcumentatin f the system. The analysis f an iteratin is based upn user feedback and the prgramme analysis facilities available. It invlves analysis f the structure, mdularity, usability, reliability, efficiency and achievement f gals. The prject cntrl list is mdified in light f the analysis results. Agile 12 Agile methds are a family f develpment prcesses that build n iterative develpment, nt a single apprach t sftware develpment. Agile evlved in the mid 1990s as part f a reactin against "heavyweight" methds such as the waterfall mdel f develpment that ften became heavily regulated, regimented and micr-managed. The prcesses riginating frm this use f the waterfall mdel were seen as bureaucratic, slw, demeaning and incnsistent with the ways that sftware engineers actually perfrm effective wrk. 13 In 2001 seventeen prminent figures in the field f agile develpment (then called "light-weight methdlgies") came tgether t discuss ways f creating sftware in a lighter, faster, mre peple-centric way. They created the Agile Manifest and accmpanying agile principles: 14 The highest pririty is t satisfy the custmer thrugh early and cntinuus delivery f valuable sftware. Welcme changing requirements, even late in develpment. Agile prcesses harness change fr the custmer's cmpetitive advantage. Deliver wrking sftware frequently, frm a cuple f weeks t a cuple f mnths, with a preference t the shrter timescale. Business peple and develpers must wrk tgether daily thrughut the prject. Build prjects arund mtivated individuals. Give them the envirnment and supprt they need and trust them t get the jb dne. The mst efficient and effective methd f cnveying infrmatin t and within a develpment team is face-t-face cnversatin. Wrking sftware is the primary measure f prgress. Agile prcesses prmte sustainable develpment. The spnsrs, develpers and users shuld be able t maintain a cnstant pace indefinitely. Cntinuus attentin t technical excellence and gd design enhances agility. Simplicity--the art f maximizing the amunt f wrk nt dne--is essential. The best architectures, requirements and designs emerge frm self-rganising teams. At regular intervals, the team reflects n hw t becme mre effective, then tunes and adjusts its behaviur accrdingly 12 Taken frm The Agile Manifest (http://agilemanifest.rg/) 13 Surce: Wikipedia - http://en.wikipedia.rg/wiki/agile_sftware_develpment 14 Surce: Wikipedia - http://en.wikipedia.rg/wiki/agile_sftware_develpment 2007 Data Management & Warehusing Page 15
White Paper - Data Warehuse Gvernance Cwby Cding 15 Cwby cding is a frm f sftware develpment methd withut an actual defined methd team members d whatever they feel is right. Typical cwby cding will invlve n initial definitin f the purpse r scpe f the prject, n frmal descriptin f the prject and will ften invlve nly ne develper 16. The develper will ften be wrking frm his wn idea f what the sftware shuld d. It is ften characterised by a lack f any dcumentatin fr either the requirements f the prject r the design f the sftware verall. Within a data warehuse prgramme it is strngly advised never t allw the cwby methdlgy t appear, hwever any f the ther three can be used. There are tw deciding factrs as t the apprach that will be used. The first is the apprach that is already being used within the rganisatin and this may dminate ver all ther factrs. The secnd is the pint in the lifecycle f the warehuse. In the early part f the prgramme prjects using agile methdlgies ffer fast develpment and quick delivery fr new applicatins built by expert teams that engage the users. Twards the end f the lifecycle a mre waterfall like apprach t maintenance and de-cmmissining peratins run by systems management teams is likely t be mre successful fr individual prjects. This highlights the need fr the gvernance itself t be adaptive t the changing envirnment. All frmal methdlgies (lightweight and heavyweight) can still lead t failure as the methdlgy team members attempt t perate within the scial/plitical envirnments f the rganisatin. The prbability f such a failure is related t the degree f prcesses inhibiting the user(s) frm deviating frm the rganisatin standard n ne hand and the cst in terms f lst efficiency and lst creativity f implementing such prcesses n the ther. Success is therefre reliant n the management chsing apprpriate prcesses that find the crrect balance between these tw cmpeting aspects. 15 Cwby cding definitin: Wikipedia: http://en.wikipedia.rg/wiki/cwby_cding 16 Cwby cding can ccur even within prjects that are using mre apprpriate develpment methds. Warning signals that cwby cding may be ccurring include: Secrecy abut what a develper is wrking n. The inability t describe the functinality f current develpment. The tendency t d lts f quick fixes. Wrking hard and nt wrking clever, wrking withut priritisatin. 2007 Data Management & Warehusing Page 16
White Paper - Data Warehuse Gvernance Best practices The fllwing items are sme f the key best practices that shuld be implemented: Mmentum Get the prject ging fast and then keep it mving by keeping the scpe and deadlines fr delivery very tight. This avids analysis paralysis and ensures that there is space t take crrective actins if necessary. Mnitr and Measure 17 Large lng-term prjects are difficult but if the management team cannt see what is happening then they d nt knw when things are ging wrng. Ask questins abut whether teams deliver n time, t budget and scpe. Track the number f issues raised in develpment, test and prductin, mnitr the service level agreements, design metrics fr the quality f the cde and the user satisfactin, etc. Much f this is built in t appraches such as agile. Priritise Everyne will want their cmpnent nw, but it cannt all be dne at nce. Have a clear, transparent prcess fr deciding the priries and then stick t them. One useful technique is t say that (prvided the prject uses small scpes) nce started a phase cannt be stpped but that befre the next phase is started all pririties can be reassessed. This frces lw value pririties t the back but rapidly brings higher pririties t the fre. The steering cmmittee shuld be respnsible fr these pririties and the prject methdlgy shuld supprt the priritisatin prcess. Light tuch This dcument has cvered many aspects f gvernance hwever the prgramme shuld implement the minimum necessary because therwise the prject will becme abut gvernance and nt delivery. Review the prcesses regularly t add prcess where required and d nt be afraid t remve un-necessary prcess. Cmmunicatin The data warehuse is a lng-term prject fr which the perceptin will shift ver time frm all parts f the business. Clear cmmunicatins will help peple understand what is available fr them, what the pririties are ging frward and hw t access the infrmatin. Publish these widely t encurage invlvement and interactin with the system. 17 A useful bk n this subject is Cntrlling Sftware Prjects: Management, Measurement and Estimates by Tm DeMarc 2007 Data Management & Warehusing Page 17
White Paper - Data Warehuse Gvernance Standards Develp and enfrce clear standards fr all aspects f the warehuse such as naming cnventins fr cde and dcuments, data wnership, data cleaning, access t data, service levels, perfrmance etc. 18 Track prcesses Ensure that there are clear frmal prcesses fr handling change requests, risks and issues. These are the aspects f the prject that cause divergence frm the baseline and therefre it is critical t knw what is happening, when and why. It allws scarce resurce t be fcused n the crrect actins. Use key design decisin templates t ensure that design tpics are nt cnstantly revisited. Ensure that all change related prcesses have pwerful change management rutings t back them up. Understand cst and value Define methds fr determining the cst and value f any actin, sme apparently lw cst (ften called quick-win r tactical) slutins deliver little value and their true cst is disprprtinately higher as they have t be backed ut after a shrt life. D nt frget t include end f life cst in the calculatins and make budget hlders respnsible fr demnstrating the benefit that has been derived because f the effrt. Cntinuus Learning and Imprvement Carry ut stage pint audits and pst-mrtems n all aspects f the system, use the lessns learnt frm these t imprve the prcesses t ensure that the next develpment will nt make the same mistakes. Pilt & Prttype Pilt and prttype high risk r cmplex aspects f the develpment using methdlgies like Agile but ensure that the pilts and prttypes d nt g int prductin until the nrmal quality threshlds have been met. Authrity t act Ensure that the steering cmmittee has the authrity t act crss functinally using sufficient infrmatin t make gd decisins. A steering cmmittee that wants a change t a surce system that has a critical effect n the data quality is ineffective if they are tld that it will be put in a queue and shuld be delivered in tw years time. In the mean time is it acceptable that the data warehuse will just have t make d with pr data? Cnversely a lack f sufficient infrmatin might be making the same demand f a system that is abut t be de-cmmissined r fr data fr which the business user cannt articulate a use. 18 See the Data Management & Warehusing Dcumentatin Radmap at http://www.datamgmt.cm 2007 Data Management & Warehusing Page 18
White Paper - Data Warehuse Gvernance Plan fr the lng term Ensure that the business and users understand that whilst aspects f the data warehuse will be available quickly the system is a lng-term cmmitment. It will need funding and maintenance acrss its entire life span. During that lifetime the system will als grw in size and cmplexity, increasing the financial burden. Gvernance and Success Gvernance is the cntrl aspect f management and shuld prmte a situatin where senir management are in cntrl f the situatin much like the maestr cnductr f a fine rchestra. They need t understand what is ging n and where t g fr the next actin and hw t respnd t a changing situatin. Smetimes the reality is that managers are wrking in a suppressed panic, nt believing what peple are telling them r what they themselves are cmmunicating t their management. When managers knw that peple have t wrk utside the prcess t get things dne and nt knwing what is actually ging n then gvernance has failed. Gvernance is therefre nt the dcumentatin, the prcesses r the frmality, but is abut develping a culture where understanding, discipline and skill are regarded as virtues in teams that have leaders with strng technical skills, initiative, cmmunicatins skills and persnal authrity. Organisatins that buy the bk, be it frm a bkshp r as a frmal methdlgy frm a vendr and fllw it rigidly are guaranteed t achieve the lwest cmmn denminatr slutin, ne that checks all the bxes but underwhelms the management and disappints the users. Thse rganisatins that use gvernance and methdlgies as enablers and allw systems t fulfil their ptential by meeting the cnstantly changing requirements f the users succeed. Creating effective gvernance fr an rganisatin requires imaginatin: What we need is imaginatin, but imaginatin in a terrible strait-jacket. We have t find a new view f the wrld that has t agree with everything that is knwn, but disagree in its predictins smewhere... And in that disagreement it must agree with nature. If yu can find any ther view f the wrld which agrees ver the entire range where things have already been bserved, but disagrees smewhere else, yu have made a great discvery....a new idea is extremely difficult t think f. It takes a fantastic imaginatin. 19 19 Richard Feynman, The Character f Physical Law, 1965, Chapter 7, "Seeking New Laws. 2007 Data Management & Warehusing Page 19
White Paper - Data Warehuse Gvernance Summary An rganisatin that is embarking n a data warehusing prject is undertaking a lng-term develpment and maintenance prgramme f a cmputer system. This system will cst a significant amunt f mney and shuld be well cntrlled. Gvernance defines the mdel that the rganisatin will use t ensure ptimal use and re-use f the data warehuse. It will als enfrce crprate plicies (e.g. business design, technical design and applicatin security) that ultimately derive value fr mney. This paper has identified five surces f change t the system: Demand External Change Maintenance Risks Issues It has als lked at the aspects f the system that they will affect: Requirements Catalgue Technical Infrastructure Cnfiguratin Management Test Management Data Stewardship Service Level Agreement Supprt Mdel Training Plans Schedules Mnitring Using these it has been able t highlight standards and best practices that shuld be emplyed by the rganisatin t deliver effective gvernance. The paper has als highlighted the fact that because gvernance is abut fitting int the existing rganisatin. It als highlights the fact that the system grws significantly ver time. Cnsequently there is n single right slutin fr gvernance but instead there are ways f wrking and adapting ver time that will ensure success. 2007 Data Management & Warehusing Page 20
White Paper - Data Warehuse Gvernance Appendices Appendix 1 - Prgramme r Prject? This dcument has several times differentiated between prgrammes and prjects. The table belw 20 utlines the differing characteristics f the tw: Prgrammes: Address the entire business change Fcus n strategic gals May have imprecise definitin May have uncertain timing Evlve ver a perid f time t derive ptimum benefit fr the rganisatin Require much senir management attentin, ften including strategic and plitical debate acrss rganisatinal bundaries Prduce an verall imprvement in the business that may be multi-faceted and nt fully defined at the utset f the prgramme Require a manager wh is highpwered, high-level, visinary, strategic, plitical, sales-riented and wrks with peple at the tp and acrss the rganisatin Prjects: Deliver a specific change cmpnent Fcus n tactical delivery Have a precise bjective Are defined with a specific timeline and budget Try t avid change t the defined scpe in rder t ensure delivery Require management cmmunicatin primarily at an peratinal level cncerning peratinal details Prduce specific pre-defined deliverables Require a manager wh pays attentin t detail, has gd team leadership, plans in detail, fllws a disciplined apprach and delivers the gds. 20 The epmbk by Simn Wallace (http://www.epmbk.cm/) 2007 Data Management & Warehusing Page 21
White Paper - Data Warehuse Gvernance Appendix 2 The "Declaratin f Interdependence" Thrughut this dcument reference has been made t the agile develpment methdlgy. Because f the develpment f agile, a set f principles characterised by statements We accmplish X by ding Y. was develped fr prject managers. These principles are f value utside the agile methdlgy itself and shuld be used by prject managers in any data warehuse prgramme. The Declaratin f Interdependence fr mdern (agile/adaptive) (prduct/prject) management 21 "We... increase return n investment by making cntinuus flw f value ur fcus. deliver reliable results by engaging custmers in frequent interactins and shared wnership. expect uncertainty and manage fr it thrugh iteratins, anticipatin and adaptatin. unleash creativity and innvatin by recgnizing that individuals are the ultimate surce f value and creating an envirnment where they can make a difference. bst perfrmance thrugh grup accuntability fr results and shared respnsibility fr team effectiveness. imprve effectiveness and reliability thrugh situatinally specific strategies, prcesses and practices." 21 2005 David Andersn, Sanjiv Augustine, Christpher Avery, Alistair Cckburn, Mike Chn, Dug DeCarl, Dnna Fitzgerald, Jim Highsmith, Ole Jepsen, Lwell Lindstrm, Tdd Little, Kent McDnald, Pllyanna Pixtn, Prestn Smith and Rbert Wyscki: http://alistair.cckburn.us/index.php/the_declaratin_f_interdependence_fr_mdern _management 2007 Data Management & Warehusing Page 22
White Paper - Data Warehuse Gvernance Appendix 3 - Team Values and Principles The authr f this white paper used t wrk at Sequent Cmputer Systems. The cmpany had a set f values and principles that the authr feels have always prved useful t teams building data warehuses by creating a culture in which gvernance can wrk. Our Values: Easy t D Business With The Custmer ALWAYS Cmes First D All We Can t Ensure Custmer Success - Des Nt Always Mean Saying Yes Three Keys -- Listen, Cnsider and Act G the "Extra Mile" fr the Custmer Prfitability Required fr Our Success Allws fr Grwth & Investments fr Future Everyne is Respnsible fr Prfitability Fllw the "Spend Smart and Invest Smart" Cncept Teamwrk Strng & Balanced Teams Put Grup Gals & Interests Ahead f Persnal Reward Open & Hnest Cmmunicatin Yu're Accuntable fr Yur Cmmitments Nbdy Wins Unless the Team Wins Quality Peple Exceed the Custmer Expectatins Strive fr Cntinuus Imprvement in All Yu D Knw Wh Yur Custmers Are Establish Clear Mutually Acceptable Expectatins Managed Expectatins, Cnsistently Achieved Assimilating & Integrating New Members is Critical Open, Hnest & Timely Cmmunicatin Treat Each Other With Respect Take Time t Say "Thank Yu" fr a Gd Jb Strive fr Win-Win Relatinships Our Principles: Act With Hnesty & Uncmprmising Integrity Take Respnsibility fr Our Custmers' Success Strive t be the Best Have a "Can D" Attitude Respect Each Other Exhibit Team Pride Take Calculated Risks Be Active Cmmunity Citizens Urgently D the Right Thing Make Cnsultative Decisins 2007 Data Management & Warehusing Page 23
White Paper - Data Warehuse Gvernance References The sectin belw represents sme useful resurces fr thse cnsidering building a data warehuse slutin. Web resurces Organisatin Data Management & Warehusing DM Review Data Warehuse Prject Management The Data Warehuse Institute Data Warehuse Gvernance Data warehuse gvernance Best practices at Blue Crss & Blue Shield f Nrth Carlina Data Warehuse Prject Management The Seven Pillars f Data Warehuse Gvernance The epm Bk Sftware Testing Resurces Frm Here t Agility: The Physics f Speed Agile Manifest Agile Alliance e-prgramme Data Prtectin Law Alistair Cckburn Wikipedia Website http://www.datamgmt.cm http://www.dmreview.cm/ http://www.tdwi.rg/ http://www.amazn.cm/ http://www.amazn.cm/ http://www.itweb.c.za/ http://www.epmbk.cm/ http://www.testingfaqs.rg/ http://www.stickyminds.cm/ http://agilemanifest.rg/ http://www.agilealliance.rg/ http://www.e-prgramme.cm/ http://www.dpalaw.inf/ http://alistair.cckburn.us/ http://en.wikipedia.cm Acknwledgements The authr and Data Management & Warehusing wuld like t acknwledge and thank all the clients and clleagues wh have taken the time t review and cmment n this dcument prir t publicatin. Special thanks t Rn Ballard fr the mst detailed f reviews and prf reading. Cpyright 2007 Data Management & Warehusing. All rights reserved. Reprductin nt permitted withut written authrisatin. References t ther cmpanies and their prducts use trademarks wned by the respective cmpanies and are fr reference purpses nly. 2007 Data Management & Warehusing Page 24