by Adam Michelson Director, Open Source Enterprise Architecture

Transcription

1 Delivering Service Oriented Architecture by Adam Michelson Director, Open Source Enterprise Architecture This paper focuses on pragmatic steps to deliver and maintain a successful Service Oriented Architecture (SOA). There are two sides to the SOA coin: the technologies to implement SOA and the organizational processes to make them successful. This paper does not focus on the depths of SOA technologies and standards. SOA is not just a set of integration technologies, but a philosophy, a strategy for reuse at an enterprise level. The issues surrounding enterprise reuse represent the real challenge facing SOA. How effective is reuse when it crosses organizational boundaries? From an idealistic perspective, enterprise reuse makes sense, but it often fails due to localized concerns. This paper addresses such concerns, discusses how services can be defined, and describes the necessary methodologies to make SOA successful thus avoiding the inevitable crush of services that over-step their bounds. The paper outlines specific processes for building complex enterprise architectures such as: Defining your organization s services and constructing a roadmap to deliver them Developing a SOA organizational methodology that describes the process to build services Executing on proof of concepts and pilots Deploying SOA successfully There are plenty of articles and books focusing on SOA as a set of technologies including web services, the enterprise service bus (ESB) and message queues. SOA is very synergistic with a specific set of emerging technologies based on open source software (OSS). Exploring the relationship between SOA and OSS is beyond the scope of this paper. If you are interested in how SOA and OSS work together, please visit to obtain the paper entitled Service Oriented Architecture and Open Source Software. O verview SOA generally describes a technology-agnostic architecture that is comprised of loosely-coupled modules called services that interoperate to perform desired functionality and foster reuse across an organization. A service is a set of cohesive business logic units of work that are encapsulated into a coarse-grained autonomous module. These services are then loosely coupled to form applications. This approach has many benefits, including reuse, that drive down total cost of ownership and improve functional consistency. In addition, the loose coupling of SOA allows your organization s internal and external services to be integrated easier. This paper outlines several steps to achieve a successful SOA. Figure 1 shows these steps broken down into phases and artifacts that we will explore in detail throughout the rest of the document: Figure 1 S O A Strategy The SOA process begins with the SOA Strategy phase. In this phase the SOA Blueprint is created along with the Services Dictionary and the Application Services table that work together to help define the services in the SOA and how they interoperate. The SOA Roadmap is also created in this phase to build a plan that describes the steps to implement the blueprint. Copyright Optaros, Inc Some Rights Reserved. This work is licensed under a Creative Commons Attribution 2.5 License

2 Delivering Service Oriented Architecture - 2 of 8 SOA Blueprint Defining your services is essential. A good first step to building your SOA is to define your services in a technology-agnostic SOA Blueprint. The SOA Blueprint is not concerned with technologies, but rather with the core business functions of your organization. These services then will interoperate to replace the business applications you have today. As previously stated, a service is a coarse-grained module. But what does coarse-gained mean in this context? For example, a service may be a security module to manage authentication and authorization or a service may be an inventory management module to handle the movements and status of stock. The scope of a service may also be defined relative to software modules that exist today. In these terms, a service is larger than a component, but smaller than an application as shown in figure 2: Figure 2 Services can be thought of as a new layer of abstraction, larger than components such as Enterprise Java Beans, smaller than complete business applications. Components are more abstract than objects because they do not support inheritance and the coupling that goes along with it, but components often run in too homogeneous of a run-time environment to be reused effectively across an organization or with outside partners. Services are built for this more abstract level of software interaction; they can be built to run in a heterogeneous environment. Building an application is the process of integrating the appropriate services, creating any required functionality unique to the application that the services do not provide, and adding an interface. Creating a SOA Blueprint of your organization s optimal services is a good first step, and may look like the example in Figure 3: Figure 3

3 Delivering Service Oriented Architecture - 3 of 8 Services Dictionary and Application Services Table Two small documents accompany this diagram to complete the SOA Blueprint, the Services Dictionary and the Application Services table. The Services Dictionary describes at a business level the functionality of each service. The Application Services table simply documents the one-to-many application-to-service relationships. Figure 4 is an example of a specific application and its services. In this case the sample point of sale (PoS) application was selected because it may use many services. The PoS application uses a security service for user authorization and authentication, and an ecommerce service to complete sales transactions. The ecommerce service in turn uses many other services to complete its functionality. Of course, the specific services for your organization will be different than the ones shown. Ideally, services do not share data repositories, at least not for write access. Instead, a data source is owned by a particular service that will take responsibility of writing to and updating the data source. This keeps the functionality of a service and its supporting data well encapsulated. If service A needs data that is managed by service B, then service A should request the data from service B instead of accessing the data directly. Also the data should be requested when it is actually needed, just-in-time, not batched and replicated ahead of time. If service A must access service B s data directly for performance reasons, service A should access the data in a read-only mode so there is no confusion as to which service is truly managing the data. If multiple services are allowed to update a single data source, the interactions can get very confusing and functionality such as transactionality and caching can become almost unmanageable. This is an issue SOA attempts to solve, not compound. For example, in the diagram below, the content management system (CMS) and search services both have access to enterprise content data, but the search service is read-only because search needs frequent access to the content data but will never update the data. Point of Sale ecommerce Audit & Logging Payment Processing Inventory Mgmt Content Management System Workflow Business Rules Financial Inventory Enterprise Content Index Customer Audit & Logging Figure 4 As you can see, the service interactions can get complex, so the major interactions are documented. Drawing a diagram of the interactions is good for illustration purposes, but too difficult to maintain in a dynamic enterprise. Therefore, we will use tables that are easier to maintain to describe the interactions. The Application Services table below (Table 1) is used to describe these relationships. Application, Service or Source Relationship Application, Service or Source PoS Application Uses ecommerce Service PoS Application Uses Service Service Owns ecommerce Service Uses Payment Processing Payment Processing Service Owns Financial ecommerce Uses Inventory Mgmt Inventory Mgmt Owns Inventory ecommerce Uses CMS Content Management System Owns Enterprise Content ecommerce Uses

4 Delivering Service Oriented Architecture - 4 of 8 Application, Service or Source Relationship Application, Service or Source Reads Enterprise Content Owns Index ecommerce Uses Workflow ecommerce Uses Business Rules ecommerce Owns Customer PoS, ecommerce,, Payment Processing, Inventory Management, CMS,, Workflow, Business Rules, Financial, Inventory, Enterprise Content, Customer, Uses Audit and Logging Audit and Logging Owns Audit and Logging Table 1 The actual services are described in the Services Dictionary in Table 2. Service Description Owner ecommerce Service Service A service to manage user purchases including payment processing and shopping cart. Manages enterprise authorization and authentication via corporate LDAP repository. Sales Payment Processing Manages payment status and various forms of user payments. Finance Inventory Mgmt Content Management System Workflow Business Rules Audit and Logging Keeps track of inventory and manages procurement and user delivery. A corporate repository of content and images. Allows for distributed administration. A corporate search service. This service includes an indexing and search capability for individual applications. A flexible meta-data driven workflow engine that individual groups may use. A flexible meta-data driven Business Rules engine that individual groups may use. A corporate auditing and logging tool individual groups may use. Table 2 Manufacturing Marketing This logical architectural blueprint need not be completely implemented, as its function is to serve as a map. It is a guide to help determine your organization s actual services. This is the top-down design that helps to form the SOA architecture as it is being implemented service-by-service and application-by-application. This blueprint is about your business and should be technology agnostic. This is important because technologies change constantly and the blueprint needs to be valid as technologies come and go. Your organization s SOA is continuously evolving towards this ideal blueprint, but changes in technology will cause you to never complete the blueprint in a single SOA technology. SOA can be achieved by bridging older technologies to newer ones as technologies evolve. This is the idea behind a methodology called refactoring ( and is a good way to implement a SOA Blueprint using an evolutionary model, where change is constant. SOA Roadmap Once the services are defined in the SOA Blueprint, a rough schedule should be put together to specify timeframes for building the services and applications on the SOA. The SOA Roadmap (Figure 5) shows a high-level schedule of the next few SOA activities as well as the initial SOA applications and services to be built. Only the first few applications and services are shown because of the difficulty in planning with certainty far enough into the future. Instead, a progressive elaboration technique is used where the near future is clear and the more distant future only an estimate that will most likely change. The SOA Roadmap reflects some predefined priorities to determine the order of implementation for a given application and its services. Also, the Application Services table has dependencies between applications and services that should be taken into account when determining a rollout plan. The SOA Roadmap may take the form of a Gannt chart, or any diagram showing the high-level plan.

5 Delivering Service Oriented Architecture - 5 of 8 Proof of Concept (2 Months) Pilot (3-6 Months) Departmental Web Sites (3 months) User Provisioning (6 months) Rolling Rollout with SOA Infrastructure & Services Support Sales Portal (6 months) Basic CMS Collaboration Workflow (4 months) Config. Mgmt Systems Admin. (3 month) CMS (1 month) (Already Built) Collaboration (Already Built) CRM (5 months) Auditing & Logging ecommerce (4 months) CMS (3 months) PoS Part I (6 months) Auditing & Logging (Already Built) (Already Built) Business Rules (3 months) Payment Processing Workflow (1 month) PoS Part II (6 months) Complete (1 month) Inventory Mgmt (4 months) Figure 5 S O A M e t h o d o l o g y Once your organization s services are identified on a SOA Blueprint and your SOA Roadmap is complete, a good next step is to define a SOA methodology. A SOA methodology is your organization s process for building and rolling out your SOA, such that key process issues are resolved before any architecture is implemented. SOA methodology aims to address the issues surrounding reuse at an enterprise level proactively, since internal politics and norms can thwart even the best laid SOA plans. The SOA methodology should include the primary considerations to make SOA successful, including: Rules of engagement Quality assurance Compliance objectives Evolutionary approach Rules of Engagement If two applications should share common functionality for optimal cohesion and encapsulation, the functionality should become a shared service. You may be required to address a number of corporate complexities to foster shared services, including organizational boundaries such as P&L, political, and geographic. If left unaddressed, these boundaries will undermine the SOA strategy. Any issues should be addressed proactively with the various constituencies. If agreed-upon rules of engagement can be achieved, a common service can be built. Otherwise, your organization may be forced to create similar services, each belonging to the respective group. It is easier to break a service apart then merge two together, so try to work out the group s differences and share services whenever possible. If necessary, the service can be divided in the future. Defining the ownership of your applications should not be a significant challenge because your organization has already defined who owns your existing applications, and SOA does not change these relationships. Defining the ownership of your services may be more ambiguous, and can become a major issue if left unaddressed. The rules of engagement must specify roles and responsibilities for each service. Your SOA Blueprint will likely contain services that cross existing profit and cost center domains, which we will refer to as P&L groups. Typically, each group has funded its own applications in the past, but where reusable services may cross P&L group boundaries, friction can arise and the service can become a scapegoat. This should be addressed proactively, with the various groups agreeing to the common services and the rules of engagement that surround them. These rules may contain factors such as: How a service is funded Which group(s) participates in the development and maintenance of the service Which individuals participate in functional scope and release timeframe decisions What are the agreed-upon service levels such as maintenance windows and support escalation

6 Delivering Service Oriented Architecture - 6 of 8 A RACI-type matrix (Responsible = developers, Accountable = business sponsors, Consulted = business or QA users, Keep Informed = service consumers) may be used to define users, roles, and their relationships to services to help manage the inevitable ownership issues that arise with enterprise reuse. This matrix should be defined and agreed upon before the relevant services are constructed so ownership issues can be addressed, or at least acknowledged, proactively. Quality Assurance Quality assurance (QA) is another important methodology issue to address when dealing with a service that has a high degree of reuse. At times it is hard to know all of the applications that use a service, making QA difficult. Automated QA unit test cases covering all functionality that a service can perform is a good goal, but difficult to achieve. To deal with these issues, it is a good idea to build a process to discover all applications using the service, such as logs of all services and applications that have used the service, into the SOA architecture. Then, all relevant teams should be alerted of impending functionality changes to a service, so the groups can individually test and report issues to the service s development group proactively. The relevant teams who have been identified as a party to be informed on the RACI matrix can also be notified of impending changes. Compliance Objectives SOA can help organizations conform to compliance requirements by managing auditing and security access rules. A SOA s loosely-coupled properties can help with compliance because it represents a great central place to log and audit messages flowing through the SOA infrastructure. SOA also acts like a message hub or traffic cop, by assuring messages have the appropriate access rights to each service therefore providing a central location to administer and comply with security rules and access rights. Evolutionary Approach An evolutionary development strategy is the most appropriate for a SOA implementation. A revolutionary approach is likely to be too heavy, involving too much risk. A more phased approach limits risk, allows you to learn as you go, and allows the SOA infrastructure to scale and grow at a reasonable pace. Typically, early SOA infrastructure will be deployed on the fringes of an organization, complementing the existing integration infrastructure. As the architecture proves its value, the SOA will expand and penetrate into the core integration infrastructure. Due to the changing nature of technology, a revolutionary SOA model is almost impossible. As stated previously, technologies will change over time. It is likely that the first two services you build will use different technologies than the last two. This should be expected. An evolutionary model using refactoring techniques addresses changing technologies by bridging older technologies to newer ones. The software architecture is never considered complete due to technology innovation. The SOA Blueprint should stay relatively static when defined properly. Allow the technologies to change and make evolutionary implementation a natural part of the SOA strategy. These methodologies may seem arduous, but can pay back generously when it comes to enterprise-level reuse, total cost of ownership, and continuity and consistency of business logic. Once you have addressed the key areas around the SOA strategy and methodology for your organization, you are ready for implementation. Proof of Concept The first step towards implementation is often a proof of concept (PoC) to prove that a given SOA infrastructure will meet your needs. Choose an application that is architecturally significant so it challenges the SOA infrastructure to assure the PoC will accurately represent the challenges the SOA will face in full-scale deployment. The intent is not to rebuild the application s services at this point; it is only an example to gather SOA infrastructure requirements. Once the application is chosen, identify the services with which it must integrate. These requirements are used in the PoC of your SOA architecture so you can evaluate the myriad of SOA technologies independently of a production implementation. Then design and build a PoC that shows off the properties of the requirements you gathered. Characteristics of an application that lead to a productive PoC involve integrating with services that require features such as orchestration, transformation, a measure of transactionality, and a defined message type such as point-to-point or publish-subscribe. The following list describes some of these service features and design considerations you may want to include in your PoC.

7 Delivering Service Oriented Architecture - 7 of 8 Transport A SOA often includes support for message queuing to provide guaranteed message delivery, publishsubscribe and request-response messaging. Orchestration and routing This includes the directing of messages through the SOA to perform a business function. The complexity of orchestration will vary greatly depending on your organization s requirements. Message transformation Two levels of transformation will likely be required, protocol-level and application-level. The SOA will need to translate from one protocol to another across heterogeneous systems through connectors (e.g. JMS to SOAP transformation). The SOA will also need the ability to marshal data into various formats to allow applications and services to interoperate. Transactionality The ability to isolate and manage stateful transactions. Service lifecycle management How a service is instantiated, destroyed, load balanced, and failed-over so the SOA can scale appropriately with the appropriate service level agreement. Discovery A repository so consumers can find available services and learn how to use them. Administration A mechanism to monitor and control all of the SOA infrastructure components. Auditing The ability for the SOA to audit messages which is especially helpful with compliance regulations. Authorization and authentication rules. Creation of the initial API Domain Model This API model will grow into your organization s application interface protocol grammar used to call SOA services. This is a standard API representing your organization s domain model. XML is an obvious choice for this model. A centralized API Domain Model will help maintain naming consistency across services with similar properties. For example, two services may return the same property such as a customer identifier. The API Domain Model will define a property for this identifier, say as the customerid. This way the service developers both name the property customerid, and do not each create their own unique name for this property which would confuse the service consumer and make the SOA unnecessarily complex. Some of the technologies used in a SOA infrastructure stack may include: Message queue Provides a transport layer for guaranteed messaging. Enterprise service bus (ESB) ESBs are an emerging software product. It is a relatively new term that represents the next evolution of middleware. It can contain many types of functionality, but typically will sit atop a message queue, provide orchestration and can contain connectors to various heterogeneous applications and environments. Application server Acts as a container for services associated with the SOA. Services do not have to reside within an application server, but many newer services do. These services may perform complex rules. It is up to the SOA architect to determine if there are services that should be contained within the SOA solution, such as logging, administration, or complex orchestration or transactionality. These technologies combine to provide SOA functionality as show in figure 6: Figure 6

8 Delivering Service Oriented Architecture - 8 of 8 For a more in-depth review of SOA infrastructure components and how they interrelate to form a stack, please refer to the Optaros paper entitled Service Oriented Architecture and Open Source Software. Pilot After the PoC is complete and the baseline SOA infrastructure is defined, a pilot program is a good next step to achieve SOA. The pilot may implement two or more actual services that are built and consumed by one or more applications. The PoC infrastructure is reengineered as necessary to create the first production-ready instance of the SOA infrastructure. The SOA foundation is created and an initial application is converted to the SOA with the appropriate services. These should be architecturally significant to exercise the SOA, but the application should also be well controlled and not mission critical for this initial release. The new SOA applications and its services then go into production and are evaluated. The pilot is complete only when this suite of applications and services function as required with the appropriate quality of service. The final stage of the pilot is to update the SOA Roadmap to plan the what and when around the SOA rollout and a rollout kit that describes how groups within the organization can leverage and build on the SOA. The rollout kit should include documentation about the SOA infrastructure and how to build and use SOA services. This kit contains the materials for the SOA to be socialized to the rest of the organization and communicates how groups can participate. Responses from the socialization will refine the roadmap as groups within the organization opt in to the SOA strategy. Evolutionary Rollout The evolutionary rollout is the final step. We have arrived! At this point your organization can begin to roll out the SOA solution. An evolutionary style means SOA is rolled out to production service-by-service, application-by-application. This pace will allow the SOA solution to evolve as the requirements and technology innovations warrant. Using an evolutionary rollout, your organization implements the SOA Blueprint according to the SOA Roadmap, using the SOA methodology. This may involve adding a new interface to existing back-end applications, completely rebuilding services, or anything in between, depending on how well your existing application architecture maps on to the SOA Blueprint. Summary and Next Steps SOA provides a foundation on which your IT portfolio can evolve over time to accommodate your business s IT demands. As we have seen, SOA is a new way to design and implement enterprise architecture. It allows for a high-degree of reuse and loosecoupling of services across your organization which has many benefits including cost, maintainability, functional consistency and ease of integration. SOA technologies are rapidly maturing. We have tried to demonstrate that SOA is not only a set of viable technologies but also represents a refined way to implement enterprise architecture. There are several challenges to SOA s vision of enterprise reuse that can push it to the back of the collective software engineering consciousness. If these are overcome, SOA will change the way architectures are built. The definition of your organization s services is essential to realize the potential of SOA as is the process to build and roll out the services. SOA is usually viewed as an evolutionary process, and embracing an iterative implementation strategy is a good way to cope with the dynamics of enterprise architecture. Once the design, roadmap, and methodology concerns are understood, a good next step is to delve into the specific SOA infrastructure technologies available. We have found that the products and components required to deliver SOA functionality exist today. There are also many compelling open source solutions available for SOA implementation. Given the maturing state of SOA-related technologies, we believe that now is a great time to evaluate a SOA solution for your organization. ABOUT OPTAROS Optaros is a consulting and systems integration firm that helps enterprises solve IT business problems by providing services and solutions that maximize the benefits of open source software. Bringing together experts in creating enterprise IT solutions and experts in the power of open source, Optaros plans and builds business systems that give you better value today and increased control in the future. CREATIVE COMMONS LICENSE This work is licensed under a Creative Commons Attribution 2.5 License CONTACT Brian Otis VP, Sales and Partnerships botis@optaros.com phone: (617) x110