1. Introduction 1
2. Lessons Learned 1
3. Web Services Standards Gap 3
4. Tools Gap 3
5. Design Constraint Implications 3
6. Best Practices & Recommendations 4
6.1 Using open source tools ANT and DPTK 4
6.1.1 Ant 4
6.1.2 Design Pattern Toolkit (DPTK) 4
6.2 Deployable Portal Artifacts 4
6.3 The Build and Deploy Sequences 7
6.4 Interaction between the Architecture Layers 12
7. Conclusion 13 5/24/2005 10:15 AM
1. Introduction
One of the goals of the IE Pilot project was to demonstrate building and deploying the various development artifacts in a repeatable way. This was accomplished using a process that was developed to ensure quality by automating as many of the build and deploy processes as possible. The high level procedure can be described as follows:
• Developers check-in various artifacts into a version control system (java code, JSPs, deployment descriptors, etc.)
• Developers also check-in build and deployment scripts for each artifact. (The creation of the build and deploy scripts is automated with a pattern-based toolkit.)
• For build and deployment, the system operator of a target environment (such as an integration, QA, or production system) will then request a specific version of artifacts from the version control system, and make a local copy.
• The system operator will then update any properties (in the build properties file) and invoke the build and deployment scripts (which were included with the code).
• The system operator then checks-in to version control the log files which contain the build and deployment execution results.
This process ensures that build and deployment can be repeated easily on different platforms, and there is an audit trail. If a problem occurs, developers can look at the build log and assist with problem determination.
2. Lessons Learned
During the IE Pilot, we have acquired deep knowledge with the tools and their capability, as well as the run into road-blocks of which we have overcome. These are very lessons that should be captured and shared with the development, and are listed as follows:
• Developers develop using specific server names or IP addresses, but when you deploy you must specify a different target server. These server names should be set as properties (e.g. portlet init parameters).
• Model Driven Architecture (MDA) works well for build & deploy automation. Using MDA we were able to quickly automate the process of creating build and deploy scripts (batch files, UNIX script files, ANT files, XMLAccess command files, etc.). We leveraged the features provided by Eclipse and the Design Pattern Toolkit (DPTK), and built reusable patterns for generating the following artifacts:
o Portlet scripts to: add/delete/update/export/activate/deactivate portlets
o Portal Page scripts to: add/delete/update/export pages
o Portal Theme scripts to: package and un-package the theme files
• Use Interaction Diagrams to model build & deploy processes. Interaction diagrams provide a way to capture the sometimes complex interactions between people and systems. Also, this kind of diagram helps to identify opportunities for automation (new patterns).
• Use Portal’s “Custom unique names” feature for portlets and pages. The Portal product includes an administrative portlet for assigning custom names to system elements, such as pages and portlets, which are easier to remember than the system-generated IDs. When developing a script to export a page definition, for example, you can use the custom unique name you assigned to the page, and this name can be the same across environments.
• Be aware of user Distinguished Names (DNs) in exported files. Some deployment files (such as page definitions) contain LDAP DNs of users. The ‘owner’ of the page, for example, is part of the page definition when a page is exported to an xml file. These may be different between environments. An extra step may be needed to switch out the DNs in an exported file, before importing into a different system, if the new system is pointing to an LDAP that contains a different set of users.
• The XMLAccess examples provided with Portal are useful for developing scripts. We used several of the example XMLAccess scripts in the directory:
• Create handy diagrams We found it very useful to create diagrams when faced with complex deployment relationships, for example, the relationships described in a portlet.xml deployment descriptor file are important to understand when creating deployment scripts, so we created the following chart, and color-coded parts according to whether they were needed during deployment, export, or delete operations: UIDPortletApp NameWARPortal AppDefConcretePortletPortlet App(web-app)PortletConcretePortletApp*hrefid*11WAR StructureDeployDeleteExportUIDName
• It’s important to have several Portal installations to perform various kinds of testing and development:
• Developers often need a full Portal installation (not just the Portal Toolkit that runs inside of Eclipse). Due to differences in the Toolkit and full-Portal environments, such as Java JVM version differences, most developers should install a full copy of Portal on their own workstations, to use in conjunction with the Portal runtime environment that runs inside Studio.
• It’s useful to have an environment for configuring Global Security in the various Application Servers, and connect to the LDAP server to test schema and filter settings, as well as user and group access.
• Another environment is required for Load and Stress Testing (LaST). It’s important to have a dedicated LDAP server and database server in this environment that mimic the production environment as much as possible.
3. Web Services Standards Gap
4. Tools Gap
The WebSphere Portal Server administration can be accomplished in two ways. First is through the graphical admin console, and the second is using WPSConfig and XMLAccess.
The graphical admin console is meant to be used by the portal administrator and he/she can use the graphical interface to interact with the portal components such as page, theme, and portlets. With the graphical admin console, you can also manage the portal security.
XMLAccess is the XML configuration interface that allows you to import, export, and update (management) of portal components. XMLAccess can be use to perform any of the operations on the graphical admin console.
XMLAccess complements the graphical admin console of portal and WAS. The operation on the graphical admin console is prone to errors. With the XMLAccess, the fully tested scripts can be easily repeated over and over again. However, XMLAccess has its limitation. The configuration as well as the command xml files for XMLAccess normally contains ‘static’ information. That means the administrator has to still perform manual changes to the configuration/command files in their management tasks, so that also is prone to error.
The issue of reliability and duplicable can be resolved by the Design Pattern Toolkit (DPTK). See section 6.1.2 for more information on DPTK.
5. Design Constraint Implications
There were several design constraints that we limit us in design the build and deployment process for the IE Pilot. Mainly they are:
Can’t automate the code generation of BPEL processes – Currently there is a known issue with the WSADIE’s code generation engine. The issue is that when trying to invoke the codegen engine, the activity will terminate with an exception – it is an internal but with the system. We’ve captured it and opened a defect with our tools vendor (IBM). Until it is addressed in future fix-packs, the build engineer has to invoke the codegen from within WSADIE and has the option of compiling and packing the artifacts inside the Studio or run the conventional Ant build script from the command line (on the generated classes).
Because of the inability to fully automate some aspects of the build & deploy process, only the portal deployment/management is fully implemented using Ant and DPTK.
6. Best Practices & Recommendations
6.1 Using open source tools ANT and DPTK
We recommend using Ant and DPTK together as the new build and deployment paradigm.
6.1.1 Ant
Ant is an open source build tool that is very popular in the development community. Ant is popular because it is extended using Java classes. Instead of writing shell commands, the configuration files are XML-based, calling out a target tree where various tasks get executed. Each task is run by an object that implements a particular Task interface.
The latest version of Ant, version 1.6.2 has been exported to support the checking/checkout of Clear Case that enables you to perform all activity of build with just one call.
6.1.2 Design Pattern Toolkit (DPTK)
When performing build and deployment, our goal was to create a repeatable, automated process. Since there are several artifacts that require step-by-step procedures, we desired a way to automatically create the build and deployment scripts for each artifact, given just the minimum information required. We used the Design Pattern Toolkit, an Eclipse plug-in, in order to achieve these goals.
• The DPTK is based on MVC pattern. The model is the .appdef that the controller is driving off from. The controller will iterate through the XML tags defined in the .appdef and perform the appropriate view. Together, they form a particular pattern.
A set of ‘patterns’ were developed, which follow Model Driven Architecture (MDA) principles, and allow us to generate build and deployment scripts for each artifact.
6.2 Deployable Portal Artifacts
There are several Portal components that require a build process, a deployment process, or both. The following table describes each artifact:
Artifact Name
Description
Packaging
Build Needed?
Deployment Process
Portlet
The java code, JSPs, image files (gifs), and deployment descriptors (web.xml, portlet.xml) needed to render a Portlet
Web Application Archive (WAR)
Yes (java compile, WAR packaging)
XMLAccess (a portal-specific xml-based deployment program)
Page
A Portal Page definition, which contains the layout design of portlets on the page, as well as permissions (ACLs) for the page, and a Distinguished Name of the ‘owner’ of the page, and a ‘theme’ that will be used for the page.
An XML file.
No, page definitions are built with the Portal’s page creation tools (using a web based GUI) and exported as a single XML file.
XMLAccess
Theme
A Portal artifact that provides the ‘outer shell’ of the page display. This includes navigational elements (tabs for pages), the branding, and really anything that isn’t provided by portlets. The top two page levels in the portal page hierarchy can be associated with a theme. All sub-pages inherit the theme of the parent.
A directory tree, containing image files and JSP files. (WAS_ROOT/installedApps/ NODE/wps.ear/wps.war/ themes/THEME_NAME)
Yes. The themes are a part of the Portal’s own EAR file. Therefore, if you want to add new themes, you must repackage and redeploy the wps.ear file.
EarExpander
Skin
A portal artifact that provides the shell around a portlet. Each portlet on a page can have a different skin. Skins control the look of the buttons, the title bar, and the border around the portlet (if any).
A directory tree, containing image files and JSP files. (WAS_ROOT/installedApps/ NODE/wps.ear/wps.war/ skins/SKIN_NAME).
Yes. The skins are a part of the Portal’s own EAR file. (Requires repackage and redeploy of wps.ear.)
EarExpander
Screens
Standalone pages that don’t contain portlets. The Login page, the Self Care pages, and the Error page are examples.
One or more JSP files. (WAS_ROOT/installedApps/ NODE/wps.ear/wps.war/ screens/html/ SCREEN_NAME.jsp).
Yes. The screens are a part of the Portal’s own EAR file. (Requires repackage and redeploy of wps.ear.)
EarExpander
Portlet configuration
There are several roles related to portlet usage and management that can be assigned to users and/or groups. These roles are assigned to portlet applications (not instances.) Typically these ACLs are assigned
Can be exported to an XML file. (The runtime ACLs are stored in the Portal’s admin repository, eg. Oracle).
No.
XMLAccess
using the Portal Admin GUI, and then exported to an xml file.
In addition, the deployment order of these artifacts is important. There are dependencies among them, so it’s important to deploy them in the following orders:
Artifact
Dependencies
Portlet
none
Page
Portlets, theme, owner, other users based on assigned ACLs
Theme
none
Skin
none
Screen
none
Portlet Config
Owner, other users based on assigned ACLs
For the IE Pilot, we have identified the followings as the need to be generated and tracked.:
Script Name
Description
XMLAccess command file
The Portal deployment tool, XMLAccess, requires a command file in order to perform an operation. This command file describes the type of action to perform (import, export, etc.) and the specific page, portlet, etc. to act upon.
ANT file
We found it convenient to use ANT (a popular scripting tool available on apache) to kick off the XMLAccess task. Portal provides a tools.jar file that contains an XMLAccess ANT task, making it easy to call XMLAccess from within ANT. Other operations, such as copying files, echoing status, and version control system calls are often easier to perform in ANT. ANT scripts are portable across Windows and UNIX platforms.
ANT .properties files
The .properties file stores information about paths, user names, passwords, and URLs. Since these settings are often different across environments (testing, QA, Load And Stress Test, production), having a properties file for each makes script easier to maintain.
.BAT and .SH scripts
These are used to kick off the ANT tasks. The ‘setupCmdLine” script from WebSphere App Server is first called, to ensure that the proper environment is set up for Java and ANT.
The following diagram depicts the relationship between the programs, properties file, and the output file:
environment
DOS Batch file
ANT script
Unix Shell file
results.out
XMLAccess
xmlAccess.command.x
Portal Admin Servlet
Portal Admin Repository DB
6.3 The Build and Deploy Sequences
In order to visualize the interaction between the activities involved in building and deploying Portal artifacts, it’s helpful to model the steps using an interaction diagram.
The deployment operations performed by a Deployment Engineer would follow this series of steps to gather files, publish them to a target server, and execute the scripts that will perform the deployment:
: Deploy Engineer : Build Area : ClearCase : Target Area Areaget: artifactsartifacts (EAR, WAR, JAR)get: scriptsftp: artifacts & scriptsexecute scriptsscripts
The following diagram shows how two actors, a Developer and a Build Engineer, cooperate to prepare for deploying a portlet:
: Portlet Developer : WSAD : ClearCase : Build Engineer : Build Area Areaportlet codeweb.xmlportlet.xmlcheck in: portlet codecheck out: portlet codeweb.xmlJSP, Java, images, and descript...buildapply patternportlet scriptscheck in: portlet scripts.warPortlets now ready to be deployed to the design server. WAR file and scripts will be FTP and executed.
Figure 1 - Portlet Check-in and Preparation for Build
Theme development and deployment is depicted below:
: Theme Developer : WSAD : Design Server : ClearCase : Build Area : Build Engineer Engineercreate projectimport base themecustomize themeexport "project to filesystem"add theme in Admin Portal (name, location)use theme in a test pagetest themegenerate scriptscheck in: themes and scriptscheck out: themes and scriptsthemes and scriptsftp: theme & scriptsre-package wps.warwps.warThe re-packaged wps.war will contain the new theme(s) under the appropriate fol...
Figure 2 Theme Development and Deployment
Page Development is performed by a Site Designer, who works with a Business Expert to place and organize Portlets on the page. Once the layout is complete, the page definition is checked-in to version control:
: Site Designer : Use Cases Guy : Design Server : WSAD : ClearCase ClearCaseget: layoutpage layout, ACLcreate page (theme, name in different language)place portlets on pageadjust page settingsassign ACLassign unique name for pagecreate projectFile -> New -> Other -> DPTK -> SampleAppdefset name, uniq name, and version in Appdefapply patterngenerate scriptscheck in: scriptsscriptsftp: page export scriptexecute scriptpage.xmlcheck in: page.xmlPage now ready to be deployed on target server. To deploy, need to ftp the scripts, as well as the page.xml to the target and execute the script using the generated .bat or sh script.
Figure 3 - Page Development and Check-In
Sometimes artifacts that have been deployed previously just need to be updated. For example, a page might have a portlet added to it by a developer in an integration environment, and the updated page xml file needs to be loaded into a production environment. The following diagram shows how a Site Designer could work with a Use Case (business requirements) person to update a page definition and check-in the file to the version control system:
: Site Designer : Use Cases Guy : Design Server : ClearCase ClearCaseUpdated page now ready to be deployed. To deploy, ftp the scripts as well as the page.xml to the target server and run the generated .bat or sh script.get: layoutpage layout, ACLplace portlets on pageadjust page settingsassign ACLassign unique name for pageftp: page export scriptexecute scriptpage.xmlcheck in: page.xmlextract scriptpage export script
Figure 4 - Page Update
6.4 Interaction between the Architecture Layers
In order to understand the build and deployment environment of the IE Pilot, it is important to understand that there are three main components of the system which interact with each other:
1. Portlets – these are applications that users interact with directly on the web.
2. Web Services – implement the business logic and access back-end systems.
3. Business Processes – these run inside Process Choreographer’s Business Process Engine, which models and executes business workflows.
The following diagram shows how these components work together at a high level:
7. Conclusion
In this white paper, we shared the lessons we learned from developing the build and deployment process and DPTK pattern generation. We also have a discussion on the artifacts for the components of which we are using to help us plan and implement the deployment templates.
