Defining Project Scope

I am confused as to why some companies insist on running huge projects that invariably run into trouble with dates, quality and budget.

Here is an example of what I mean. I have worked for several customers who initiated the "SOA Project". These were multi-year, multi-team, multi-budget, multi-million dollar projects. And all of them ran into trouble one way or another just through being too big.

The last time I worked on one of these, my company was brought into to replace another vender who had thrown in the towel. i.e. It was so bad they would rather lose the account and walk away than complete the engagement.

SOA is not a project, it is an architecture pattern. Same thing applies to EDA. These are design principles that are applied to projects.

My thinking is this. A long term strategy should not be encapsulated as one project with a dozen phases. Instead, divide projects up into two categories:

Business Value projects are those that have a clearly defined goal with a concrete return-on-investment. These projects should be accounted for using a budget from a business unit. A couple of examples: Stream-line processes to reduce the average call handle time in a call center. Enable customer self service via the web.

Infrastructure projects involve bringing new technology into the enterprise that will increase the return-on-investment of Business Value projects. These projects should be classed as capital expenditure. For example: Implement a new middle-ware product to enable application integration with a legacy system. Implement an enterprise message system.

It may be that an infrastructure project becomes a pre-requisite for a specific business value project. If this is the case, then we should be clear on the cost accounting as this will dictate whether the infrastructure project is a good investment. For example, will this infrastructure enable other business value projects? Will it enable the IT team to better serve the business?

I look at these things as a Techie. Big problems are easier to solve if they are broken up into smaller problems. Small problems can be solved quickly (and for less cost) while working in the context of a long term strategy. We should maintain an agility that will enable us to change direction, as needed, without discarding that which has already been acheived.

In my view projects should not exceed six months in durarion. Three to four months would be the optimum size. In this way the business receives regular benefit from the IT team. Business propsers, makes more money, more money funds new projects, new projects help the business to prosper.

It boils down to this: Clearly defined goals based entirely on a well understood business value proposition. Small, agile teams working to a limited scope for rapid delivery.

James
Atlanta
April 2009

Shock Absorption for Legacy Systems

When it comes to performance testing, some people (not used to working with legacy systems) can have a hard time understanding that limiting the number of concurrent requests that hit the back end application can actually improve performance.

A very talented colleague of mine quite often refers to "the knee of the curve" i.e. the point at which response times dramatically increase because the target system is having a hard time of it.

In pre-sales calls we quite often describe our Legacy Integration products as the shock absorption layer for the back end application. We have a handy little feature that enables us to limit the number of concurrent requests active on a connector (a connector being the implementation of a specific protocol for a specific resource.e.g. TN5250, TN3270, JDBC and anything else you care to shake a stick at).

I spent this week tuning an implementation of our Legacay Integration product for a rather large client. The target was a test database on an underpowered iSeries. Past a certain point we saw dropped connections and all manner of strangness. I played around with our throttle control and the size of the connection pool. It didn't take long to find the sweet spot and the average response times dropped back to where they should be.

This is a good example of less is more :-)

To many of you reading this I am sure that the above seems like common sense. Any system, legacy or contemporary has a finite set of resources, so of course performance will degrade as the load exceeds the systems capabilites. My point is that shock absorption must be considered when planning and implenting an integration solution for Legacy systems.

For example, most iSeries shops run at or near capacity. When the iSeries Admins express concerns about the additional load that an integration solution may bring, the handy throttle control can save the day.

So, if your current legacy integration product does not cater for shock absoption or transaction throttling, feel free to drop me a line :-D

James
Atlanta
August 2008

Palringo

I was looking for an IM Client for my wifes new iPhone 3g and came across Palringo. It is free and works realy well. It'll run on a bunch of platforms, so now I am using on my laptop to replace Windows Live Messenger and AIM.

I love finding those realy useful gadgets where it is apparent that the designers realy thought about it. I cannot beleive it is free. Considering some of the applications on iTunes that people want money for, I would have happily spent some cash on it.

Integration with Social Networks; What , Why & How

While debating this topic in the car yesterday with Chris on the way back from Charlotte, NC Chris, always the practical one, asked "Why do this, what is the point?". My immediate answer was "If some VP of Marketing wants to pay me to open up a new marketing channel then I'll take his money and implement a solution." Which I admit was pretty weak.

I am not too good at the vision thing. When some new concept is thrown out there I usually ignore all the noise until I see a solid application of it. Then I try to cut through the spin and the BS spouted by the wannabes trying to be part of the "next big thing" and find out what is it all about.

What is it? Why do it? How do I implement it?

So I decided to do a little more home work and come up with a better answer. One that would satisfy Chris.


The What

To enable any website to incorporate their own little mini social network among their own users and to have the capability of Identity Mapping to allow users to hook into the big Social Networks like Facebook and MySpace.

For example: When a user signs-in to your site, present a summary of their profile from Facebook. Query Facebook and MySpace to present your user with a list of their friends with upcoming birthdays and ask if they would like to send birthday e-cards, order a gift or send flowers.

The Why

It all comes down to getting the best return on investment out of the Marketing budget and helping your company to sell more product/services/widgets/stuff/dual-phase-inter-wobblers:

• Learn more about your consumer to design better campaigns
  
• Get feedback as to how your brand is faring or how a campaign is operating
• Find new ways to sell your product or service; create special offers for each individual much like Amazon does.
  

The How

The main players in the Social Networking space are opening up by providing open source APIs. Facebook has aleady announced this.

Here is a quote from Bob Bickel
"We see the Facebook API and the Open Social API becoming the two standards for Social Application development over time – just like .Net and J2EE became the standards for web applications around the turn of the century. Ringside allows any website to build Facebook and/or Open Social applications today that will run on both your website as well as Facebook and the large Open Social websites like MySpace, hi5 and others."

Essentially Ringside is providing an abstraction layer to existing Social Network APIs. If you need to get going with implementing this kind of functionality for your company Ringside is saying "Start here".

Once you have your shiny socially aware website I think you will need to consider Business Intelligence tools to help you understand all that data you are collecting.

And you may also need to integrate with your internal systems to further enhance your site's capabilities. For example, customer self service functions that enable your users to tell you about changes of address, telephone number and other data private held by you on behalf of your customer.

James
Atlanta
June 2008

Application Integration with Social Networks

A former colleague of mine called yesterday to talk about a new idea being proposed by Ringside Networks.

These guys were the power behind Bluestone and JBOSS. My friend's thinking is that these chaps are two for two and if they think this is a good thing to invest in then it is worth a close look.

It certainly is a cool idea. They have come up with an application engine, the Ringside Social Application Server, that is designed to add a social context to a web application. The idea is to make your web site more of a community thing. So, for example (taken from Ringside's web site), if you sold running shoes then you could provide functions for your sites users to organize and take part in events or find a training partners of similar skill & experience in the same area.

It strikes me as a new concept in sales & marketing. Social networking sites represent a new way to reach a large group of people who are actually interested in your products or services and will come to you.

Ringside estimate that there are 67 million users of social network sites and that advertising from search engines can only take you so far. They propose that the smart companies will leverage social networking for more targeted marketing. They also estimate that there are approximately 19,000 applications leveraging the Facebook API alone.

The key fact about this new push for an open source social networks application server that strikes a chord with me is that the idea came from a real, business driven need. The original authors were paid to create a social context for the running shoe company described in the example above. They then set about creating a more generic engine that could be quickly implemented for anyone else.

What would turn me off would be to hear that someone dreamed it up and decided to make it. I am not a fan of "build it and they will come". I much prefer to see that an idea has real business value. The fact that someone has already been paying for services to implement this idea tells me that it does.

The next thing to discover is whether or not there is value in integrating these applications with back end systems be they legacy or contemporary. With our ability to expose legacy business functions as services, for example as web services, and a range of options for communications (HTTPS, JMS, etc) it should be easy to integrate with the Ringside Social Networks Application Server.

James
Charlotte, NC
June 2008

Lights, Camera, Action! Project Planning & Execution

As you can see, I haven't been blogging for a while. I spent half of 2006 and most of 2007 on a project that, quite frankly, made me question my will to live.

2008 is being kinder. Just finishing up phase one of a rather enjoyable project. Phase two is coming up and couple of other smaller projects are about to kick off.

After sitting down and watching an episode of "So you think you can dance?" with my wife this evening, I decided to spend a few moments before bed reading posts from some of my favorite bloggers. I guess I have been inspired, so consider this an attempt at "So you think you can blog?"

I started to write the following last year, during that "challenging" project. I am not sure why I didn't post it at the time. Oh well, no point in wasting the words :-)

James
Atlanta
June 2008

I can see the light at the end of the tunnel. Another project is drawing to a close. It has been a wild one. By the time we are done it will have been two and a half years of blood and sweat. (Actually, only a year and a half for me, but I was using the royal "we")

At this point in a project I try to look back and consider the good and the bad, the highs and lows, etc. I think of the way things would have been done in an ideal world and recall the numerous reasons why compromises had to be found for "this", for "that" and for "the other".

We faced many challenges on several fronts: technical, political and of course, financial. This is normal. Any project with a large scope will experience all of these things to varying degrees. It the way we deal with them that counts.

When I boil it all down. Distill my experiences of this project and those projects passed, I come to the conclusion that any project dealing with Information technology, be it integration or otherwise, will always be successful given the following items:

During planning:

• Make realistic estimates. There is no point in under estimating just to win the business.
• Add more wiggle room to the project plan for red tape for the larger corporate customer
• Allow enough time for testing
  
• Assume that bugs will be found and allow enough time to resolve and re-test. Testing is an iterative process
• Where appropriate make time to load test and measure performance. This is also an iterative process
• Assume a 40 hour working week. Morale and quality go out the window if time lines are too aggressive.
  

Before writing a line of code:

• Document unit tests
• Prepare data for each unit test
• Create a backup of prepared data
• Create routines to automate unit testing
• Prepare a source control repository
  
• Create routines to automate the build/compile from source
• Document Standards & Conventions
  

During development:

• Build the application every night
  
• Restore test data and run regression tests every night
• Peer code reviews
• Re-factor code without mercy; Quality should be the primary focus
  
• Stay in touch with Subject Matter experts (especially important for legacy integration)
• Team meeting for 30 minutes at the start of every day. Identify road blocks, share information, share the "big picture"
• Maintain morale; celebrate milestones as a team. Especially important for projects that will run for a long time.
  

Continuity of Staff is essential if deadlines are to be met as is a strong project manager. Tech Leads who understand that to lead is to serve. SMEs are your friends.

James
Atlanta
May 2007

Non-Invasive Application Integration - Not your Father's screen scraping

Let's talk about integrating with a Legacy application via its user interface.

If you can access the data underneath a legacy application, nine times out of ten it is going to be faster than getting it from the screen. As no one is going to advocate one application updating a legacy database concurrently with another application sometimes the user interface is the only way to go.

There are good reasons; the legacy application provides all the validation and marshalling of data into the appropriate (usually) non-relational tables. And there are those occasions where the legacy application is presenting some calculated value, the result of a piece of business logic that is entombed inside a tight ball of dreaded spaghetti code.

Vendors of legacy integration tools wince at the term "screen scraping" and this term appears to emote fear and loathing among IT professionals looking for a way to glue their legacy application together with a shiny new contemporary application.

I think I know how this came about.

In the 80's most products that I shall refer to as "first generation" legacy integration tools, were effectively glorified terminal emulators. These tools relied on a state-full connections with the host system over indirect connections (networks) and direct connections (serial communications and multiplexers).

A number of factors combined to create painful experiences for the IT pros called upon to implement solutions with these tools.

First, it was early days - slow communications and lack of experience with this emerging market gave consumers a collection of flakey tools to choose from.

Second, legacy applications weren't so "legacy" back then. They were living and breathing applications being actively maintained by the thick eye-glasses and pocket protector brigade (yeah, that's right, the same guys who got paid mega-bucks in the late 90's to make their own damn code Y2K compliant) which meant that those X-Y screen maps that you spent hours creating were rendered useless the very second that a maintenance release hit the production environment.

So what has changed between then and now? On the first point, Darwin's theory of natural selection. Some vendors went bust, a few changed direction or went back pushing regular terminal emulation. But a few vendors remained in the market, learned from their early mistakes and those of others and a second generation of "re-facing" products emerged delivering smarter development tools and strategies for change control to mitigate the work of pocket protector brigade. Also, the legacy applications became more “legacy” as some of the guys with thick glasses moved to Florida and there were less of them around to mess with the display.

Nowadays we are on the third generation. Web-to-host, thin clients, service interfaces, integration servers, stateless connections, pooled host sessions and code generators no less. Sounds cool, huh? Well yeah it is. I especially get a kick out of the code generation. Programs begetting programs, it's all a bit Arthur C. Clarke.

However, I think the biggest innovation comes to us in the form of the alternative routes into the legacy application, especially mainframe applications. 3270 and 5250 are nice protocols to work with. IBM did a good job when it designed them as data streams. They were light years ahead of character mode protocols such as DEC VT and the myriad of variants of the day. The concept of delegating the work of interacting with a user to a client device thus enabling the server to handle much higher rates of concurrency is the same approach used by that there new fangled electric inter-web.

But protocols are usually stacked on top of protocols and each layer becomes less friendly yet more efficient the deeper we go until it becomes positively primitive. Sneaking in underneath 3270 we find 3270 Bridge, FEPI (Front End Programming Interface) and deeper still COMMAREA. The ability to plug in at any point in the stack is a powerful one yielding large gains in throughput as we strip away unnecessary fluff.

Of all the legacy applications out there those hosted by mainframes present the most options for robust non-invasive programmatic integration, especially CICS applications.

The iSeries with TN5250 comes in a close second, given a sub-system that has been tuned to recognize that robot users resemble batch jobs versus the traditional 12 second key/think time normally allowed for interactive jobs.

Following up the rear are the character mode applications. Traditionally hosted on UNIX systems and not usually written to any UI standard. Every application is entirely different from all the others. Integration with those applications that are based on some kind of framework or were generated by a 4GL tend to be easier to work with, in that screen A behaves pretty much the same as screen B.

Integration tools that perform well with character mode applications come from vendors who understand that it is not about managing screens as it is managing the state of a screen and having smart algorithms for figuring out when the application has finished painting. Input Inhibited Flag? Pah! Luxuary!

I have seen plenty of vendors who do well with page mode protocols (3270 & 5250) but truly suck at character mode protocols and vice versa. Only a few vendors do well at both and offer the ability to drill into the stack of supporting protocols where available.

The best performers are those tools that can pool host sessions and park them for re-use. Serving 100 users with 10 interactive host sessions is an efficient use of resources.

The need for non-invasive programmatic integration with legacy applications is going to be with us for quite some time. In the past year two of my clients purchased brand new mainframes from IBM while at the same time developing new enterprise class applications (one went with .Net, the other with J2EE). While many discuss their company’s “mainframe replacement” strategy, few fully appreciate the time and planning required to make this a reality. Good quality application integration can offer a roadmap towards this goal by enabling an enterprise to control the pace of change while still responding to the requirements of the business, which is why we all have jobs in the first place.

A wise man once told me that any application currently in production is "legacy". Already vendors are offering tools to expose services from legacy Win32 applications as well as tools for 'clipping' content from legacy web applications. The wheel keeps turning and many of the techniques used currently for non-invasive programmatic integration can be applied to contemporary applications.

I hope that we learn the lessons of the past and embrace concepts such as Service Oriented Architecture while striving for interoperability with standards such as web services.

Leveraging standards is the key to successful legacy integration. So when today's sexy enterprise applications become tomorrow’s dinosaurs our children will either praise us or flame us based upon our adoption of well defined standards.

So, as I grow older I remember what has gone before and wonder what is to come. Stronger eye glasses for sure and maybe an urge to carry several colors of ball point pen with me "just in case". But I do know this….there will always be a legacy application that needs liberating.

James
June 2006
Minneapolis

Making Legacy Applications "Open"

Service-Oriented Architecture promises agile, open information technology solutions that will enable businesses to better serve the customer, to reduce costs and to be more competitive. SOA is not something that can be ordered for next day delivery. It is an ideology that can get lost in the rush to meet deadlines.

There are so many ways in which to expose core business functions that are locked away in legacy applications that there is almost no excuse for not doing it. Trouble is that there seems to be a general perception that programatic integration with legacy applications is un-reliable. In my experience this is not the case.

There are many vendors in this market with many years of experiencing in providing tools for just this job. The biggest challenge for an IT department these days is choosing the right tool from the right vendor. Proof of Concept, dude! There is no such thing as a bad POC; except for a vendor of course :-)

I'll be upfront about the purpose of this web log. It is a place to share ideas and dicuss technical details. It is not a place to promote one vendor over another. This is a big deal, I am a Solution Architect for a vendor of Legacy Application Integration tools.

There are a lot of misconceptions about integrating with Legacy applications, and I am hoping that this will be a place to dispell many of them.

James
June 2006
Atlanta