When there are problems with a mission-critical application, playing the blame game can stall progress and destroy your team's morale. How do you put an end to chaos and get your team back on track?

The following is an actual incident related by one of my company's consultants. A nervous young man greeted the consultant in the lobby on the IT floor. He simply said, "Follow me." As he walked down the hall to the application team room, the sounds of a heated discussion echoed in the corridor and grew louder.

The large meeting room was divided into "camps." Employees and contractors working for the systems integration firm were gathered in one corner around tables with laptops and printouts. The IT operations team was in another corner. Other camps included database management and mainframe administrators. In the center of the room a man and a woman were squared off facing one another. "Your architecture won't work, that's obvious! This performance is completely unacceptable! How can we can do so well in QA, but as soon as we go live we choke?" said the woman. "It's not the architecture," retorted the other, his voice rising in volume. "We've gone over this a dozen times. It's got to be either the way you have configured the app servers or your mainframe application."

The consultant's host pulled him aside. "It's been like this for two months. This is destroying our team's reputation and morale, and some execs are talking about killing the whole project. Can you do something about it?"

The visitor said to his host, "You are playing the blame game. The first step is to stop the chaos."

Twenty-four hours later, the team room had been transformed. Members from each team were huddled around a projected image on the wall. "Look at this metric," one was saying. "Database response time goes off the chart when users enter their lookup using a social security number. Remember the one we found yesterday, with the security server? This is similar. Okay, Bob, have your DBAs take a look at your lookup/join. Here's something else. On the WebSphere monitoring dashboard, watch what happens when that lookup takes too long." The project leader smiled, clearly satisfied with this newfound visibility into the whole Java application environment.

What had transformed the group from angry camps into a focused, high-performing team in such a short time? There were no group hugs, no team-building outings, no singing "Kumbaya" around the fire. There were two primary reasons. First, the visitor was able to recognize the symptoms of the blame game. Second, he was equipped with the software to give the deep visibility needed to bring the game to an end. Through no fault of their own your staff may also be playing this game. This article is about the blame game - what it is, how it starts, how to recognize it, and how to stop it.

What Is the Blame Game?
Mission-critical applications need to be managed to achieve their business objectives - revenue generation, customer service, and so forth. Application performance is measured on three factors - availability, performance, and control. Get high marks on these three factors and your application - and your application team - will succeed.

But the blame game can stall your progress and devastate your team. Why? Java applications are unlike monolithic legacy applications. A Java application is constructed more like a three-dimensional mosaic (see Figure 1). The application stack can include an operating system; a Java Virtual Machine (JVM); WebSphere; connectors to databases, mainframes, or other connected applications; and finally the application components themselves - servlets, EJBs, etc.

Due to the complexity of the application, management becomes a team activity. If a team-based problem resolution process does not exist or if it does not include enabling tools shared across the whole team, your team will descend into an endless round of fruitless experiments to find the root causes of the problems. This is the blame game.

The blame game, therefore, is defined as the uninformed finger-pointing that occurs among application team members when an application fails to achieve its goals. The game can result in loss of revenue, damaged brand reputation, damaged IT credibility, and even lawsuits and terminations.

How Does the Blame Game Start?
The blame game starts when a lack of hard data about production application performance forces each functional team member to defend his or her portion of the application under the philosophy that "If you can't prove it's my fault, it isn't." In other words, "Don't blame me!"

Consider all the people in your organization needed to develop and manage production Java applications - developers, architects, operators, QA engineers, DBAs, WebSphere administrators, CICS or MQ administrators, outside partners and suppliers. Typically, when a problem arises, the help desk notes it. If it's simple, operators fix it, restart the application, and move on. However, the complexity of Java applications causes most problems to be escalated to someone with an understanding of Java, of the architecture, of WebSphere, etc. At Wily we call this individual the Level 2 application support manager (ASM.) It's the ASM who performs problem triage and directs the problem to the right specialist for correction (see Figure 2).

If your organization has not nominated an ASM or if the ASM is not equipped with the tools needed to perform rapid triage, it's likely you are playing the blame game.

How to Tell If You Are Playing the Blame Game
1.   Cross-department finger-pointing is rampant. The project leader is caught up mediating battles between groups.
2.   Your best people - architects, senior engineers, and directors - are taking help desk calls. If the blame game goes on too long, they may consider leaving for calmer waters and more interesting work.
3.   Despite your investment in specialized coding, profiling, and testing tools, application quality is not improving. Development tools cannot be deployed in a production environment, where problems related to actual customer transactions first appear.
4.   Operations management is hiring Java programmers because their own monitoring tools are inadequate to isolate the root causes of problems.
5.   Development is hiring operators to monitor applications. They figure that since they are getting all the calls for poorly performing production applications anyway, they may as well do their own monitoring.
6.   Every new application throws the team into chaos. Just when you thought it was safe to launch an application, all the problems start again. If the blame game is not stopped permanently, it will recur with each new project.

If you've recognized any of these symptoms, your team is probably playing the blame game. You need to step in and put an end to it.

How to Stop the Blame Game
Again, IT teams fall into the blame game through no fault of their own. It's not because they are stubborn, stupid, or insensitive. The only way to end the game is to follow the steps outlined below.

1.   Stop the Chaos: The chaos can only be stopped by better information about the whole application - the Java components, the JVM, WebSphere, and the connections to databases and mainframes - which can only come from better application management tools. You must adopt tools that enable your ASM and the entire application support team to see inside a running production Java application and monitor its performance 24/7.

When problems occur, the application management solution you use must be able to provide everyone on the team - developers, architects, DBAs, etc.- with the data they need. The solution needs to show real-time performance of components and how they interact with one another. Problem ownership will then be obvious, not a subject for debate. The chaos will become control.

2.   Bridge the Gap Between the Teams: Once the chaos has been stopped and your team is at a point of relative stability and productivity, it is time to set the wheels in motion to permanently end the blame game.

One of the primary causes of the blame game is the breakdown in the key relationships between development and operations. Your team needs a common language and a shared view into the whole application. Your team members need a tool they all can use.

If you have not created a cross-functional application support team, this is the time. Establish the role of application support manager. Allow this team to monitor, improve, and manage applications to achieve service-level agreement targets for which they can be held accountable.

3.   Deploy to Avoid: Ultimately your application support team must develop processes and disciplines before deployment that enforce standards for application quality. Have the architects and developers build the production-monitoring dashboards while the application is being tested (see Figure 3). No one knows better what is most critical to an application's successful performance than the team that built it.

Finally, use your application management system to establish benchmarks for application quality. Put teeth into your development-staging-deployment process by prescreening applications and certifying them for memory usage, expected service levels, runtime load, external dependencies, etc.

For many readers, just achieving the first step - stopping the chaos - may represent a victory. However best-in-class IT organizations will not stop there. They know that lurking just around the corner, waiting for their unsuspecting employees, lies the blame game.