Wednesday, March 10, 2010

How can a production like environment be simulated in QA?

Recently we redesigned a Java/J2EE application to perform better. We did the following:-
1. Followed ySlow recommendations.
- Moved static content to an asset server.
- Added configurations to gZip and enabled expiry headers.
- Minified and combined js and css files using Closure compiler.
2. Removed the older Dojo Ajax framework and used jQuery which uses chaining and caching for faster DOM parsing
3. Removed middle tier service applications.
4. Removed Websphere Portal Server and used Websphere Application Server.
5. Reduced size of JSPs by removing whitespaces.
6. Cached DB data into an optimized schema for the application.

However, after deployment stability/performance issues increased. It seems that there were too many database materialized views being refreshed(some from previous applications and some introduced by the new application) that the database server was unable to cope with the load.

Before going live, the system was tested rigorously in QA:-
1. Load testing with 100 user load.
2. Performance testing across geographies.
3. Integration testing by starting other applications in a QA that were sharing the same infracture.

However, there were obvious gaps in our simulation of production in QA -
1. All back end jobs were not running in QA.
2. Even though the other applications were up and running, they were operating on no load.
3. The QA infrastructure is inferior to production (less RAM, lesser number of servers, etc.).

Which brings me to the questions, "how can a production like environment be simulated in QA?"