How do I benchmark (compare two different implementations) and profile (find performance bottlenecks in an implementation) behavior of a web application (in Perl) under heavy load? This web application is (supposedly) I/O bound rather than CPU bound.

I'd like to compare proposed implementation (proposed improvement) with the current solution, either on a real load, but minimizing an eventual performance hit (perhaps two different implementations on different servers behind a load balancer), or replay / simulate load on a new implementation, or simulate heavy I/O usage.

I'd like also profile CPU and I/O bottlenecks in proposed implementation, but testing behavior under heavy load.

The OS is Linux, and the web server is Apache.

The same question is posted on ServerFault: How do I benchmark and profile IO bound Perl web application under heavy load?

I think the answer here can be about benchmarking and profiling the web application written in Perl, and perhaps also about replicating realistic load conditions, or finding measure / benchmark which would predict which implementation would perform better under I/O bound conditions under heavy load.