I’ve been trying to locate an existing tool that will record bunch of HTTP requests I make through my browser along with the responses, and then later run through though same requests again and verify the responses are identical to what was previously recorded. I want to use such a tool to test that refactorings don’t change the behavior of my web game at all, which is why I want the responses to be exactly identical (no allowance for even changes in whitespace even).
As I don’t plan to keep use the tests beyond this refactoring work I was hoping to find something that would completely automate the process (other then me needing to go through the scenario once in a browser), but the comparison part seems to the thing that is hard to find in no small part because using http in a search tends to bring in a lot of URLs presented as plain text rather then being actual HTTP links.
If you are aware of such a tool please post a comment!
A search for HTTP recorder immediately turned up the Perl HTTP::Recorder project, but that article only indicates that it records interactions (i.e. requests) and plays them back without doing any comparison of the responses.
I found this extensive list of web test tools, and tried out a few the offerings that looked the most promising.
I downloaded Minq Software’s PureTest, Web2Test‘s product, Versium’s vTest and tried each one’s record and playback. PureTest and Web2Test had no option, which I could find, for the comparison of the responses, while vTest has an HTML comparison option that has takes several manual steps to enable for each and every request. Both Web2Test and vTest actually bring up a browser and track the mouse versus PureTest (and HTTP::Recorder) that use a proxy to track the HTTP requests. Web2Test wasn’t able to get into my scenario as it uses of IE embedded in a window of Web2Test creation that wasn’t following HTTP redirects, which is pretty lame for a dedicated web testing tool that costs 1595 Euros considering vTest only costs 495 USD and it actually was able to record my scenario.
That said even after manually enabling page checkpoings and HTML comparison for some of my recorded requests in vTest, I found that it was generating false negatives on some of the HTML comparisons. The test run report only states that the page check point failed on the HTML comparison and vTest’s off by default logging shows no error details once enabled. Just to be sure I grabbed the page a couple of times and did a diff, but it didn’t show up any differences and none were expected based on how the HTML is generated for that particular URL, so vTest’s HTML comparison doesn’t actually work correctly and I couldn’t locate any kind of online community for the product to inquire as to what might be going wrong.
The last one I tried was Badboy and it also does not do wholesale response comparison although it does actually capture the responses and allows them to be manually compared (i.e. manually select two response then select diff). Badboy only costs $45 for folks at for-profit companies with more then five people and is free for everyone else including me and when its recording is saved as XML, it looks like I should be able to parse it in Python and use of the libraries I mentioned here to run through the requests and diff the current response with the baseline.
Now the question is just whether its worth the trouble considering I do have pretty good coverage via integration tests and it doesn’t take all that long run through the scenario manually.