Ready, aim, fire: an open-source tool to test web security scanners

Securing modern web applications can be a daunting task—doubly so if they are built (quickly) with diverse languages and technology stacks. That’s why we run a multi-faceted product security program, which helps our engineers build and deploy secure software at every stage of the development lifecycle. As part of this effort, we have developed an internal web application security scanning tool, codenamed Inquisition (as no bug expects it!).

The scanner is built entirely on Google technologies like Chrome and Google Cloud Platform, with support for the latest HTML5 features, a low false positive rate and ease of use in mind. We have discussed some of the technology behind this tool in a talk at the Google Testing Automation Conference 2013.

While working on this tool, we found we needed a synthetic testbed to both test our current capabilities and set goals for what we need to catch next. Today we’re announcing the open-source release of Firing Range, the results of our work (with some help from researchers at the Politecnico di Milano) in producing a test ground for automated scanners.

Firing Range is a Java application built on Google App Engine and contains a wide range of XSS and, to a lesser degree, other web vulnerabilities. Code is available on github.com/google/firing-range, while a deployed version is at public-firing-range.appspot.com.

How is it different from the many vulnerable test applications already available? Most of them have focused on creating realistic-looking testbeds for human testers; we think that with automation in mind it is more productive, instead, to try to exhaustively enumerate the contexts and the attack vectors that an application might exhibit. Our testbed doesn’t try to emulate a real application, nor exercise the crawling capabilities of a scanner: it’s a collection of unique bug patterns drawn from vulnerabilities that we have seen in the wild, aimed at verifying the detection capabilities of security tools.

We have used Firing Range both as a continuous testing aid and as a driver for our development, defining as many bug types as possible, including some that we cannot detect (yet!).

We hope that others will find it helpful in evaluating the detection capabilities of their own automated tools, and we certainly welcome any contributions and feedbacks from the broader security research community.

Posted by Claudio Criscione, Security Engineer