Announcing our open source security key test suite

Posted by Fabian Kaczmarczyck, Software Engineer, Jean-Michel Picod, Software Engineer and Elie Bursztein, Security and Anti-abuse Research Lead

Security keys and your phone’s built-in security keys are reshaping the way users authenticate online. These technologies are trusted by a growing number of websites to provide phishing-resistant two-factor authentication (2FA). To help make sure that next generation authentication protocols work seamlessly across the internet, we are committed to partnering with the ecosystem and providing essential technologies to advance state-of-the-art authentication for everyone. So, today we are releasing a new open source security key test suite. 

The protocol powering security keys

Under the hood, roaming security keys are powered by the FIDO Alliance CTAP protocols, the part of FIDO2 that ensures a seamless integration between your browser and security key. Whereas the security-key user experience aims to be straightforward, the CTAP protocols themselves are fairly complex. This is due to the broad range of authentication use cases the specification addresses: including websites, operating systems, and enterprise credentials. As the protocol specification continues to evolve—there is already a draft of CTAP 2.1—corner cases that can cause interoperability problems are bound to appear.

Building a test suite  

We encountered many of those tricky corner cases while implementing our open-source security-key firmware OpenSK and decided to create a comprehensive test suite to ensure all our new firmware releases handle them correctly. Over the last two years, our test suite grew to include over 80 tests that cover all the CTAP2 features.

Strengthening the ecosystem 

A major strength of the security key ecosystem is that the FIDO Alliance is an industry consortium with many participating vendors providing a wide range of distinct security keys catering to all users' needs. The FIDO Alliance offers testing for conformance to the current specifications. Those tests are a prerequisite to passing the interoperability tests that are required for a security key to become FIDO Certified. Our test suite complements those official tools by covering additional scenarios and in-market corner cases that are outside the scope of the FIDO Alliance’s testing program.

Back in March 2020, we demonstrated our test suite to the FIDO Alliance members and offered to extend testing to all FIDO2 keys. We got an overwhelmingly positive response from the members and have been working with many security key vendors since then to help them make the best use of our test suite.

Overall, the initial round of the tests on several keys has yielded promising results and we are actively collaborating with many vendors on building on those results to improve future keys.

Open-sourcing our test suite 

Today we are making our test suite open source to allow security key vendors to directly integrate it into their testing infrastructure and benefit from increased testing coverage. Moving forward, we are excited to keep collaborating with the FIDO Alliance, its members, the hardware security key industry and the open source community to extend our test suite to improve its coverage and make it a comprehensive tool that the community can rely on to ensure key interoperability. In the long term, it is our hope that strengthening the community testing capabilities will ultimately benefit all security key users by helping ensure they have a consistent experience no matter which security keys they are using.

Acknowledgements 

We thank our collaborators: Adam Langley, Alexei Czeskis, Arnar Birgisson, Borbala Benko, Christiaan Brand, Dirk Balfanz, Guillaume Endignoux, Jeff Hodges, Julien Cretin, Mark Risher, Oxana Comanescu, Tadek Pietraszek and all the security key vendors that worked with us.