In this blog we outline a FUZZ alternative: CoSMo KITE WebRTC testing – check out an alternative FUZZ
There has been quite some buzz (and fuzz) in the webrtc world recently due to a series of great posts from Natalie Silvanovich, Project Zero in which she demonstrated the power of end-to-end fuzzing to find vulnerabilities and bugs in webrtc stacks and applications, including browsers, FaceTime, WhatsApp….
Inspired by the above wonderful but admittedly complicated work, we decided to investigate how we could provide a much easier to use fuzzer to stress test all webrtc apps with. So we came up with cosmo-kite-webrtc-testing … Combined with the automation and interoperability testing capacities of KITE, it would become the perfect stress tool for the entire WebRTC community.
Let’s address the elephant in the room. Our goal was not to replace unit test fuzzing which is guaranteed to give better results than our solution.
Unit test fuzzing, and LLVM project’s implementation libfuzz, is the best way to go at it. While very well documented (*), using libfuzz, or reproducing the process followed by project-zero require great computer science skills, as well as time and resource that most projects simply don’t have.
Practically, not all teams have resources in-par with Google and can afford to spend weeks (or months) to create either unit tests for every single line of code, nor mockups for all functionalities to enable end-to-end testing.
We wanted something usable right-away out-of-the-box without any modification of your existing client app. We wanted non-coder to use it. We wanted CEOs to use it. We want to redefine “easy testing” of webrtc applications and systems.
FUZZING OF WEBRTC
According to Wikipedia, a fuzzer can be categorized as follows:
- A fuzzer can be generation-based or mutation-based depending on whether inputs are generated from scratch or by modifying existing inputs,
- A fuzzer can be dumb or smart depending on whether it is aware of input structure, and
- A fuzzer can be white-, grey-, or black-box, depending on whether it is aware of program structure.
We have decided to implement a dumb mutation-based black-box fuzzer inside a webrtc-capable browser.
Typical webrtc attack vectors for an SFU are:
- Signalling, SDP O/A,
- DTLS, SRTP
- RTP, RTCP
- Codec payloads
We decided to focus our attacks against the RTP and RTCP protocols, as well as the corresponding encoded media payload part. The reasons are manifold:
- Most SFUs uses standard libraries for SRTP/DTLS and ICE, which are extensively tested. There would be but poor targets.
- Equally, signaling and SDP O/A are very often parsed and/or serialized, which are two operations very well suited for unit test fuzzers.
- Most of the workload on an SFU is RTP/RTCP packets management and corresponding encoded media payload inspection.
Our fuzzer works by mutating a certain amount of bits within the RTP and RTCP payloads before the SRTP encryption is applied. We don’t randomize whole bytes in the input as proposed on the project-zero article, but instead flip individual bits as it will be more likely to pass RTP parser checks and generate errors deeper into the WebRTC stack. Grenade have a better impact when exploding if ingested
Cosmo KITE WebRTC testing or “CoSMo’s HOT n EASY FUZZER”
As libsrtp performs some rtp header verifications before protecting each packet, we have implemented the fuzzing logic inside libsrtp itself. This way we are sure that the modifications are made at the last step of the packet creation, and that no error can be generated on the sender side.
Having extensive experience in all kinds of libwebrtc-based builds, we can then leverage this “weaponized” libsrtp in libwebrtc, in open-source browsers using it (chrome, firefox, …), in electron build and/or in native client trivially.
- Work against any webrtc-based solution that has a web client available.
- Easily embeddable in any native client code.
- No set-up.
- No coding required.
- No modifications on the SFU (if any).
- End-to-end testing on real-life scenarios.
- Somewhat slower at finding bugs than testing tens of thousands of RTP/RTCP packets per second as an unit test fuzzer would.
- Not deterministic, as it work on real-life inputs from the browser.
- Requires being able to debug or troubleshoot the SFU (if any) or remote client (if p2p) to actually find the root cause of any crash.
- Not very efficient against java-based implementations like Jitsi.
- Real-Life test and validation
TESTS AGAINST REAL TARGETS
We compiled the current stable version of both Chrome and Firefox against our modified version of libwebrtc.
We first used those modified versions with appRTC, a p2p 1:1 webrtc application, against unmodified browsers. It looks like the zero project team had done a good job, and after 15mn we did not have a single crash against Chrome and Firefox.
We then ran the hosted demos of several webrtc SFUs. Medooze, MediaSoup and Janus were down in a matter of seconds. A Jitsi Video Bridge instance kindly provided by the Jitsi team sustained 15mn of attack without crashing (thanks in part, we assume, to java, and in other parts to the great job of the Jitsi team). We did not test any other SFU. Several bug fixes have been provided to the open-source repositories of the above-mentioned SFUs.
We eventually ran it against the web clients of several well know PaaS and CPaaS, for no more than 15 mn at a time, and for most of them we communicated them the results of our run.
We’ve shown that we can easily democratise fuzzing by modifying webrtc internals and building native apps (or browser) with the modifications in. Now, anybody with absolutely no coding skill at all can test any webrtc system or app, wether the media is exchanged in P2P or through a media server.
Of course this naive testing will be much slower at discovering vulnerabilities than fuzzing unit tests. Of course a non-coder will only be able to tell is a vulnerability exists by observing the service or app crashing, without being able to actually point to the source of the vulnerability, but isn’t that already a great information? Moreover in the hands of people that can actually code, and attach a debugger to the right app or server, you have now a no-setup, no homework fuzzing tool at disposal in your ever-growing testing toolbox.
Even easier: this external fuzzing capacity has been added to the KITE testing engine for automation, and is currently available with open-source browsers like chrome and firefox. Packages of modified libwebrtc are available for quick integration in SDKs to address the testing needs of the mobile-first / mobile-only enterprises.
Key word cosmo-kite-webrtc-testing