๐Ÿ‘จโ€๐Ÿ‘ฉโ€๐Ÿ‘งโ€๐Ÿ‘ง

E2E Testing

For test cases which aren't testable using lower levels tools in a pragmatic way (according to our testing mindset), we employ E2E tests. This will most likely always be the case for complex collaborative scenarios, testing different browsers, visual testing and rendering performance.

We do not test smaller details with E2E tests which are more pragmatically tested on a lower level. For example validation error messages.

Existing tools

Cypress E2E tests

Traditional sort of e2e tests using Cypress. Currently these are good for single user E2E scenarios.

Visual testing with Cypress

The 3d objects on the webgl canvas cannot be validated using the tools to assert DOM. If we find that we're having serious regressions with canvas rendering, we can setup the Cypress Snapshot plugin and use its image snapshots. These tests probably need to be very focused to test single rendering scenarios. Should probably only snapshot the canvas specifically, ignoring the rest of the page.

Performance

This is testing the rendering performance. It is running puppeteer inside a lambda, walking the user across the canvas and measuring the average FPS during its walks.

Bots

We have web socket bots which can be used to simulate a great deal of users in the room, but these move randomly, and do not have video or audio which they send. The bots are hosted in a lambda. Currently this is something which can't be triggered from e.g. cypress tests, nor do the bots have real audio/video feeds or exhibit deterministic behaviour.

TestRTC

TestRTC is made for advanced WebRTC testing, creating long running scenarios with multiple participants under various machine profiles and network conditions. However, since it exposes a full Nightwatch/Selenium agent you can also use it to do regular e2e testing, although it is not as nice as cypress.

The advantage is that TestRTC has a cloud which allows us to fire up tens or hundreds of browsers with real audio and video feeds, that we can control using test scripts. This allows us to run complex multi-user scenarios, even against localhost when doing development.

The disadvantage of TestRTC vs Cypress is that TestRTC doesn't have as great development experience. It doesn't do video screen capture of tests runs, making it more difficult to find the reason for a failure. This puts a heavier burden on the script writer to make a more easily debuggable test script.

TestRTC is by default a web based SaaS, but it exposes an api. We used this api in our e2e-testrtc-tests package on the frontend, to make it possible to manage and run these tests from our frontend repository.

Video/Audio quality of service testing

TestRTC provides ways to simulate many network conditions and assert quality of streams. Stream quality is not a focus yet until we've tackled "lower-level fruit".

TestRTC Spike