All the projects of the Bulletin Board are comprehensively covered by unit tests, while the Bulletin Board server is also end-to-end tested with Cypress exploiting the sandbox.
You can execute the whole test suite by running
make test from the root folder of the repository.
If you want to only run the tests of one project, you can do it by running
make test_electionguard singularly.
All tests are run in the CI process.
The aim of load tests is to make sure the system can handle the expected load and to spot possible performance leaks that only become apparent with high loads of traffic.
Load tests have to be tailored to the event we are preparing for, and as such we need to configure and run them manually.
In our case, we will primarily focus on load testing the Bulletin Board during election time.
We divide the test phase in 4 steps:
The Bulletin Board comes with a sandbox that ships some tools to help with the load tests. Make sure you set the
SANDBOXenvironment variable to any value to activate it;
The web server and the background worker server shall run in separate machines in order for web traffic and background jobs not to affect each other;
Use Sidekiq as queue adapter;
iat_expiration_minutesto few hours/days. This is the amount of time a vote is considered valid from its emission - since the vote generation phase could take a few hours, you don’t want it to be too low and have the Bulletin Board to reject the votes when you start the load test.
As mentioned above, the sandbox provides some utilities to help us to run a load test. Between them, there’s the vote generation.
The vote generation utility emulates what happens in the voters' browser when they encrypt the vote, but it does it on the server and in bulk. Notice that the votes are randomly generated - from the selected answers to the voter ids.
To generate the votes, follow these steps:
Access the sandbox at
/sandbox/elections/- make sure the sandbox is activated;
Set up a new election. Notice that the more questions and answers, the slower the encryption and decryption process will be. It is hence important to have a similar election setup to the one we are preparing for to have valid results from this test. Also, make sure you select
voting scheme name;
Perform the Key Ceremony following the wizard and start the voting period;
You can now click on
stress teststo access the vote generation utility.
Input the number of votes you want to generate and hit
The vote generation process is heavy and can lead to high memory consumption. With 1Gb of RAM you could run out of memory if you generate more than 2000 votes in one go. You can launch several consecutive generations and merge the results at the end.
The vote generation process can take a long time (proportionally to the number of votes).
Avoid launching other generation processes when one is already running.
You can check Sidekiq’s queues at
The generation process uploads a CSV with the votes data to transfer.sh. Once the process is finished, you can download the CSV, hitting
Download votes. Notice that you’re also prompted the time when the file has been uploaded. That is for you to know when the new file is available in case you want to launch another votes generation right away.
If you generated the votes in more than one process, merge the CSVs into one.
We are now ready to start the actual load test.
We will employ Apache JMeter 5.4.1 and feed it with the CSV we just generated.
Download and unarchive JMeter. Open its UI by running:
Open the following file to use as a template for the load test configuration:
You will need to adjust some configs to your needs:
HTTP Request Defaultsput the hostname/IP of your BB installation in
Server Name or IP;
CSV Data Set Configselect the CSV with the generated votes in
Thread Groupyou can choose:
How many votes to cast (
Number of Threads);
What span of time should the votes be sent (
This configuration will send 1 vote every 0,5 seconds
To check if the configuration is correct, send a low number of votes (e.g.
Number of Threads = 2).
Hit the play button and check in
HTTP Request →
View Results Tree if there is any error.
Once you are happy with the configuration, hit the save button and close the JMeter UI. To run the actual test, we will use JMeter in the more lightweight CLI mode.
Here is the command:
./bin/jmeter -n –t jmeter-bb-load-test-config.jmx -l testresults.jtl
Every 30 seconds, JMeter will prompt some stats in the terminal, so you can follow along the execution of the test.
It also logs the results in
testresults.jtl, that you can later analyse.
While JMeter is running the test, you’ll want to check the Sidekiq queues state on its dashboard at
There are three main metrics we can measure:
Roundtrip time for a request from JMeter;
Votes processing time;
Server resources consumption.
Roundtrip time: we can measure the requests roundtrip time over time to check if the web server can handle the predicted volume of traffic over a long period of time.
Remember that the votes are not processed synchronously: every request enqueues a job that is processed in the background, so this only covers the web server load.
This metric is provided by JMeter itself using the
testresults.jtl generated during the test (see https://jmeter.apache.org/usermanual/generating-dashboard.html)
Votes processing time: the Bulletin Board sandbox also provides a tool to gather some basic metrics of the vote processing.
Load test stats to gather the stats of the votes processed during the last 2 hours.
Server resources consumption: we can check how the server(s) reacted to the load by checking the resource consumption over the time - memory, CPU, response time.