Benchmarks concepts¶
The same FunkLoad test can be turned into a load test, just by invoking the
bench runner fl-run-bench.
Page¶
A page is an http get/post request with associated sub requests like redirects, images or links (css, js files). This is what users see as a single page.
Note that an xmlrpc call or a put/delete is also taken in account as a page.
Test¶
A test is made with 3 methods: setUp/test_name/tearDown. During the test_name method each get/post request is called a page.
[setUp][page 1] [page 2] ... [page n] [tearDown]
======================================================> time
<----------------------------------> test method
<--> sleeptime_min to sleeptime_max
<-----> page 1 connection time
Cycle¶
A cycle is a load of n concurrents test during a ‘duration’ period. Threads are launched every ‘startupdelay’ seconds, each thread executes test in a loop.
Once all threads have been started we start to record stats.
Only tests that end during the ‘duration’ period are taken into account for the test stats (in the representation below test like [—X are not take into account).
Only pages and requests that finish during the ‘duration’ are taken into account for the request and pages statistic
Before a cycle a setUpCycle method is called, after a cycle a tearDownCycle method is called, you can use these methods to test differents server configuration for each cycle.
Threads
^
|
|
|n [---test--] [--------] [--|---X
|...
| | |
|2 [------|--] [--------] [-------] |
| | |
|1 [------X | [--------] [-------] [--|--X
| | |
|[setUpCycle] | | [tearDownCycle]
===========================================================> time
<------ cycle duration ------->
<----- staging -----> <---- staging ----->
<-> startupdelay <---> sleeptime
Cycles¶
FunkLoad can execute many cycles with different number of CUs (Concurrent Users), this way you can find easily the maximum number of users that your application can handle.
Running n cycles with the same CUs is a good way to see how the application handles a writing test over time.
Running n cycles with the same CUs with a reading test and a setUpCycle that change the application configuration will help you to find the right tuning.
cvus = [n1, n2, ...]
Threads
^
|
|
|n2 __________
| / \
| / \
|n1 _________ / \
| / \ / \
| / \ / \
| / \ / \
==================================================> time
<-------> duration <-------->
<-----> cycle sleep time
Tips¶
Here are few remarks/advices to obtain workable metrics.
Since it may use significant CPU resources, make sure that performance limits are not hit by FunkLoad before your server’s limit is reached. Check this by launching a bench from another host.
Having a cycle with one user gives a usefull reference.
Run multiple cycles for a bench.
A bench is composed of a benching test (or scenario) run many times. A good benching test should not be too long so you have a higher testing rate (that is, more benching tests can come to their end).
The cycle duration for the benching test should be long enough. Around 5 times the duration of a single benching test is a value that is usually a safe bet. You can obtain this duration of a single benching test by running
fl-run-test myfile.py MyTestCase.testSomething.Rationale : Normally a cycle duration of a single benching test should be enough. But from the testing platform side if there are more than one concurrent user, there are many threads to start and it takes some time. And on from the tested platform side it is common that a benching test will last longer and longer as the server is used by more and more users.
You should use many cycles with the same step interval to produce readable charts (1:10:20:30:40:50:60 vs 1:10:100)
A benching test must have the same number of page and in the same order.
Use a Makefile to make reproductible bench.
There is no debug option while doing a bench (since this would be illegible with all the threads). So, if a bench fails (that is using fl-run-bench), use
fl-run-test -dto debug.