By Refael Botbol
This article will describe the steps you need to take to smoothly run a 50K concurrent users test, at least from the load testing point of view.
Check out the webinar recording at the bottom of this article.
Quick Steps Overview
Write your script
Test it locally using JMeter
BlazeMeter SandBox testing
Setup the amount of Users-per-Engine using 1 Console & 1 Engine
Setup and test your Cluster (1 Console & 10-14 Engines)
Use the Master / Slave feature to reach your max CC goal
Step 1 : Write Your Script
Before you begin, make sure to get the latest JMeter version from the JMeter Aapche community jmeter.apache.org .
You also want to download these additional plugins as they will make your life a lot easier.
There are many ways to get your script:
Use the BlazeMeter Chrome Extension to record your scenario
Using the JMeter HTTP(S) Test Script Recorder which basically sets up a proxy so you could run your test through and record everything
Going manually all-the-way and constructing everything from scratch (probably for functionality/QA tests)
If your script is the result of a recording (like steps 1 &2), keep in mind that:
You will need to change certain parameters such as Username & Password or you might want to set a CSV file with those values so each user can be unique.
You might need to extract elements such as as Token-String, Form-Build-Id and others using Regular Expressions, JSON Path Extractor, XPath Extractor – in order to complete requests as “AddToCart”, “Login” and more…
Keep your script parameterized , and use config elements such as HTTP Requests Defaults to make your life easier when switching between environments.
Step 2 : Testing Locally Using JMeter
Start debugging your script with 1 thread, 1 iteration, using the View Results Tree element, Debug Sampler, Dummy Sampler and the Log Viewer opened (in case some JMeter errors are reported).
Go over all the scenarios (True and False responses) to make sure the script behaves as it should…
After the script has run successfully using 1 thread – raise it to 10-20 threads for 10 minutes and check:
If you intended that each user be unique – is that so?
Are you getting any errors?
If you are doing a registration process, look at your backend – are the accounts created according to your template? are they unique?
From the summary report you can see statistics about your test – does it make sense? (average response time, errors, hits/s)
Once your script is ready:
Clean it up by removing any Debug/Dummy Samplers and deleting your script listeners
If you use the Listeners (such as “Save Responses to a file”) make sure you don’t use any Paths! , rather if it’s a Listener or a CSV Data Set Config – make sure you don’t use the path you have used locally – instead, use only the filename (as if it was on the same folder as your script)
If you are using your own proprietary JAR file(s) be sure to upload it too.
If you are using more than 1 Thread Group (or not the default one) – make sure to set the values before uploading it to BlazeMeter.
Step 3 : BlazeMeter SandBox Testing
If that’s your first test – you should review this article about how to create tests in BlazeMeter.
SandBox it actually any test which has up to 300 users, uses 1 console only and up to 50 minutes.
The configuration for SandBox allows you to test your script and backend and ensure everything works well from BlazeMeter.
To do that, first press the grey button: JMeter Engines I want complete control! – to get full control over your test parameters..
Common issues you may come across:
Firewall – make sure your environment is open to the BlazeMeter CIDR list (which is being updated from time to time) and whitelist them
Make sure all of your test files e.g: CSVs, JAR, JSON, User.properties etc.. are present
Make sure you did not use any paths
If you still having trouble look at the logs for errors (you should be able download the entire log).
A SandBox configuration can be:
Engines: Console only (1 console , 0 engines)
Ramp-up: 20 minutes
Iteration: Test continues forever
Duration: 30-50 minutes
This will allow you to get enough data during your ramp-up period (in case you will get some issues there) and you will be able to analyze the results to insure the script executed as expected.
You should look at the Waterfall / WebDriver tab to see the requests are ok, you shouldn’t get any error at this point (unless that was your intention).
You should watch the Monitoring tab to see how much memory & CPU was used – this should help you with step 4 while you will try to set up the amount of users per engine.
Step 4 : Setup The Amount Of Users-Per-Engine Using 1 Console & 1 Engine
Now, that we are sure the script runs flawlessly in BlazeMeter – we need to figure out how many users we can apply to one engine.
If you can use the SandBox data to determine that, great!
Here, I will give a way to figure this out without looking back on the SandBox test data.
Set your test configuration to:
Number of threads: 500
Ramp-up 40 minutes
Duration: 50 minutes
Use 1 console and 1 engine.
Run the test and monitor your test’s engine (through the Monitoring tab).
If your engine did not reach either a 75% CPU utilization or 85% Memory usage (one time peaks can be ignored) :
Change the amount of threads to 700 and run the test again
Raise the amount of threads until you get either to 1000 threads or 60% CPU or memory usage
If your engine passed the 75% CPU utilization or 85% Memory usage (one time peaks can be ignored :
Look at the point of time you first got to 75% and then see how many users you had at that point.
Run the test again, instead of a ramp-up of 500 put the amount of users you got from the previous test
This time put the ramp-up you want to have in the real test (5-15 minutes is a great start) and set the duration to 50 minutes.
Make sure you don’t go over 75% CPU or 85% memory usage throughout the test…
You can go safer and decrease 10% of the threads per engine just to be on the safe side.
Step 5 : Setup and Test Your Cluster
We now know how many threads we can get from 1 engine, at the end of this step we will know the amount of users 1 Cluster (test) can get us.
A Cluster is logical container which has 1 console (only 1) and 0-14 engines.
Even though you can create a test with more than 14 engines – it actually creates 2 clusters (you can see that the amount of consoles will grow) and clone your test…
The maximum number of 14 engines per console is based on BlazeMeter’s own testing to guarantee that the console can handle the pressure of 14 engines which creates a lot of data to process.
So at this step, we will take the test from step 4, and change only the amount of engines and raise it to 14.
Run the test for the full length of your final test (1,2,3..) hours. While the test is running, go to the monitoring tab and verify:
None of the engines is passing the 75% CPU, 85% Memory limit
Locate your console label (you can find its name if you will go to the Logs Tab -> Network Information and look for the private IP of your console) – it should not reach the 75%CPU or 85% Memory limit.
If you console reached that limit – decrease the amount of engines and run again till the console is within these limits
At the end of this step you know:
The Users per Cluster you will have
The Hits/s per Cluster you will reach
Look for other statistics in the Aggregate Table found under your Load results graph for more information about your Cluster’s throughput.
Step 6 : Use the Master / Slave feature to Reach Your Maximum CC Goal
We’ve got to the final stage.
We know the script is working, we know how many users 1 engine can sustain and we know how many users we can get from 1 Cluster.
Let’s assume these values:
1 engine can have 500 users
The cluster will have 12 engines
Our goal is 50K test
So to do that, we will need to create 50,000 (500*12) = 8.3 clusters..
We could go with 8 clusters of 12 engines (48K) and 1 cluster with 4 engines (the other 2K) – but it would be better to spread the load like this:
Instead of 12 engines per cluster we will use 10, so we will get 10*500 = 5K from each cluster and we will need 10 clusters to reach 50K.
That will help us by:
Not maintaining 2 different test types
We can grow by 5K by simply duplicating an existing cluster (5K is much more common than 6K)
We can always add more if we need to…
We are now ready to create our final Master / Slave test with 50K users:
Changing the name of the test from “My prod test” to “My prod test – slave 1”.
So we go back to our test in step 5 and under the Advanced Test Properties we change it from Standalone to Slave.
Pressing save – we now have the first out of 9 Slaves and 1 Master.
Go back to your “My prod test -slave 1”.
Now repeat steps 1-5 until you create all 9 slaves.
Go back to your “My prod test -salve 9” and press Duplicate.
Change the test name to “My prod test -Master”.
Go to Advanced Test Properties and change it from Slave to Master.
Check all the slaves we’ve just created (My prod test -salve 1..9) and press save.
Your Master-Slave test for 50K users is ready to go. By pressing start on the master you will launch 10 tests (1 master and 9 slaves) with 5K users from each one.
You can change each test (slave or master) to come from a different region, have a different script/csv/other file, use different Network Emulation, different parameters.
The aggregate report of your master and slave will be found in a new tab at the master report called “Master load results” and you could still see each individual test result by simply opening its report.