Load Tests Aren’t Just About Threads, Response Times and Hits/s..
If you often run load tests, you probably have a mental checklist of questions that run through your mind, including:
– How many threads per JMeter engine should I use?
– Can my engine handle 10% more threads?
– Should I start with the maximum number of threads, or add as I go?
All of these questions are important and should be carefully considered – but what about the load test itself? Have you considered how the actual procedure will be managed?
It’s easy to get so wrapped up in the technical details that you lose sight of the overall picture. Even with BlazeMeter’s self-service load testing – which is designed to simplify performance and load testing for developers and QA testers – it’s still vital to take control of the process management to ensure each test runs smoothly.
Why Should I Care?
Load tests require a great deal of cooperation and teamwork. Everyone needs to coordinated in order to quickly analyze the data, identify the bottlenecks and even solve them (if possible) during the test.
For example: the developers/QA testers should create and share the script with all departments involved in the test, to make sure that everyone understands the scenario and how it can affect their particular system. An n-tier application is made up of several disciplines, each one requiring the same level of attention – be it database, front-end, back-end, or monitoring services. It is imperative that the staff understand the upcoming load test scenario, regardless of their knowledge of JMeter and/or load testing, so as to create the most agile environment for the duration of the test. If the staff clearly understand what is happening when a bottleneck is hit, it’s much easier for them to analyze the data and quickly tend to the issue. If they aren’t on par, there’s a much greater chance that they won’t be able to overcome the bottlenecks and delays will occur.
Once the ins and outs of the infrastructure are understood, further managerial tasks are needed. For example: sending out updates and involving your operations team and third parties like Akamai or Amazon to avoid complications such as blocking, shaping, or even inadvertently breaking a legal agreement during the test. While they may seem excessive, it’s really worth taking these supplementary steps before starting the load test to ensure the best results.
Technical and Managerial Perspectives of a Load Test
The Technical Aspects of a Load Test
When it comes to the technical aspect of load testing, as detailed in How to Run a Load Test of 50k+ Concurrent Users, it’s important to supply everyone involved with a test scenario outlining the exact processes of the test. By taking this step, even staff members who are not JMeter experts will understand that during the test, they may experience issues like an excessive number of log-ins, ‘add to cart’ requests, image usages, database queries, and so on. The scenario should be simplified and easy to read for everyone involved. It should include a defined scaling process and explain that the ultimate goal for the number of users may only be met within a number of load tests.
The technical approach usually includes building the script (and its data) to address the specific scenario, testing it locally and verifying that it actually works in JMeter. Once the script performs as expected using the SandBox feature, you should evaluate how many threads can be applied to a single load engine without crashing (ensuring the load engines themselves won’t be the bottleneck). Then, the script will be scaled to a cluster of engines, and more clusters can be added as you go. Learn More
Important points to consider:
1.Does your data need to be unique per engine? If so:
– Set up a different CSV for each engine
– Use the “Split any CSV file to unique files and distribute among the load servers”
– Use JMeter functions like __InstanceID, __RandomString(), __threadNum() to add unique data to your test.
2.Does your scenario have data manipulation in real time? If so:
– Use BlazeMeter’s real time feature
– Control Hits/s with multiple engines using BlazeMeter’s formatter feature
– Use the dynamic Master/Slave configuration
3.Do you use timers? If not, you should (even if it’s just 300ms) because every user has some think time between pages
– You can use the Throughput Shaping Timer or the Constant Throughput Timer to control your test hits/s
Managing Your Load Test
From a management perspective, of course, every company is unique. But most share the same key criteria. These include:
1. Scheduling a meeting before the actual test. Two weeks is generally enough time for this. It allows everyone involved to see the expected script scenario and everyone can discuss issues like who to alert about the test and possible interferences.
2. Including a representative from each department during the initial load tests. This will ensure smoother sailing once the test is up and running. In later tests, you might want to schedule the actual environment you’re going to load test. For example: in a staging environment, the VP of R&D needs to know that the environment is going to be used, which may affect updates to production. Whereas, if load tests are run on production environments, a landing page should be created to notify users that maintenance procedures will take place during that particular time.
3. Creating a drawing or flow chart of the actual script. This is an additional measure but a useful one – after all a single picture says a thousand words. It helps people who aren’t familiar with JMeter or aren’t aware of what’s happening with particular clients or in the front-end to relate and understand what’s expected to happen.
Depending on the test’s complexity, you can make a flowchart of:
1. Your script flow
2. Cache warmup (after doing a reset cache)
3. Test scenario
This method effectively divides the load test into two parts: the pre-load test, which creates all of the data, and the actual load test itself. Appointing a load test supervisor to announce the various stages of the test will help everyone involved to stay on the same page. Utilizing an organized platform, such as Campfire or HipChat will allow everyone to communicate quickly during the load test, as well as enable the supervisor to control and deliver critical assignments when needed. Additionally, these platforms provide a space for each department to present their conclusions from the test. You can also record the test – which will be a great help when it comes to running future tests and reports.