Testing Tools’ Evolution towards Qualitative Analysis Testing Tools’ Evolution towards Qualitative Analysis

Testing Tools’ Evolution towards Qualitative Analysis

by ,
on 26 April 2017


Yesterday, one could launch very simple much talked-about “quick win” tests to gain a few optimization percentage points. Today, the search for a viable hypothesis with an interesting ROI is more complex particularly for mature sites.

The use of analytics AND qualitative analysis tools makes it possible to fully understand a problem: who, what, where, how, how much, where and why ?


Today this second component is still little used. Testing tools have grasped this and have recently started to offer this type of analysis.

VWO and the complete range

Visual Website Optimizer bought the Navilytics solution in 2015, and was the first testing tool to move towards qualitative analysis. The Indian publisher did not beat around the bush and offers this tool box:

  • Recording user sessions
  • Advanced heatmaps (scrollmap, clickmap, etc. per variation)
  • Analysis of forms
  • Analysis of pathways ( a mini analytics)
  • Online questionnaires and feedback

These tools are already available and make it possible to understand user behavior on a variation of identified tests before or after A/B testing. Below is the interface for reading and filtering recordings:

Session recording VWOInterface for Presentation of User Session Recordings and Filter System

The heatmap system now offers a view per click, per mouse movement and per zone. They can be visualized and filtered on the last click (very cool!) and on different devices:


An analytics tool, light yet sometimes useful, makes it possible to follow a user’s pathway performance. It will be useful for the CRO consultant who does not want to wait for an updated tag measurement plan. The analysis of questionnaires is very useful and makes it possible to understand fields that present problems.

The publisher is already known for its infallible test tool and with these latest additions, it will become a favorite weapon for the consultant!

A/B Tasty & Nirror

A/B Tasty is also moving in this direction and in 2016, announced the purchase of the Nirror solution which makes it possible to record user sessions. Here is an interface screen:

Rachat de la solution Nirror pour enregistrer les sessions sur ABTasty par Fauveaux Mathieu.Recording of User Sessions on AB Tasty.

Overall, you’ll find the same functions as on other session recording tools. More practical, an indicator shows the volume of interactions in the video.

It’s missing the ability to filter recordings in order to visualize a specific passage or behavior on a page. The solution is in beta testing and numerous evolutions are expected in the coming months.

Other functions such as the feedback system or heatmap creation are available but once again, they are less advanced than those offered by Visual Website Optimizer.

The suite will change again with an “Insight” section that’s to appear in the coming months. We’ll talk about it again soon. #teaser


Optimizely and Kameleoon do not offer these native functions except heatmaps for the latter. However, they offer the possibility of easily coupling with solutions like Clicktale, SessionCam or CrazyEgg. To note, CrazyEgg just announced the implementation of session recordings! This choice allows for a certain flexibility and, like us, you can use the Lucky Orange or Hotjar solutions that cost between 20 and 100€/month!

It’s impossible to do better in terms of quality per cost. Other solutions such as Content Square are interesting with a dedicated CRO team but the cost is very different.


These evolutions are beneficial for CRO teams which take user behavior into account before and after A/B testing. The lessons have more value in the long run than just the conversion gain. With time, I hope that publishers will offer this type of function at an affordable price.

There are new actors in the field who offer advanced management systems for roadmaps like Liftmap or ExperimentEngine. Tomorrow, these systems will be indispensable to manage personalization roadmaps and large scale testing.

I’ve also heard talk of automatic testing with AI. A way of replacing human analysis for conversion gains. Should I be worried about the future of my career? The floor is open to debate.


, , ,

to read next...

Leave a Reply

Your email address will not be published. Required fields are marked *


Testing Tools’ Evolution towards Qualitative Analysis