PROGRESS!
I extracted data from the feedback people sent, using the small survey at the end of each level. Pic related, the closer the values are to 100%, the better.
(it's for my game God is a Cube, which is a puzzle game with currently 51 levels in the alpha youtube.com/watch?v=3VrRv0nVz-g )
I'm gonna give a few explanations to explain why surveying tools (by asking directly people) are great to make your game better.
Checking the data from the last 6 months, it looks like people really like the levels, except for a few levels too difficult (high red spike) or too easy (low red spike). I created a "satisfaction" indicator (bottom chart) by averaging the fun, interest and right difficulty ratings for each level. The overall satisfaction is really good with an average of 87%.
As you can see, the first 21 levels have less good values, mainly due to the fact that developers find those levels boring.
But I added a panel at the start of the game asking people who already know programming to start at level 22. The next version should have better values, by letting only beginners play this part.
Why the 3 curves? It's really important to have more than one factor to judge the gamers satisfaction.
By example, Level 31 is really difficult (123% difficulty) but people find it interesting (100%) and fun (97%).
On the other hand, Level 41 is even more difficult (135% difficulty) and people find it as interesting (100%), but far less fun (80%). Typically, that's the kind of level where I will add a few runes (the solution will be partly completed) so people can more easily find the solution. It should make the difficulty easier and the fun better, while not impacting the interest rating.
On the values themselves, the interest and fun ratings were a simple boolean (fun / not fun) with average values ranging between 0% (not fun) and 100% (fun). Because it wouldn't bear sense to ask people if a level is "too much interesting" or "too much fun".
On the other hand, the difficulty rating needs to check if levels are too easy or too hard, so there are three values (0, 1 and 2) ranging between 0% (no difficulty at all) and 200% (insanely difficult).
The satisfaction is calculated by taking an average between fun (0-100%), interest (0-100%) and how far difficulty is from 100%. Which means that a 150% difficulty (too hard) will be considered as bad as a 50% difficulty (too easy).
On the values themselves, the interest and fun ratings were a simple boolean (fun / not fun) with average values ranging between 0% (not fun) and 100% (fun). Because it wouldn't bear sense to ask people if a level is "too much interesting" or "too much fun".
On the other hand, the difficulty rating needs to check if levels are too easy or too hard, so there are three values (0, 1 and 2) ranging between 0% (no difficulty at all) and 200% (insanely difficult).
The satisfaction is calculated by taking an average between fun (0-100%), interest (0-100%) and how far difficulty is from 100%. Which means that a 150% difficulty (too hard) will be considered as bad as a 50% difficulty (too easy).
Here is the formulae:
((100 - ABS(difficulty-100)) + interest + fun) / 3
Examples:
- difficulty 50, interest 98, fun 99
( (100 - ABS(50-100)) + 98 + 99 ) / 3
= ( (100 - ABS(-50)) + 98 + 99 ) / 3
= ( (100 - 50) + 98 + 99 ) / 3
= 82.33% satisfaction
- difficulty 125, interest 98, fun 99
( (100 - ABS(125-100)) + 98 + 99 ) / 3
= ( (100 - ABS(25)) + 98 + 99 ) / 3
= ( (100 - 25) + 98 + 99 ) / 3
= 90.66% satisfaction
In conclusion, when working on a game, rather than just asking "do you like it?", it's far much better to ask several questions, especially because you want to know what could be deceiving for your players.
This kind of survey allows you to find what is wrong in a level or a part of your game (the difficulty? the lack of fun? no interest?), while still allowing you to have a way to know if people are globally satisfied.
Note: I would like to say that putting a survey at the end of each level is an idea I took from Infinifactory (by the guy who made SpaceChem), with its author's blessing.