Below is my comment to Rockys post. If you have comments to the below lets keep the thread over at Rockys blog.
Rocky,
in the wake of the various blogging presenters have been making their evaluation results public, I have been thinking a little about the evaluation forms at Lotusphere. Of cause I don't know how the other scores look so take the below with a grain of salt...
Although I agree that many of the presentations and the presenters were in fact excellent I think the evaluation form lacks options and that it shines through in the published scores. I really think that question 1 ("How would you rate the quality and relevance of the information in the session/BOF?") and question 3 ("How would you rate the quality and effectiveness of the speaker(s)/facilitator(s)?") lacks a fourth option where one could indicate that the speaker wasn't really doing a good job.
While I recognize that given too many options it is sometimes difficult to choose the correct answer, I still think that only giving people options for positive feedback doesn't really suffice. I know one have the option of commenting on specific sessions and giving ones opinion in prose I still think that it twists the results of the evaluation if I have to give at least a "Fair" evaluation. Sometimes a "Dissapointing" or similar option would be nice. At the end of the day, to get an overall picture of the event, you would have to look at the overall picture of the response and here comments don't really get through unless I, as the one responding, can couple it with a score of "Disapointing" or similar.
Another option would be to go the other way and to restrict the number of options from 3 ("Excellent", "Good", "Fair") to 2 ("Above expectations", "Below expectations"). Only having two options would really make people make a choice. I recognize that it also makes the responses less fine grained.
I'm not really educated in making feedback forms or in statistics but i think it could be worth looking into whether the evalutation forms tell you what you want to know. Given that a person is more likely to publish his/her evalution if they are all good, I think that an overall score of 95% "Excellent" is a problem. The score should probably be as in school where the majority of presenters would receive a "Good" score and a minority receive an "Excellent" score (the famous "Camel"/"Bell" curve). Otherwise it devaluates the "Excellent" score.
Just some thoughts.