Criteria | Guiding questions | Check - | Check | Check + |
Breadth of idea generation (40%) | Does your team generate lots and lots of diverse possible solutions to your problem? | The team generated the minimal number of ideas. They are obscure and repetitive. | The idea generation resulted in a broad set of possible solutions, making them amenable to synthesis methods. The ideas are understandable and unique. | The team produced many creative and diverse possible solutions. The ideas are thoughtful, diverse, and different than existing market solutions. |
Effectiveness of online idea generation (30%) | Did your team create a clear task for workers on Mechanical Turk to generate ideas for your problem? Did you systematically read and rate each idea? | The team did produce no results from Mechanical Turk or the task was so poorly framed that it produced only noise. | The team launched a task on Mechanical Turk and got a range of interesting results. The ideas were rated and sorted and added to the overall list of concepts. | The team's Mechanical Turk task helped workers produce lots of creative and practical concepts. The task clearly explained the problem and prompted workers with a generative question. |
Clarity of idea synthesis and six chosen ideas (30%) | Did your team use a formal synthesis technique to sort and analyze ideas? Did you synthesis process result in new ideas? Do the final six chosen ideas hold promise? | The team did not systematically analyze the space of solutions. The chosen concepts have a low probably of success. | The team systematically analyzed their ideas and chose concepts that have a high probably of success. | The team analyzed their ideas using one or multiple synthesis techniques. Concepts show promise and can make a big impact. |