[110] in the Methods You Don’t Use Yet series
Expert Roundtable Review
Problem: For a product or service inquiry, we often see the need to rapidly gather highly relevant feedback and informed opinions on a new concept. A similar problem is noted when a project team is identifying the opportunities for innovation and must conduct a rapid but deep scan of the current products or competitors in the field. When the problem necessitates access to highly informed professionals, the focus group, field study, or survey methods are not feasible, at least within the timeframe necessary to make an initial business decision.
Solution: The Expert Roundtable Review, based on the Heuristic Evaluation [35] and Videoconference Focus Group [42], provides a dialogic tool for engaging a small group of informed participants online and by teleconference at a time convenient to their schedule.
Use When: Useful for collecting responses, opinions, and insights from busy professionals and respondents that may be difficult to recruit for focus groups, interviews, or even surveys. Also useful as a technique for convening an online focus group supported in advance by an evaluation or ideation activity explored within the focus group.
Use for: Product opportunity appraisal, Early concept design, Project planning and business case phase. Roundtables can be phased in sequence, starting with competitive analysis, then early concept testing, followed by prototype evaluation (with different samples recruited from the same population for each of these phased activities).
Process: The researcher identifies the need for the session based on the customer/user segments needed for the study. The participants can be recruited and scheduled from 10 – 14 days before the roundtable session. Incentives paid to participants are commensurate with position and expectations.
Typical steps involve:
- Identify candidate participants and their contact information, and circulate invitations with 2-3 date/time options. (If two sessions are needed, this is a way to determine the best allocation.)
- Prepare a draft evaluation worksheet for use in the homework, the advance task given to participants that drives the moderated discussion. Refine the worksheet to provide a document allowing participants to review and evaluate (typically) 2 of 3 or 4 options in some depth. These are often online websites or services that would be known to or of interest to the participants.
- Refine the heuristics in the worksheet (questions based on categorical criteria) so that each option is interrogated by 12-20 questions in 4-5 categories. Review and refine the draft worksheet with members of the service design team so that it reflects consensus on priorities.
- When candidates respond to preferred dates, confirm their appointments and provide the blank worksheet with instructions for advance preparation before the teleconference.
- Remind participants one day before the call and ask for worksheets in advance of the call.
- Moderate the teleconference with 4-7 participants (and with invited clients and observers), using the worksheet responses as a guide to the discussion. Record and transcribe the discussion.
- Prepare a summary report by compiling the worksheets into a composite review report, showing participant priorities in summaries based on the information collected. Some researcher commentary on the report helps contextualize the meaning for readers.
I have used this method dozens of times for situations where we convene experts in a domain to quickly review a variety of current resources to evaluate their fit to practice and to strategy. We often start with competitive analysis sessions, and as we learn more, build prototypes and review those head-to-head with the competition. After each set of sessions, a summary report is provided along with a discussion of the findings and value gained from the scan.
