In the previous DIY assessment posts, we talked about what information sources to use and what considerations to take into account when planning your process assessment project. Now it is the time to actually put together the instruments that can be used to gather the data and do the assessment. For the data gathering phase, there are three major activity areas to cover and manage:
- Survey Management
Individual surveys and questionnaires, either in paper or electronic format, are often used to collect assessment data. Compared to the workshop or interview methods, surveys are also generally less expensive to administer and can help in collecting sensitive data. The challenge for the survey method is that the questions need to be precise with little room for personal interpretation in order to ensure accuracy and usefulness of the data. Due to the difficulty of crafting precise questions for all situations, the survey is less suitable for questions that are complex and open-end by nature, though free-form comments can still be collected via a survey.
In managing the surveys, here are some general steps and sequence to consider:
- Determine the survey mechanism, paper, electronic, or both.
- Select the processes to assess and the questions to ask.
- Identify the potential participants by process and collect their contact information to where the survey should go.
- Send out the survey communication with the timeline, login credentials, and other survey or assessment related information.
- Send out the surveys and monitor the completion progress. Send reminders to participants about completing the survey if necessary.
- Close the survey at the deadline.
- Follow up with the survey participants for further clarification, if necessary.
- Compile the survey results and provide the results to the lead assessor. The survey results can also be used as the input into the follow-up workshop and/or interview stages, if planned that way.
- Workshop Management
The workshops usually get several or more individuals together, so the assessor can interview the participants together. The workshops can be a time-saver, compared to the one-on-one interviews. The workshops sometimes can also spur more discussions that yield good insights and useful information. At the same time, the workshop setting can also limit the sharing of some sensitive information due to participants being careful of revealing confidential data in front of other participants. During the workshops, I would recommend having someone from the assessment team served as the scribe. The scribe allows the assessor to concentrate on facilitating the workshop and the interaction between participants, without the distraction of having to write down the conversations as they take place during the workshop.
A workshop can be delivered using a number of formats, but it usually includes the following elements:
- Workshop kick-off. Participant introductions. Review of agenda and schedule.
- Review of the assessment project, scope, and goals.
- Review of the process in question, scope, and definitions.
- Further data gathering with more in-depth interviews within the workshop.
- Review of findings and discussion of the next steps
- Interview Management
The one-on-one interviews can be the most labor-intensive but also have the most flexibility in collecting assessment data. It is also the best mechanism to handle complex questions and responses. A good deal of interview management is calendar, schedule, and time management. Scheduling meetings well in advance with timely reminders should go a long way to help the participants in sticking with the interview appointments. Similar to the workshops, I would also recommend having a scribe, so the assessor can focus her attention on asking the questions and assessing the quality of the answers from the participants.
The types of question asked in an interview typically include:
- Open-ended questions where the participants can share descriptive information without a lot of structure.
- Closed-ended questions where the participants answer with short responses like “yes,” “no,” or some others.
When asking a question, the interviewer should always frame the question first by stating the topic or background for the question. The interviewer will ask further probing questions, if necessary, to obtain the most complete answers for the question. When framing the questions, it is also important that the framing comments do not evolve into a leading question, where the participants feel compelled to answer the question a particular way based on how the question was asked to begin with.
Depending on the complexity and timing considerations of the assessment project, most organizations will likely take a blended approach of using all three methods for data gathering. Throughout the data gathering phase, it is important for the assessor and the scribe to maintain objectivity and not projecting their views or bias over the conversations or interviews. The questions on the surveys or interviews should always be clear and precise, so they minimize personal interpretation that could lead to biased answers. The scope assigned to the participants should also be reasonable but still comprehensive enough so the most needed answers can be collected. On the next post, we will discuss a process survey example using the problem management process as the part of the assessment scope. The example will illustrate what defines various levels of maturity within problem management and what questions to ask in the survey or interview.