1.
The evaluation design is both quantitative and qualitative.
Qualitative data will be acquired in a participatory evaluation.
Implementation of the evaluation will be based on lessons learned from
the Communities In Schools and Sharing Our Future participatory evaluations.
The purpose of the participatory evaluation is to improve the
implementation of this project and to provide information for others who
might want to replicate this process.
2.
The projected evaluation committee includes the director of the Community
Education Network; the facilitator of the Zap Survey; the executive director of the Long Range Regional Economic Development Board; the OST/Search Design facilitator, and the community forum facilitator.
Reporting and facilitating the evaluation is the responsibility of the
facilitators. However, as a participatory evaluation reflects the
observations and opinions of all stakeholders, each subregional community committee also has
responsibility for implementation.
As in the CIS and SOF evaluation, we use video for data collection so our
resource requirements are videotapes (as well as ENG and editing
equipment supplied inkind by Ryakuga).
3.
During the CIS evaluation, it was found that the video data collection
process surfaced issues which did not arise in the normal reporting
process. It was possible to respond immediately with the implementation
of positive change in the process. The evaluation committee, communicating in the "virtual office" (see appendix 1), references the data collected to the established goals,
objectives and success indicators of the project.
4. Other committees informed by the evaluation process include the steering committee and the advisory (design team) group. The steering committee includes the project community development professionals working in the region - the zone board field officers; the director of the Community
Education Network; the executive director of the Long Range Regional Economic
Development Board; the OST/Search Design facilitator, and the community forum facilitator. The advisory (design team) group is a component of the virtual office or online project management. It includes the evaluation committee and is open to interested professionals working in rural development across the country. The advisory (design team) group will be asked to make suggestions pertinent both to the implementation and analysis of the evaluation.
5. The Zap Surveys designed and implemented by Dr. Ivan Emke are not specifically part of the evaluation but they will serve an evaluative function. The survey will be implemented during and immediately after the event.
The survey would have three major components, but each is dealt with briefly: (i) subjects' prior knowledge of and attitude toward the issue; (ii) subjects' exposure to the event (how much they watched/listened to, how they heard about it, whether they liked it, etc.); (iii) subjects' learnings from the event (what they might do differently, whether attitudes were changed).
The survey will be designed in consultation with the evaluation committee and the advisory group.
6. The steering committee will distribute the CRP Stakeholders Questionnaire. The possibility of a questionnaire specific to each subregion will be discussed by the various committees and college student work placements.
7. Survey questions related to the design of a search conference and a qualitative evaluation of the process is crucial. If so directed, the sponsor, the Community Education Network, will attempt to implement a full community search conference whereby the community will determine its strategic plan and commit to its implementation.
8. Success Indicators
A. By March, 2003, the project deliverables will be successfully implemented.
(a) By January 31, 2003, there will be three face to face meetings between the fieldworkers and the local committees.
(b) By January 20, 2003, the community forum facilitator will prepare a prototype collaboration agreement; proposed work plan; virtual office for reporting and networking; prototype community forum manual, and a prototype website.
(c) By January 31, 2003, the community forum facilitator will prepare for and facilitate a meeting with the three local committees.(d) By January 31, 2003, the local committees will sign collaboration agreements; negotiate work plans, and review prototype community forum manuals.
(d) By February 8, 2003, the search conference design facilitator will prepare for and facilitate a meeting with the three local committees.
(e) By January 31, 2003, LRREDB economic development officers and the local committees negotiate an agreement to host a cultural communication event.
(f) By February 15, 2003, cultural events in three communities are broadcast on community radio and webcast for
other communities and expatriates while local people are trained in use of equipment. The participatory evaluation process, utilizing video as a data collection tool, will begin the day after the event.
(g) By February 28, 2003, the community forum facilitator prepares/delivers community forum promotion/implementation workshops for each of the three local committees.The workshops include the designation of time/place/topic of the next community forum.
(h) By February 28, 2003, the search conference design facilitator prepares/delivers a design workshop for all participants including three local committees; partners and LRREDB economic development officers.
(i) By February 28, 2003, a Zap survey will be designed and implemented to evaluate the reaction of the participants to
facilitating a community search conference. Hard and electronic report copies will be available within 30 days of the end of the project.
(j) By March 10, 2003, the community forum facilitator, assisted by the economic development officers and the local committees,
prepares/promotes an issue based community forum in each of the three subregions.
(k) By March 22, 2003, community forums, designed, promoted and produced by the community forum facilitator, with assistance from local committees and the economic development officers are broadcast on community radio and webcast for other communities and expatriates. Local people are trained in use of equipment.
(l) By March 31, 2003, the local committees, assisted by the economic development officers, prepare/promote an issue based community forum in each of the three subregions.
B More Success Indicators
Other appropriate methods and/or indicators may become apparent during the course of the project, and may also be applied to enhance the evaluation of the project.(see appendix 2)
9.
Quantitative data will be presented in the monthly reports of the facilitators, economic development officers and the committees.
Facilitators' reports will include the implementation of the Enlarging the Circle work plan; number of volunteers on the committees and working at the forums; number of interactions at community forums (and issues raised); participation and topics raised at meetings of the steering committee, evaluation committee and advisory group, and numbers of volunteers at the workshops. Economic development officers will report on meetings of local committees. Another data source are the community forum program logs. Finally each local committee will update the community communications needs, skills and resources assessment maps.
Further quantitative data (number of interactions, issues discussed) will be available from the web board set up for networking and project management. The board is moderated by Ryakuga with results included in a monthly report. Other data which may be extrapolated from the report include number of meetings of the overall steering committee and the evaluation committee; number of video conferences with the advisory group;
community forum/webcasts; training events, and community visits.
The actual quantitative requirements are described in the workplan. But it would seem reasonable to project - the community forum facilitator
will visit each subregion four times; three community forums will be held in each subregion; the OST/search design facilitator will visit the community subregions once and facilitate a weekend long design workshop; the local committees and the economic development officers will meet twice a month and the overall steering committee will meet twice a month; facilitators and the consultant will report monthly; the evaluation committee will meet each month; there will be a videoconference each week; facilitators and committees will post to the web board each week, and there will be a community forum
training event in each subregion plus skills training for the volunteers.
Qualitative data is collected on videotape during interviews with the stakeholders. In this project, potential stakeholders include the local committees; volunteers including youth; members of the steering, evaluation and advisory committees; and the community at large.Specifically the steering committee and the advisory group will be asked to comment on the effectiveness of the communication tools of the virtual office. The local committees and public at large will be questioned on the effect of the community forums.Economic development officers and the local committees will be asked to comment on the degree of collaboration between the community development professionals and the community groups.
The interviews are recorded on digital videotape and dubbed to to VHS with visible timecode for analysis by facilitators and
the evaluation committee.
10.
During the CIS evaluation process, the interviews surfaced information which allowed coordinators to make immediate changes to the
implementation of the program in the community. In the SOF evaluation process, it was the responsibility of the evaluation committee
to meet monthly to analyse the data with reference to the success indicators of the project. In the Enlarging the Circle process, the facilitators will also analyze the data.
The standard evaluation questions of - What worked best for you? What should be done differently? What advice would you have for
others wishing to implement a similar program? - produces information which demands adjustments to be made.
11.
The evaluation is designed to allow for continuous and incremental change in the program.
This participatory, grassroots evaluation process typically includes two reports - in writing and on video. The video report in itself
is a useful method of letting others see how the project is viewed by its stakeholders. It becomes a potential tool for replication.
The evaluation reports and the Zap Survey are due within 30 days of the completion of the project.
The evaluation begins with the development of goals,
objectives and success indicators for the program.
(m) By March 31, 2003, community forums, designed, promoted and produced by local committees are broadcast on community radio and webcast for other communities and expatriates.