Local L&E Overview
The purpose of this wiki-BHC Learning Site is to collect information on local Learning and Evaluation (L&E) efforts at Building Healthy Communities (BHC) sites. This information will allow BHC sites to better understand each other’s work and to identify where to go for self-directed peer learning and exchange.
This wiki-BHC Learning site responds to input heard from local learning staff, Hub Managers, and Programs Managers that additional information is needed on evaluation strategies being used across sites to better connect and to facilitate peer learning.
The wiki site collects, stores, and shares information that can be accessed by local BHC staff for learning purposes. The hope is that local L&E teams will be able to seek out peers at other sites to exchange information on evaluation design, evaluation challenges and lessons. By facilitating this type of exchange, it is hoped that sites build evaluation capacity through peer sharing and that the wiki-site provides added value to local L&E teams and to respective BHC sites.
Directions: Local learning and evaluation staff are welcome to complete this tool. Please share information with your Hub Manager and Program Manager prior to submission. Enter your responses into the blank spaces of the below tables. It is essentially a “fill-in-the-blank” form.
For all questions with pre-defined response options, you may mark “X” in the cell that corresponds to your response or you may type additional details in the space provided. For all open-ended questions please provide as much explanation as you feel another local L&E staff member, Program Manager, or Hub Manager would need to grasp where your site is at. Please note that answers to questions 1, 2, and 3 are required and other questions are optional.
To fill out this form, log in to the site and click on the “Edit Page” icon, which looks like a pencil, in the upper right corner of this page. Then click on the information in the box you wish to edit and make any needed changes. Click “save” when finished. Site | Long Beach | Person Completing Tool | Parichart Sabado Evaluation Manager/BHCLB L&E Team Member | Email Address | parichart.sabado@csulb.edu | Phone Number | 562-985-2180 | Date Tool Completed | February 26, 2016
|
|
1. Do you have an L&E team at your site? | Yes | Yes | No | |
2. Briefly describe the current L&E structure at your site. | | BHCLB’s
L&E team is comprised of BHCLB Program Manager, BHCLB Hub Manager, and staff
at the CSULB Center for Health Equity Research (CHER), Laura D’Anna (Director
of CHER) and Parichart Sabado (Evaluation Manager). Evaluation activities are informed by an
L&E Committee that is comprised of BHCLB partner organizations and a
consultant who serves as the evaluator for several TCE grantees. A meeting with the L&E Committee is
convened when input is needed on specific evaluation activities. Most recently, the L&E Committee provided
feedback on a set of questions to be used for interviews with key stakeholders
and helped to identify who should be included on the list of key
stakeholders. The L&E team meets
bi-weekly to share updates on data collection and emerging or ongoing projects
that need L&E support. |
|
|
3. Describe your site’s current L&E approach, including your L&E design and measurement focus should it be developed at this time. | BHCLB’s L&E approach follows
TCE’s learning framework, where evaluation activities are developed with input
from the BHCLB partners and community residents to build research and
evaluation capacity, and results are shared with the larger collaborative to help
inform strategic decision making and foster adaptive change. |
|
BHCLB’s L&E team takes a mixed
methods, social ecological approach with regard to measurement. Data collection activities fall into the
individual, group, and population level.
Activities at the individual level include the Resident Power Survey, a
local tool that was developed to measure resident engagement, leadership
skills, and ability to influence change.
At the group level, evaluation activities include cross-site tools such
as the Collaboration Assessment Tool, Resident Driven Organizing Inventory, and
the Policy Advocacy Inventory. The Workgroup Observation Form is a local
tool that was developed to also collect data at the group level. The purpose of this form is to standardize qualitative
data collection as it ensures that those who complete it observe the same key
elements (e.g., opportunities for capacity-building, relationship-building,
conflict resolution, etc.). Qualitative
data from the Workgroup Observation Form will be used to supplement the
Collaboration Assessment Tool, as they are observations of workgroup dynamic
and member interaction from an outside participant. At the population level, data
collection activities include tracking BHCLB’s impact using health indicators
and interviewing key stakeholders to learn about the community’s perspective of
BHCLB and its effort to address health disparities in the Central/West Long
Beach area. More specifically, the
purpose of the interviews is to learn whether or not BHCLB has changed the
narrative around health and to solicit feedback on areas that are lacking from the
initiative. |
4. Does your L&E team attend BHC-related workgroup/action team meetings, community events, etc.? | Yes | X | No | | We don’t currently have an L&E team. | |
5. Do you have an L&E team member assigned to each BHC workgroup/action team to track ongoing work? | Yes | X | No | | We don’t currently have an L&E team. | |
6. How often does the L&E team communicate with the Program Manager at your site? Please mark all that apply. | Weekly | | Bi-weekly | X | Monthly | | Quarterly | | Bi-annually | | Informally | | Other | |
7. How often does the L&E team communicate with the Hub Manager at your site? Please mark all that apply. | Weekly | | Bi-weekly | X | Monthly | | Quarterly | | Bi-annually | | Informally | | Other | |
Please indicate your level of agreement with the following four statements. Statement | Strongly disagree | Disagree | Neither agree nor disagree | Agree | Strongly agree | 8. Our L&E team has been engaged in organizational development efforts at our site. | | | | | X | 9. Our site’s evaluation design has the flexibility to adapt to changes in the policy environment and our related work as needed (e.g. benchmarks and indicators can be modified as the project evolves). | | | X | | | 10. Our site has tools and methods for evaluating the unique and dynamic nature of policy work. | | | | | X | 11. Our site has regularly updated and/or revisited the original logic model and community action plan. | | | | | X |
12. If your site has developed reports related to learning and evaluation efforts, please list the title of each report below and upload each document to the Learning Document Library at (https://sites.google.com/site/tcelearningevaluation/products-services). Report title | Report uploaded to library? (Y/N) | | | | | | | | |
13. What evaluation tools/assessment tools are currently being administered at your site? Please list all tools that you have used within the last 12 months in the below table and upload each tool to the Learning Document Library at (https://sites.google.com/site/tcelearningevaluation/products-services). Name of tool | Purpose of tool | How were findings used or applied? | Report uploaded to library? (Y/N) | Resident Power Survey | | To
measure resident engagement, leadership skills, and ability to influence
change. |
|
| | Findings
were shared with Hub staff and the larger collaborative to highlight areas that
need improvement (i.e., areas that were rated fairly low by residents). The
survey was administered in 2013 and re-administered in 2015. Data from 2015 have not yet been analyzed, but
we expect to see an improvement. |
|
| No, but available upon request | | Workgroup
Observation Form |
|
| | To
standardize qualitative data collection. |
|
| | Data
will be used to supplement the Collaboration Assessment Tool. |
|
| No, but available upon request | | Health
Indicators & Selection Criteria |
|
| | To
track population level changes. |
|
| | A
comparison of baseline (2009) and follow-up (2020) data will inform the
evaluation of BHCLB’s impact in the target area. |
|
| No, but available upon request | | | | |
14. Please describe how you currently and regularly engage BHC partner agencies/orgs/grantees in evaluation efforts. If you do not, please share barriers to engaging this stakeholder group. | The process of selecting health
indicators involved interviews with key stakeholders working in each of the
target areas listed in the Community Action Plan (e.g., school climate, quality
housing, mobility & safe passage, food access, etc.). A preliminary list of health indicators was
reviewed by each of the workgroups and ratified by the L&E Committee. |
|
With regard to data collection activities,
partner organizations/grantees provide insight on how best to administer data
collection tools to their resident pool.
Once data are collected, they are shared with those who were involved
with data collection and with the larger collaborative to help inform their
work. As mentioned previously,
evaluation activities are also informed by the L&E Committee, which is
mainly comprised of BHCLB partner organizations that are working on the target
areas listed in the Community Action Plan and, therefore, are most familiar with
the initiative and its mission and values. |
15. Please describe how you currently and regularly engage residents in evaluation efforts. If you do not, please share barriers to engaging this stakeholder group. | | The
Resident Power Survey was developed with input from community residents,
particularly when defining resident power and determining its three core
elements (i.e., engagement, leadership, influence). Residents are also engaged in BHCLB Learning
activities, breakfast/book club/brown bag discussions that foster dialogue on
thought-provoking issues related to BHCLB target areas. |
|
|
16. Please briefly describe how you may be building evaluation capacity within your site. | | BHCLB’s
L&E team serves as a resource for partner organizations and to date, a
number of partners have asked CHER staff to help conceptualize an evaluation
study, for referral to specific data, and to serve as a member of a research
team to help inform campaign work. BHCLB’s L&E team has also held workshops to
build research and evaluation capacity among partner organizations and
grantees. These workshops include the Evaluating
for Change training, which was held at the BHCLB Hub Office in collaboration
with the TCC group. |
|
|
17. What evaluation models or frameworks have been useful for your site? (Example FSG Collective Impact Framework, Community Action Plan) | Evaluation model or framework | How have you used it? |
| | As
a feedback loop, where BHCLB key stakeholders are part of evaluation design and
data collection, and results are shared back to help inform their work. |
|
| | | To
ensure that evaluation activities involve, and results reflect, all levels of
the community (i.e., individual, group, population). |
|
| | | | |
18. What stage is your L&E team at with respect to the following evaluation components? Please mark the box underneath the number that best matches your assessment. | | 1 Forming | 2 | 3 Developing | 4 | 5 Mature | Planning & evaluation design | | | | | X | Implementation of the evaluation design | | | | X | | Using data for learning & continuous improvement | | | | X | | Sharing out evaluation results with diverse audiences | | | | X | |
19. Please describe an evaluation-related challenge that your site is currently working to resolve. Please briefly describe the problem, your approach to finding a solution, and any additional resources or expertise that would help your site. | One challenge relates to the
Collaboration Assessment Tool. One BHCLB workgroup recently completed the
assessment and it was very challenging to get members to complete the online
survey. The assessment for this
workgroup was initiated in October 2015 and finally completed in February 2016,
with two members completing the survey during the last month. The workgroup was given the option of
completing the assessment on paper at a regular meeting, but members chose to complete
the assessment online, as the meeting agenda is always full. However, while the online survey may be more
a convenient option for workgroup members, their busy schedules were a major
barrier to survey completion. Moving forward,
groups will be encouraged to take the assessment during a regular meeting and
the assessment will be tailored to meet the needs and/or characteristics of the
workgroup. |
|
Another challenge is related to administration
of the Resident Power Survey in 2013. We
initially planned for the survey to be taken online to minimize data entry, but
quickly learned that many residents do not have computer access, even though
they may have email addresses. Ultimately, a member of the L&E team had to
meet with residents individually at local coffee shops to administer the survey
using a computer tablet. Paper surveys
were also available; however, they were only administered to Khmer residents,
who preferred to complete the paper surveys in a group setting.
Reluctance to complete the Resident
Power Survey in 2013 may be due in part to the length of the survey, which
required approximately one hour to complete.
Residents received a $10 gift card to Target as an incentive, but the
amount may have been too low for the amount of time required. Due to the challenges encountered and the
findings from the first round of survey administration, CHER revised the survey
to remove questions that did not generate meaningful data and by doing so,
survey length was shortened by almost 50%.
The incentive for participation was also increased to a $20 gift card. Lastly, the second round of surveys was
administered using paper surveys rather than the web-based survey. A member of the L&E team visited 15
partner meetings between July and November 2015 and collected a total of 211
surveys, up from 77 surveys collected in 2013.
While residents preferred paper surveys, there are
two major challenges related this method of data collection. First, paper surveys increase the likelihood
of missing data due to non-response or incorrect responses (e.g., selecting
more than the indicated number of responses is treated as a missing value). Second, data entry by hand, rather than
directly via a web-based survey, increases the likelihood of key stroke errors.
To address the first challenge, we learned that residents need to be given very
clear instructions on how to respond to some of the questions and even still,
responses need to be reviewed when residents submit completed surveys. Any discrepancies can be resolved with
residents while they are still present.
A verification process has been implemented to address the second
challenge. This process requires three
individuals; one person to code all responses, one person to enter data using
these codes, and one person to verify that data are entered correctly. |
20. Is there anything else you would like to share? | Not at this time. |
|
|