Long Beach

Local L&E Overview 

The purpose of this wiki-BHC Learning Site is to collect information on local Learning and Evaluation (L&E) efforts at Building Healthy Communities (BHC) sites.  This information will allow BHC sites to better understand each other’s work and to identify where to go for self-directed peer learning and exchange.

This wiki-BHC Learning site responds to input heard from local learning staff, Hub Managers, and Programs Managers that additional information is needed on evaluation strategies being used across sites to better connect and to facilitate peer learning.  

The wiki site collects, stores, and shares information that can be accessed by local BHC staff for learning purposes.  The hope is that local L&E teams will be able to seek out peers at other sites to exchange information on evaluation design, evaluation challenges and lessons.  By facilitating this type of exchange, it is hoped that sites build evaluation capacity through peer sharing and that the wiki-site provides added value to local L&E teams and to respective BHC sites.

Directions:  Local learning and evaluation staff are welcome to complete this tool.  Please share information with your Hub Manager and Program Manager prior to submission.  Enter your responses into the blank spaces of the below tables.  It is essentially a “fill-in-the-blank” form.  

For all questions with pre-defined response options, you may mark “X” in the cell that corresponds to your response or you may type additional details in the space provided.  For all open-ended questions please provide as much explanation as you feel another local L&E staff member, Program Manager, or Hub Manager would need to grasp where your site is at. Please note that answers to questions 1, 2, and 3 are required and other questions are optional.

To fill out this form, log in to the site and click on the “Edit Page” icon, which looks like a pencil, in the upper right corner of this page. Then click on the information in the box you wish to edit and make any needed changes. Click “save” when finished.



Site

 Long Beach    

Person Completing Tool

 Parichart Sabado

Evaluation Manager/BHCLB L&E Team Member

Email Address

parichart.sabado@csulb.edu 

Phone Number

 562-985-2180

Date Tool Completed

 February 26, 2016



  

1.  Do you have an L&E team at your site?

Yes

 Yes

No

 

 

2.  Briefly describe the current L&E structure at your site.

 

BHCLB’s L&E team is comprised of BHCLB Program Manager, BHCLB Hub Manager, and staff at the CSULB Center for Health Equity Research (CHER), Laura D’Anna (Director of CHER) and Parichart Sabado (Evaluation Manager).  Evaluation activities are informed by an L&E Committee that is comprised of BHCLB partner organizations and a consultant who serves as the evaluator for several TCE grantees.  A meeting with the L&E Committee is convened when input is needed on specific evaluation activities.  Most recently, the L&E Committee provided feedback on a set of questions to be used for interviews with key stakeholders and helped to identify who should be included on the list of key stakeholders.  The L&E team meets bi-weekly to share updates on data collection and emerging or ongoing projects that need L&E support.   
 

 

3.  Describe your site’s current L&E approach, including your L&E design and measurement focus should it be developed at this time.

 

BHCLB’s L&E approach follows TCE’s learning framework, where evaluation activities are developed with input from the BHCLB partners and community residents to build research and evaluation capacity, and results are shared with the larger collaborative to help inform strategic decision making and foster adaptive change. 

BHCLB’s L&E team takes a mixed methods, social ecological approach with regard to measurement.  Data collection activities fall into the individual, group, and population level.  Activities at the individual level include the Resident Power Survey, a local tool that was developed to measure resident engagement, leadership skills, and ability to influence change.  At the group level, evaluation activities include cross-site tools such as the Collaboration Assessment Tool, Resident Driven Organizing Inventory, and the Policy Advocacy Inventory.  The Workgroup Observation Form is a local tool that was developed to also collect data at the group level.  The purpose of this form is to standardize qualitative data collection as it ensures that those who complete it observe the same key elements (e.g., opportunities for capacity-building, relationship-building, conflict resolution, etc.).  Qualitative data from the Workgroup Observation Form will be used to supplement the Collaboration Assessment Tool, as they are observations of workgroup dynamic and member interaction from an outside participant. At the population level, data collection activities include tracking BHCLB’s impact using health indicators and interviewing key stakeholders to learn about the community’s perspective of BHCLB and its effort to address health disparities in the Central/West Long Beach area.  More specifically, the purpose of the interviews is to learn whether or not BHCLB has changed the narrative around health and to solicit feedback on areas that are lacking from the initiative.

 

4.  Does your L&E team attend BHC-related workgroup/action team meetings, community events, etc.?

Yes

 X

No

 

We don’t currently have an L&E team.

 


5.  Do you have an L&E team member assigned to each BHC workgroup/action team to track ongoing work?

Yes

 X

No

 

We don’t currently have an L&E team.

 

 

6.  How often does the L&E team communicate with the Program Manager at your site?  Please mark all that apply.

Weekly

 

Bi-weekly

 X

Monthly

 

Quarterly

 

Bi-annually

 

Informally

 

Other

 


7.  How often does the L&E team communicate with the Hub Manager at your site?   Please mark all that apply.

Weekly

 

Bi-weekly

 X

Monthly

 

Quarterly

 

Bi-annually

 

Informally

 

Other

 

 

Please indicate your level of agreement with the following four statements.

Statement

Strongly disagree

Disagree

Neither agree nor disagree

Agree

Strongly agree

8.  Our L&E team has been engaged in organizational development efforts at our site.

 

 

 

 

 X

9.  Our site’s evaluation design has the flexibility to adapt to changes in the policy environment and our related work as needed (e.g. benchmarks and indicators can be modified as the project evolves).

 

 

 X

 

 

10.  Our site has tools and methods for evaluating the unique and dynamic nature of policy work.

 

 

 

 

 X

11.  Our site has regularly updated and/or revisited the original logic model and community action plan.

 

 

 

 

 X

 

12.  If your site has developed reports related to learning and evaluation efforts, please list the title of each report below and upload each document to the Learning Document Library at (https://sites.google.com/site/tcelearningevaluation/products-services).

 

Report title

Report uploaded to library? (Y/N)

 

 

 

 

 

 

 

 

 

13.  What evaluation tools/assessment tools are currently being administered at your site?  Please list all tools that you have used within the last 12 months in the below table and upload each tool to the Learning Document Library at (https://sites.google.com/site/tcelearningevaluation/products-services).

 

Name of tool

Purpose of tool

How were findings used or applied?

Report uploaded to library? (Y/N)

 Resident Power Survey

To measure resident engagement, leadership skills, and ability to influence change.

 

 

Findings were shared with Hub staff and the larger collaborative to highlight areas that need improvement (i.e., areas that were rated fairly low by residents). The survey was administered in 2013 and re-administered in 2015.  Data from 2015 have not yet been analyzed, but we expect to see an improvement.

 No, but available upon request

 

Workgroup Observation Form

 

To standardize qualitative data collection.

 

Data will be used to supplement the Collaboration Assessment Tool.

 No, but available upon request

 

Health Indicators & Selection Criteria

 

To track population level changes.

 

A comparison of baseline (2009) and follow-up (2020) data will inform the evaluation of BHCLB’s impact in the target area.
 

 No, but available upon request

 

 

 

 

 

14.  Please describe how you currently and regularly engage BHC partner agencies/orgs/grantees in evaluation efforts.  If you do not, please share barriers to engaging this stakeholder group.

 

The process of selecting health indicators involved interviews with key stakeholders working in each of the target areas listed in the Community Action Plan (e.g., school climate, quality housing, mobility & safe passage, food access, etc.).  A preliminary list of health indicators was reviewed by each of the workgroups and ratified by the L&E Committee. 

With regard to data collection activities, partner organizations/grantees provide insight on how best to administer data collection tools to their resident pool.  Once data are collected, they are shared with those who were involved with data collection and with the larger collaborative to help inform their work.  As mentioned previously, evaluation activities are also informed by the L&E Committee, which is mainly comprised of BHCLB partner organizations that are working on the target areas listed in the Community Action Plan and, therefore, are most familiar with the initiative and its mission and values.


15.  Please describe how you currently and regularly engage residents in evaluation efforts.  If you do not, please share barriers to engaging this stakeholder group.

The Resident Power Survey was developed with input from community residents, particularly when defining resident power and determining its three core elements (i.e., engagement, leadership, influence).  Residents are also engaged in BHCLB Learning activities, breakfast/book club/brown bag discussions that foster dialogue on thought-provoking issues related to BHCLB target areas.  

 

 

16.  Please briefly describe how you may be building evaluation capacity within your site.

 

BHCLB’s L&E team serves as a resource for partner organizations and to date, a number of partners have asked CHER staff to help conceptualize an evaluation study, for referral to specific data, and to serve as a member of a research team to help inform campaign work. BHCLB’s L&E team has also held workshops to build research and evaluation capacity among partner organizations and grantees.  These workshops include the Evaluating for Change training, which was held at the BHCLB Hub Office in collaboration with the TCC group. 
 

 

17.  What evaluation models or frameworks have been useful for your site?

(Example FSG Collective Impact Framework, Community Action Plan)

Evaluation model or framework

How have you used it?

 

TCE’s Learning Framework

As a feedback loop, where BHCLB key stakeholders are part of evaluation design and data collection, and results are shared back to help inform their work.

 

 

Social Ecological Model

 

To ensure that evaluation activities involve, and results reflect, all levels of the community (i.e., individual, group, population). 
 

 

 

 

 

 

18.  What stage is your L&E team at with respect to the following evaluation components? Please mark the box underneath the number that best matches your assessment.

 

1

Forming

2

3

Developing

4

5

Mature

Planning & evaluation design

 

 

 

 

 X

Implementation of the evaluation design

 

 

 

 X

 

Using data for learning & continuous improvement

 

 

 

 X

 

Sharing out evaluation results with diverse audiences

 

 

 

 X

 

 

19.  Please describe an evaluation-related challenge that your site is currently working to resolve. Please briefly describe the problem, your approach to finding a solution, and any additional resources or expertise that would help your site.

 

One challenge relates to the Collaboration Assessment Tool. One BHCLB workgroup recently completed the assessment and it was very challenging to get members to complete the online survey.  The assessment for this workgroup was initiated in October 2015 and finally completed in February 2016, with two members completing the survey during the last month.  The workgroup was given the option of completing the assessment on paper at a regular meeting, but members chose to complete the assessment online, as the meeting agenda is always full.  However, while the online survey may be more a convenient option for workgroup members, their busy schedules were a major barrier to survey completion.  Moving forward, groups will be encouraged to take the assessment during a regular meeting and the assessment will be tailored to meet the needs and/or characteristics of the workgroup. 

Another challenge is related to administration of the Resident Power Survey in 2013.  We initially planned for the survey to be taken online to minimize data entry, but quickly learned that many residents do not have computer access, even though they may have email addresses. Ultimately, a member of the L&E team had to meet with residents individually at local coffee shops to administer the survey using a computer tablet.  Paper surveys were also available; however, they were only administered to Khmer residents, who preferred to complete the paper surveys in a group setting. 

Reluctance to complete the Resident Power Survey in 2013 may be due in part to the length of the survey, which required approximately one hour to complete.  Residents received a $10 gift card to Target as an incentive, but the amount may have been too low for the amount of time required.  Due to the challenges encountered and the findings from the first round of survey administration, CHER revised the survey to remove questions that did not generate meaningful data and by doing so, survey length was shortened by almost 50%.  The incentive for participation was also increased to a $20 gift card.  Lastly, the second round of surveys was administered using paper surveys rather than the web-based survey.  A member of the L&E team visited 15 partner meetings between July and November 2015 and collected a total of 211 surveys, up from 77 surveys collected in 2013. 

While residents preferred paper surveys, there are two major challenges related this method of data collection.  First, paper surveys increase the likelihood of missing data due to non-response or incorrect responses (e.g., selecting more than the indicated number of responses is treated as a missing value).  Second, data entry by hand, rather than directly via a web-based survey, increases the likelihood of key stroke errors. To address the first challenge, we learned that residents need to be given very clear instructions on how to respond to some of the questions and even still, responses need to be reviewed when residents submit completed surveys.  Any discrepancies can be resolved with residents while they are still present.  A verification process has been implemented to address the second challenge.  This process requires three individuals; one person to code all responses, one person to enter data using these codes, and one person to verify that data are entered correctly.    

 

20.  Is there anything else you would like to share?

 Not at this time.

 


Comments