• Please enter minutes on this page, with the newest at the top. Put a line between entries. Highlight action items in RED with a name/initials indicating who's responsible for the task and a due date, if there is one. For minutes prior to 2014, see the Archive of work through 2013.

Assessment Committee Meeting Minutes - 5/10/2017


May 25th, 12:30-2:00 (location TBD) - New Members, Brainstorming, and Committee Directions

Present: Linda (chair), Brendan (minutes), Konstantin, Steve, Ellen
  •  Minutes & Announcements
    • Restarting rotating minute duty by last name, no announcements.

  • Debriefing from all-staff meeting from 4/19
    • Good turnout & community engagement.  Location and food were assets.
    • Not enough time.  15 more minutes as padding at the end would have helped.

  • Scheduling a library-wide open brainstorming meeting
    • Brendan & JD planning it, hoping for first session in early June
    • Initial plan to display survey data and ask "How do we fix or address this?", give everyone sticky notes, and discuss all ideas at the end.
    • Focus on 3 broad recommendations from the survey at least for first session.  Let other session topics (for another time) come from contributors.

  • Discussion: Meeting with branches and departments about survey results
    • New spreadsheet in Google Drive has all the current requests for survey data that have been made and links to the files sent out.
    • 2 emails, first one to supervisors about meeting with the committee for branch/department improvement & letting them know about open brainstorming meeting, second one a library-wide invitation to the open brainstorming session

  • Discussion: New members, needed skills, responsibilities (Names will be withheld until announcements are made)
    • Three committee members opted to leave.
    • Two candidates will be invited to join by Linda.
    • New goal to reduce "single points of failure", i.e. skill sets that go away if a person chooses to leave. Committee members will have primary responsibilities, like R analysis, NVivo analysis, project management.

  • Google Drive organization
    • We should keep all long term storage and archival items within the Sharepoint site for the Assessment committee.
    • Brendan will contact someone from the sharepoint implementation task force to discuss fleshing out our site.
    • May be able to add a widget for meeting minutes.

  • Review document: Updated Assessment Committee Practices
    • Added a block about survey practices,
    • Goal of including more meetings open to interested staff.
    • Adopt agenda and minutes templates on Sharepoint.
    • Starting on June 14th, we will try regularly scheduled, bi-weekly meetings.  Wednesdays 1:00-2:00 proposed.
    • Next meeting will be May 25th, 12:30-2:00 in 2 halves:
      • 1st half devoted to introducing new committee members & running through polished format brainstorming session including JD Kotula.
      • 2nd half will be opened up to library staff and discuss new avenues for and ideas for assessment for the committee to pursue.  

To do:
  • (Linda) Draft/send emails for
    1. Notifying supervisors about meetings
    2. Alerting library staff about open brainstorming session
    3. Alerting library staff about open half-meeting about committee's goals and new directions
  • (Linda) Formally extend invitation to potential committee members.
  • (Brendan) Contact Sharepoint committee about fleshing out the AC site.

Assessment Committee Meeting Minutes - 4/12/2017

Present: Linda (chair), Dan (minutes), Brendan, David, Konstantin, Steve, Sarah, Anna L., Mike W.

1) Run through of Library-wide presentation :

Agenda for presentation:

·         2:30-3 p.m. Refreshments

·         3-4 p.m. Presentations

o   Welcome (Bob Hudson)

o   Announcements

§  BU’s GenEd initiative (Tom Casserly)

o   Library Survey

§  Overview & quantitative analysis (David Fristrom)

§  Qualitative analysis of comments (Brendan DeRoo)

§  Use cases

·         BULS improvement (Mike Ward)

·         Law School accreditation (Anna Lawless-Collins)

§  Q&A and closing remarks (Linda Plunket)


a) David will hand out pamphlet ...  ask audience "what stands out to you?" 

  - each speaker will introduce the next in turn

 - we will project the agenda 

 - we will have a single slide deck, which we will put on maybe sharepoint, David will mention at beginning

 - Dan and Linda will be note takers 

 - Dan will release the news item on Monday next week

 - Linda will be a time keeper

b) Brendan will go through comments.... asked for suggestions on how to cover them

 - Sarah, give a few major themes that came out in comments that we didn't get through qualitative data

 - David - give top 4 or 5 themes (nodes) in nvivo represented by comments

 - put up a good comment - then a word cloud

c) Mike W  ... BULS

 - Committee decided that mentioning that he received data early is ok, folks can ask

 - data was helpful in setting priorities for next few years

 - Mike categorized impressions as positive or negative, sign ins, linking fails, etc...

 - will say something like "stay tuned for more on new initiatives" to provide the public more feedback on how we are working to improve BULS....

 - Steve. good to note that und comments mostly positive, grad and fac not so positive, on BULS

d) Anna ...  use of the data during law accreditation

 - Linda will begin the presentation with this data has helped 3 schools already with accreditation... 

 - Sarah - assessment is expected from various bodies

e) Linda... closing comments

- Sarah... suggests talking about the assessment cycle... looking for suggestions on what would help "you" (as library staff) do a better job for patrons

- David - maybe have the recommendations up on the slide behind Linda

 - Steve - include idea that supervisors should now know the report recommendations and act to make their services etc better

2)  Brendan - anonymizing comments

 - should we cut out both positive and negative comments mentioning staff by name?  YES, we will redact names

Mike should consider whether he should put up the quantitative data on BULS on a slide behind him and then give his talk

Those using slide decks please send their slides to DAVID asap.

Assessment Committee Meeting Minutes - 3/29/2017


Present: Linda (chair), Konstantin (minutes), Brendan, David, Dan, Tom

  Preparing for the presentation for staff (April, TBA)

·       Presenters

o   Bob – 5 min

o   David

o   Brendan

o   Mike

o   Ana, Ron, Ken

o   Linda

We’ve heard back from Mary concerning IRB – a clarification form that Brendan has already archived

David, Linda and Brendan are not going to post raw data about faculty affiliations in Open BU but these data may be used internally.

We’ve reviewed the agenda for the staff meeting

 We've brainstormed the following:

Process improvement

·       Process

                  o   Do as much as possible ahead of time

                  o   Need good planning process (as adjustable)

                  o   Objectives – accomplishing the goals

·       Time management and scheduling

                  o   Project schedule/management

                  o   Realistic calendar

                  o   Jump into analysis

                  o   Distracted by other projects

·       Remote participation

                  o   Clear A/V participation

                  o   Allow others to participate

·       Report production

                  o   Bare bones vs more targeted reports?

                  o   More advanced analysis

                  o   Professional copyediting

·       Workload delegation

                  o   Breakdown of sections

                      §  Design of the report

                      §  Data analysis

                      §  Data management

                       §  Statistics

                  o   Who is writing what?

·       Activities – publications (related to goals/vision)

·       Storing – Record keeping

             o   Documenting process

              o   Use dummy sets

              o   Storage

             o   Need partner

·       Non-AC assessment activities

            o   Liase with others (f.ex. Informational literacy)

            o   Consider BLC

·       Instrument, Process

           o   Sub-groups – speeding up the process vs. slowing down the concentrated effort

           o   Individuals need perspectives group

           o   Auto-generation – data sets

           o   Focus-group to critique the questions

           o   Consulting with survey experts on campus

            o   We have to be critical with the questions’ value (e.g. info)

·       Analysis

          o   Consider earlier tips

          o   Plan process

·       Skills

         o   Do skills development ahead of time

         o   Statistical skills

         o   Select tool ahead of time

         o   Consider open vs. proprietary

         o   Outside skills

·       Goals/Vision/Audience

          o   Open data?

          o   Parallel development

          o   Align goals with priorities


Ideas:  publish raw data before the report; create one-page reports


Discussion of AC as a group

Where are we going?

·       What services and resources are we  going to asses?

     o   Listening to staff group frustration

          §  Surveys, f.g. talking

      o   Working with BLC ILL COI

      o   Space assessment

      o   Physical equivalent of mines

            §  2 Q of walk-ins

       o   Survey recommendation (READ)

       o   ONE BU – other BU libraries included

       o   Other assessment groups

        o   Integration – integrating more into a process

                  §  What questions do you have?

                  §  Have more open meetings

       o   Surveys

                  §  Too Mugar focused

       o   Meetings

                 §  More structure

       o   Role

                 §  Consider coordinating role across libraries


If any member of the AC at any point would like to leave s/he  should let the Chair know  

·       Action point:  Linda will email the AC members 

Assessment Committee Meeting Minutes - 3/22/2017

NEXT MEETING: Mar 29, 2:00-3:30 (location Estin Room) 
Topics - Turning Assessment Inward: Evaluating survey design, distribution, analysis, and reporting efforts
            & Visions of the Post-survey Assessment Committee

Present: Linda (chair), Brendan (minutes), David, Steve, Dan, Sarah, Tom



  • Preparing data for distribution and archiving
    • Quantitative datasets
    • Qualitative datasets
  • Presentation for all staff
    • Date for said presentation
    • Where would be best
  • Meeting with Branches
  • Documentation
Preparing data
  • How do we anonymize comments?  Should we remove negative or hurtful things though they could be used for internal betterment?
    • Do not share on OpenBU (for now).
    • Relevant comments to be shared to branch/department supervisors and interested parties after discussion.

  • Comment demographics could be useful but would need to be checked that identifying groups weren't left in (e.g. SHA)

  • Procedure for anonymizing:
    • 500 (*edit* 700, Brendan counted wrong)  comments to review distributed to Dan, Sarah, Brendan, Konstantin, and Steve each. Linda requests all.
    • Classification system for comments: Safe/Context Dependent/Identifying.  Color coding for context dependent and identifying comments.
    • Context dependent comments may be useful for internal purposes but we wouldn't want everyone reading them (e.g. mean or hurtful comments).

  • Quantitative data needs limited demographic info as well
    • David and Brendan will meet to go over sufficient quantitative anonymity measures

  • SurveyKeeper: Consider a warning in the introduction about identifiable information left in comments.
All-staff Presentation
  • LAW would be ideal to host, around end of April, maybe 2 or 3 PM for an hour.
    • There may be space restrictions around LAW exam periods happening in April.

  • Slideshow & supplemental reading materials

  •  Tentative plan: 
    1. Linda introduces
    2. David sums up report and quantitative findings
    3. Brendan explains qualitative stuff
    4. Anna & Mike share how they used the data and comments.
    5. Linda closes.


BRENDAN to distribute comments to anonymizers.


DAVID, BRENDAN review quantitative sets for anonymizability

LINDA contact Mike and Anna about sharing their experiences

Assessment Committee Meeting Minutes - 2/16/2017

NEXT MEETING: Feb 22, 2:00-3:30 (location TBD)

Present: Linda (chair), Brendan (minutes), David, Steve, Dan, Virtual Konstantin


- Review comments about introduction & Summary

- Review feedback from specialists

- Organization of files in OpenBU

1) Organization and Files in OpenBU

    o   David proposes to merge reports into a unified report.  Will require a bit of rewriting for the references to other subreports and to remove redundancy of summary

2) Reviewed notes about Introduction/Summary

    o   Recommendations are necessarily “increase-improve-acquire” oriented (as opposed to “cut-reduce”) due to how the survey was structured.

    o   Final recommendation was agreed upon.

3) Specialist Feedback

    o   Wonderfully high response rate.

    o   Kanopy wording should not be “has been negotiated” – technically incorrect, still in negotiation.

    o   Discussion about statement about library inventory project.  We’ll decide on wording once we know for sure what it will entail and when it is slated to be completed.

    o   Only one comment about BC search interface, so our statement expressing generality is incorrect.

    o   We should ask Vika about her comments about misrepresented data in case we are missing anything.

4) Discussed statistics behind graphs

    o   Brendan fact checked the data.  David and Brendan discussed different methods of acquiring statistics, in particular the Use/Importance joint charts.

TODO: Linda to contact Doreen about timeline and scope of the inventory project 

            David to reach out to Vika to address her question

Assessment Committee Meeting Minutes - 2/7/2017

Present: Linda (chair), Brendan (minutes), Steve, David, Tom, Dan



-        Linda agrees that data spreadsheet is okay to share with the Data Repository people.

-        Linda has List of reviewers for the subreports.


Discussion about Comments on Library Reports

-        Used positive to negative ratio when there was a significant number of NA responses.

-        Recommendations for the Library Collections were discussed.  A problem lies in emphasizing the importance of the budget without pushing it in the provost’s face.  Wording was handled delicately for all parts.

-        East Asian language increase recommendation was discussed, is the problem lack of staff?  How do we phrase it that addresses the issue?

-        We reviewed new recommendations – Shelf Organization, Bib Managers


Linda will compose messages to the listed reviewers.  Will offer 3 days to review.

Next Meeting: 2/16 (12:30-2:00) in the Lg. Mugar Conference Room- To discus Executive Overview

Assessment Committee Meeting Minutes - 2/1/2017

Present: Present: Linda (Chair), Tom, Dan, David, Brendan, Steve (minutes)

Next meeting: 2:00-3:30 on Tuesday, 2/7, in the Mugar administrative conference room

1.) Review of spreadsheet on survey data broken down by departments. RESOLVED

Review of Library Support of Research report. REVIEWED AND EDITED PDF

Information needed for depositing documents in OpenBU: Eleni will need official author(s) (Assessment Committee, individual members?), titles, metadata, collection title - LINDA

4.) In addition to
Library Heads, which specialists who should review the final draft for essential changes?

Jack Ammerman, Mary Blanchard, Tom Casserly, Joe Harzbecker, Bob Hudson, Arlyne Jackson, Helen Jacoby, Anna Lawless, Tim Lewontin, Amy Limpitlaw, Ken Liss, Holly Mockovak, Beth Restrick, Jennifer Robble, Mike Ward, Ron Wheeler, Vika Zafrin

Please review this list of specialists and AULs. Please get back to Linda by the end of the day on Friday, 2/3, with any comments, deletion, or additions.

5.) We need a title for the summary/general report (Executive Summary? Overview? General Summary?)


Assessment Committee Meeting Minutes - 11/16/2016

Present: Linda (chair), Dan (minutes), Brendan, David, Ellen

Next meeting: 12:30PM on Wednesday, 2/1, in the Estin room. 

A) Discussion of Teaching and Learning report
David looked through committee comments on the report draft and where there was some grammatical error he just accepted those, whereas he marked others in red for committee discussion.  As we looked through the report, David brought up these "red" issues, the committee discussed, and we came to a resolution that David then noted in the draft. Also, Linda will work on a recommendation involving citation managers for this report.

B) Discussion of Library as Place report
We followed the same process as above for this report.  There was a discussion of the recommendation to upgrade the appearance of the libraries; David will go ahead to rewrite this.  Idea for survey keeper - partner with Global Programs or the International Student Services Office to add to our surveys, in terms of serving diverse populations.

C) Discussion of Comment Trends
Committee members responsible for subreports should go back into them, look at what text David has added for comment trends, and be prepared to report back to the group.  This is to ensure that David has relayed the trends correctly.

D) Data Sharing Group
The committee discussed Tom's proposal to share some data with the Task Force working on this.  We came to the conclusion that 
Brendan would work up two spreadsheets - one with "emp sub groups" with 4 or less affiliated faculty completely excluded; and another with "emp sub groups" with 4 or less faculty aggregated.  We hope to share this work with Tom when next available.

Out of time


Attendees: DF, DB, KS, LP, TH

1. Volunteer to take brief minutes?

2. Review of comments for the latest versions
 - reviewed report library as place - version 0.11
 - Linda will review the usage of Mugar Memorial Library (as first use) and "the" Alumni Medical Library
 - review financial implications of recommendations

 - Comments
   - Steve - Collections
   - Dan - Research
   - Ellen - Place
   - Linda - Instruction
   Please review the comments to ensure diversity of places and audience (faculty, grad, undergrad)

3. Sharing Data

4. OpenBU and the MINES report? (if time)

5. Next meeting - 1/26 - 2:00-3:30 (large conference room) ; 2/1 12:30-2:00 (Estin room)

Assessment Committee Meeting Minutes 12/2/2016

Next meeting: Linda to send out Doodle

  • Attendees: BD, EF, SS, TH, LP (chair), KS (via google)

  • David will write more of report and release it 2-3 days before the meeting as an open document for comments, then he’ll reply to the comments and we will use the meeting to hopefully get to resolutions in writing.

  • LP and BD met, and would like assessment committee members to be more involved in selection of comments and identification of trends

    • TH suggests BD be our technical navigator--we pose questions, and he pulls them out/navigates, and then saves them, so everyone can get into NVivo, and works with the saved queries

    • David is looking for catchy comments that catch your eye, also thematic comments that help illustrate our statistics → why people are strongly in favor/against something

    • LP suggests people volunteer to cover areas

      • SS: collections

      • EF: law/theo/med

      • TH: data

      • CS: circulation?

    • Those who were unable to attend today’s meeting should pick an area and email assess-l to let us know what you would like to query/cover. LP will also send an email out about this

    • We will select and have specialists (i.e., Ken, Vika, Anna) to help query their areas of expertise.

      • LP and BD will set up an intro to NVivo and Library Use Survey Qualitative data sessions to facilitate this with our selected specialists

      • Specialists will then either meet with Brendan who will do the query, or send their their questions to Brendan who will run the query

      • Linda will write this up and send more information to assess-l on Monday

    • Timeline: This should be completed by the end of January/early February

  • Brendan: Review of Comments

    • 3 data sets, one for each survey

    • Review of nodes structure, i.e., quality, which tags positive and negative comments

      • Categories are not mutually exclusive

    • Classifications: by question/answer. Each response is a “case”, so can pull out all the people who had the same answer to a question, and review their comments

    • How nodes are coded: source: undergrad survey, go to comment, right click, select “code on existing nodes,” select nodes at which comment should be coded (i.e., Mugar, study space, seating, etc)

    • Query, i.e., “what journals have people requested?”

      • CAS→ humanities→ foreign language and literature = all foreign language materials commenters requested

    • Example: comments on books or journals we are lacking, according to engineering faculty

      • Source: faculty survey

      • Code at: resources, lacking resource, subscription range, books, journals, limit to ENG

      • Run query

      • Save query

        • Each query will be named and saved. Ideally, queries (nodes selected) will be saved in notes/description field

        • SS suggests that the query is the technical way of asking the question

        • Save into existing “queries” node

        • Can also save as a node (best for large sets), and then use that set to filter further

      • “Staff” is included under library services node

    • For faculty, select by college, for undergrad/grad, look at COL1 (because can have more than one college)

  • Can also do matrix coding to see number of responses on broad topic, to better determine if subject is worth querying

  • Brendan will send around a list of nodes and descriptions and a data glossary and upload Windows and Mac versions of files to Google Drive, and change log

    • Everyone should install NVivo and download the appropriate file to start thinking about and running queries as needed


Assessment Committee Meeting Minutes - 11/16/2016

Present: Linda (chair), Dan (minutes), Brendan, David, Steve, Tom

Next meeting: 1:00PM on Friday, 12/2, in the Mugar admin conference room.

Discussion of 2016 Library Survey Report

a.  Qualitative analysis (Brendan, Konstantin, Ellen)

Discussion of comment document
at https://docs.google.com/a/bu.edu/spreadsheets/d/1onXmmLsrBK9nCY6nlNtQooMUK5ASNF1_82ypRaj7S6k/edit?usp=sharing

The sub-team working on this collected:

- interesting comments that are typical or come up often, or enriches a statistic
- things that are relevant but don't fit into a report major category
- comments that bring something up that we hadn't thought about

Linda noted we want to present the data as it is .... not negative or positive. Brendan asked, how do we want to be "representative" of the comments? Steve answered- we want to gather comments that illuminate issues, whether they are representative of the whole sentiment or not. David noted that pop out comments should be reasonably well-written, and at least have something to do with the nearby charts or data.

The committee went through comments to discuss interesting ones...  collecting them on "sheet 2" within the Google Doc above.

b. Quantitative analysis and report (David)

David continued to work up charts and went through an explanation of a few that were interesting: for instance, grad students responding to how much the library helps them teach. The committee discussed which stats were interesting and how to best format the charts. Tom suggested we show absolute number of folks on these charts; it could help give some perspective.

David continued, bringing up three charts side by side, showing use, importance, and satisfaction of course reserves. For this he limited to teaching faculty, and the committee discussed if that was appropriate. Linda noted we should change the label to Teaching Faculty to make it more transparent, if that is the population. This chart showed a relatively low number of teaching faculty use reserves, and Steve noted it is therefore the amount NOT using course reserves that may be more interesting: is course reserves a dying technology similar to the fax machine?

Out of time.


Assessment Committee Meeting Minutes - 11/2/2016

Present – Linda (chair), Brendan (minutes), Tom, Konstantin, Steve, David, Sarah


Assessment conference: BU looked great at the conference according to Linda. Congratulations everyone. Posters will be online.


Agenda: User Survey Reports (Quant, Qual, Contextual Comments)

-        David: Quantitative Analysis

o   Options presented for dealing with missing data (unanswered questions, NA responses, etc) in Likert questions.

        1) Show only 1-5 responses. 

        2) Show all possibilities (including NA, unanswered)

        3) Show all but the non-answers

        4) Show 1-5 and the % of respondents answering that part of the question

        5) Show 1-5 and the non-responses for those answering *some* part of the question.

o   David and Linda offer to go with simplest graphs with detailed, unambiguous captions.

o   In depth discussion about the significance of including/excluding “N/A”.  Include? They made an active decision in selecting N/A.  They are a significant percent of the population.  Exclude? Does it contribute to the story of the data, or is it distracting? Is it interpretable?  Ultimately every graph will need to be scrutinized and decided for individually.

o   Brendan requests we reconsider always including NA options (listed in Survey Keeper)

o   Faculty and Graduate questions about funding were different.  One had checkboxes.  One had radio Y/N/?.

o   Survey Keeper: Check boxes don’t allow explicit y/n options, meaning analysis is more confusing and requires more context.

o   Konstantin proposes having our PDFs with links to an online, publically accessible, interactive data dashboard

-        Brendan: Qualitative Analysis

o   Google sheet has categories of comments broken down roughly by topics in report, representative comments, and general findings from comments.  Research findings is the only one currently shared.

o   Brendan's spreadsheet at the following link: https://docs.google.com/a/bu.edu/spreadsheets/d/1onXmmLsrBK9nCY6nlNtQooMUK5ASNF1_82ypRaj7S6k/edit?usp=sharing

o   Brendan wants specific ideas for comment categories to pull.

o   Trends in the resources topics were discussed.

o   Steve requests comments about distance students and specific subjects for lacking resources.

Next meeting---Wednesday 11/16, 12:30-1:45 in Mugar Conference Room.

Assessment Committee Meeting Minutes - 10/19/2016

Attendees - Dan, Brendan, Steve (minutes),  Linda, David, Tom, Ellen, Sarah, and Mike Ward (invited guest)


User Survey Report- continued discussion

a.       Quantitative data & report writing – discussion led by David

For survey keeper: Storage location of research data – answers aren’t mutually exclusive (e.g., someone relying exclusively on an IS&T solution could reasonable check both ‘commercial cloud storage’ and ‘IS&T storage solution,’ Google drive being an IS&T solution)

Regarding sharing draft documents with others in the library, it was decided that we should hold off until the text of each had been fleshed out more.


b.       Qualitative data analysis - tabled

See Brendan’s spreadsheet with subjects and topics that David covered in our last meeting, with related comments pulled out of the qualitative data:



 Next meeting: 1:00PM on Wednesday, 10/26, in the Mugar admin conference room.

Assessment Committee Meeting Minutes - 10/5

Attendees - Dan B., Sarah S., Brenden DeRoo, Steve S.,  Konstantin, Linda, David, and Tom H

Absent - Ellen


Minutes, announcements, additions to the agenda?
+ None

Initial Draft of Report on Research - Discussion was led by David

Other Potential Reports
+ Support on Teaching and Learning
+ Resources and Collection
+ Library as Place
+ Executive Summary

Assessment Committee Meeting minutes 9/22/16

Present: Linda (chair), Dan (minutes), Brendan, Ellen, David, Steve, Tom, Konstantin, Sarah

Next Meeting: Wednesday 10/5, 1-2:30, Lg Mugar Conference Room


  - Linda, Ellen: consider and present to the group three or four possible ways to lay out the content of the top level report
  - David: keep working R for data.  On behalf of the team, Thank you so much!

1) Announcement:
We received a request for data about the use of library computers from IS&T.  On short notice we were able to provide great "just-in-time" stats about the use of library computers.

2) Discussion of top level report

a) looked at a series of new charts by David using graduation caps to represent respondents (each cap = 100).  These were Frequency of Visit charts, both physical and online: one for undgrads, one for grads, one for faculty, and one combining all three.  Steve suggested having the axis labels wrap around the content so that they are easier to read. David suggested he may shade alternating rows to help readability, and will otherwise beautify the content.  Sarah suggested perhaps producing interactive content online after the main report.  Dan suggested maybe the one that combines all of the patron groups might be too busy for our top level report.

b) David presented a diversion bar chart for satisfaction with BU Libraries Search.  Considering it is a major aspect of our work it may be good to feature this.  The first faculty survey in 2010 was given just before the first implementation of BULS.  There lead to discussion about what we really want to feature in the report.

c) Discussion of what belongs in the first report from the library survey 2016.

Steve suggested we include items for which we have made demonstrable (and sometimes even measurable) changes due to survey feedback.  Outlets would be an example, but we thought it may be tough to connect many improvements directly to our survey results.

From the paper flipboard in the room: High level report must include: overall satisfaction with the libraries (1), contributions (1-3), use of the libraries (the graduation cap charts).

There was much discussion. Finally we decided Linda and Ellen will meet as a subgroup to consider and present to the group three or four possible ways to lay out the content of the top level report / executive summary.

3) Discussion of the qualitative data; comments

Brendan related that the subgroup organized tagging by having members go through and create nodes: Konstantin for Faculty comments, Ellen for Graduate student comments, and Brendan for undergraduate comments.  This was phase one of the project.  Phase two would be to merge these 3 sets of nodes into one large set.  Phase three would be to un-code items that are not relevant.  Konstantin walked through his process of tagging faculty comments.  For reference, there were about 1000 faculty respondents, and a total of 695 comments.  One thing he did that was different than what the others did was that he coded comments for "attitude": - positive, negative, and recommendations.  Sarah noted that if there are questions about some aspect of comments that come up after we produce a final node set, that concepts can be pulled from nVivo afterward.  An example was mentions of peer institutions.

Out of time

Next Meeting: Wednesday 10/5, 1-2:30, Lg Mugar Conference Room


Assessment Committee Meeting 9/16/16

Present: Linda (chair), Steve, Brendan (minutes), Dan, Sarah, Konstantin, David, Tom

Next Meeting: Thursday 9/22, 2-3:30, Lg Mugar Conference Room


  - David: share paper about divergent bar chart format

  - David: make a heat plot of library use data.


  • Quantitative Data and Report Progress (David)
  • Qualitative Data Analysis Progress (Konstantin/Brendan)

Quantitative Data
  • Discussed top level report ideas
  • Graphs shouldn't be designed to present a message, rather highlight the data.  Presentation has to be consistent for both positive and negative aspects.
  • Discussions about graph format for overall satisfaction
  • For contributions graphs, we should consider excluding the people who didn't teach
  • Thoughts about cutting back on the number of contribution bars shown (at least for the top level report)
  • Our data could be interesting to many groups current on campus.
  • Library Use charts - discussions about the scatter format for comparing online and physical library use
  • "Graduate students being published" chart - Data is useful and wanted, but concerns about "top-level"ness.
  • The top level report should aim to be around 4 pages.
  • Perhaps add one important finding for each user group, space permitting.
Out of time
Next Meeting: 9/22, 2-3:30 in Mugar Conference room


Assessment Committee Meeting 7/26/16

Present: Linda (chair), Steve, Brendan, Dan, Sarah, Konstantin (minutes), Ellen, David, Tom

  1. Minutes, changes to the agenda, announcements 
  1. Library staff data survey recommendations (Tom, Ellen, Brendan)

150 library staff members and that more than 100 responded (~102 or 103).

Findings available on Google Drive

Key findings:

Over fifteen different solutions for storing data

73% respondents share with other library staff members

55% would have “most” or “all” of their products on their computer

Email is the most common way to share

Staff generally look for policies on the Library website


General Recommendations:

Begin a conversation with your department

Less is more

Only use BU-supported solutions

Develop a routine backup plan

Ensure work is saved in shared locations

Adopt simple, usable naming conventions

Move away from email as a storage and sharing tool


We can share this recommendation with staff as long as there is no personally identifiable information

Tom’s recommendation  1) adopt the Office 365 Suite as the default solution for the BU Libraries. 2) task a training team to facilitate Office 365 training by using both IS&T and local trainers 3) task a design team to design an intranet for the BU Libraries using SharePoint and explore ways to ensure the intranet has the appropriate staff resources to ensure it remains usable and functional

We should make clear that these are not final recommendations but a report on the survey. Tom and Ellen will work on the short introduction. Timeline: Linda will email Bob Hudson and AULs on Friday 7/28.


  1. Update on the analysis of the user surveys
    1. Demo of the comparison of survey questions, 2010-2016. (Brendan)

Brendan modified old data sets to match conventions we used in 2016.

    1. Demo of the ongoing quantitative analysis (David)

David is continuing work on the quantitative analysis. The trends are: faculty’s satisfaction with library is growing. The undergraduate and graduate comparison of 2016 and 2010 shows that the overall satisfaction with Library is overall did not change. David also is experimenting with scatter graphs and correlation analysis using R.

    1. Demo of the ongoing qualitative analysis (Brendan, Konstantin, Ellen)

Brendan, Konstantin and Ellen split up work between three different groups and began coding.

  1. Request to use a sample of library website Google Analytics data (Dan, Linda)


  1. Serials review, assessing journal subscriptions using COUNTER compliant data compiled in Use Stats (Steve) 

Usage stat for serials has been updated (around 1000 titles), including cost per use data.

  1. Update on posters to be presented at Library Assessment Conference (Dan & Ellen, Linda)


Next meeing: August 12, 1pm

Assessment Committee Meeting 6/24/16

Present: Linda, David, Ellen, Brendan, Steve, Dan, Tom (minutes)

Next Meeting July 26, 2016 @ 12:30 in large Mugar Conference Room

To do: Please comment on Data Management Plan by July 18 (all)

Updates from around the table

  • David
    • Working on the user  report and doing analysis in R
  • Dan
    • Using Qualtrics to review survey results with a SED/PERL focus. Currently focusing on space usage
    • Preparing a poster on MINES for the Library Assessment Conference 
    • Managing the wiki and the website
  • Steve
    • Pulling usage data for journals and updating usage data as needed
    • Pulling data from smaller publishers
    • Doing “cost-per-use” analysis
    • Doing ARL collections stats for end of fiscal year
  • Linda
    • Working assessment for SHARE
    • Setting assessment goals and milestones for SHARE
  • Ellen
    • Conducting a self-assessment for the upcoming ABA site visit
    • Law Librarians of New England are having an “assessment focused” conference this summer
    • Working on the staff storage survey
  • Tom
    • Drafting our Data Management Plan
    • Assisting with the Library staff storage survey 
    • Doing some analysis of the user surveys
  • Brendan
    • Working on user surveys
    • Working with Mike Ward on online usage and how they interact with PRIMO
    • Looking at user survey comments
    • Working on staff survey

Requests for Survey Data - We can give data for improving library processes. If you have any questions be sure to ask Linda or David.

Open Meetings - We will be opening the Assessment Committee meeting quarterly to all staff for information sharing

Publishing the MINES data - We can publish the data in the eScience journal with UMass Medical. The group is thinking about article length publication.

Staff Survey Preliminary Results

  • Things we would like feedback on
    • Who is most interested in this information? We should develop 2-3 specific recommendations for AULs and some best practices for staff
    • What end product are you interested in? Please provide a one-page report for audience
    • Do you want recommendations?A quick list of pros / cons of each of IS&T services. We should also provide some simple best practices
  • Covered the initial results
  • Timeline - Try to finish this project before the fall semester

User Surveys

  • David is using R for analysis and data visualization
  • Quantitative - David highlighted his current work and progress (which was great!)
  • Qualitative - Brendan, Ellen, Konstantin will be working as a team to conduct analysis on all the comments.
    • Going to use NVivo (data is already imported)
    • Working on a comment analysis framework
    • Goal is to be able to show comments based on several criteria (faculty from X who had a positive experience with Y)
  • Brainstorm Reports - we brainstormed ideas for our reports

Assessment Committee Meeting 5/26/16

Present: Linda, David, Ellen, Brendan, Steve, Tom, Konstantin, Sarah, Dan (minutes)

Next Meeting: 6/24/16 from 12:30-2, Mugar Admin Conf Rm

To do:
David - Start preparing a report

All - support David by looking through survey results in
Qualtrics to find interesting facts, OR
        just think up
some possible interesting themes and send

        requests to run statistics to David and Brendan

All - continue adding to the Data Management Plan

Tom H -
continue editing Plan into a more cohesive document

Jason Yee, Head of Library Computing Services at MED, was our guest
Konstantin has finished doctorate!  Congrats.

1)   Update on Staff Survey

  • Surveying everyone at BU Libraries
  • Currently receiving responses
  • Decided to close June 6th or after most responses received

2)   Update on 2016 surveys

  • David is going through results, and learning R
  • Perhaps will use Adobe Illustrator for finishing up charts
  • Stacked bar chart, centered – we really liked these
  • What is the best way to compile reports?

i.     Series of topics
Short reports of 3 or 4 pages with executive summary and methodology
iii.     Also short reports on fac, grad, and undergrads?

  • Other ideas for compiling the reports?

i.     Would it be worth doing longitudinal?
What are the changes?
iii.     Where did we make changes or improvements based on previous survey round

  • David- Time changes can be represented within individual topics
  • Sarah - Highlight positive impact from last time around
  • Tom H- Connection with Strategic Plan?
  • David wants to know about interesting things- investigate

i.     Email him and cc Brendan as well
Can ask David and Brendan to investigate, but can also view Results in Qualtrics
Dan- Qualtrics can be used to create “reports” and investigate data trends, results, other themes…. try filtering data
David will consider and edit text or data that other AC members contribute to the reports

  • Brendan also available to crunch numbers
  • Once the data is anonymized (Brendon has begun work on this), David hopes to have a first draft in mid-August

3)   Data Management Plan

  • Merged published reports and visual reports, only for the purposes of this document
  • Could have end section that details our overall data strategy. Tom will add this.
  • Tom will be molding this into a more cohesive document. He will reach out to individuals if he has a question.
  • This is meant to be a living document
  • We will pick a place for folder file structure – it will not be Dropbox due to privacy concerns
  • Tom reviews public, internal, classified, and restricted use types of data
  • Linda: after this document is complete, we may hold a library-wide meeting




Present: Linda, David, Ellen, Brendan, Dan, Sarah, Steve (minutes)

Notes from the look at the preliminary data analysis of the 2016 faculty user surveys (Brendan)

  • Teaching & funding by campus – compare to MINES data to explore relation between data sets

  • Open access: break out physicists, and how they are contributing to the overall numbers.

  • Importance/assessment of undergraduate information literacy skills Qs: 2010 – citations & ethical combined in one question? May need to explain for 2016 comparison. Can WR150, CGS, & Questrom faculty responses be pulled out for these?

  • Should be reviewed by the Library Task Force: Growing importance of info lit skills, but less satisfaction with skills of UG & grad students. Should the library be more involved in these activities?

Survey responders have been randomized and winners (5 UG, 4 G, 3 F) picked. Brendan will send the names to Linda, who will email w/ banner (running a draft by Ellen beforehand) asking them when they can come in to pick up their prizes. Pictures will be taken when possible & sent to Dan for posting & news item.

A look at how the data is being cleaned up (Brendan): Brendan has been meticulously recording everything he has been doing to clean up the data. This document is available on the Google Drive 2016 data folder as read-only

For our next meeting (5/26), complete Data Management Plan for (All):

     National library surveys including ARL, ACRL, IPEDS

    Triennial User Surveys



    Website Usage Patterns

    Visualized reports

    Published reports

    Data storage & sharing 

Next meeting: Noon to 1:30 on Thursday, 5/26, in the large Mugar conference room.



Present: Linda, David, Russ, Steve, Brendan (notes), Tom, Ellen, Dan, Konstantin

  • Announcements: Tom won two awards!  Congrats
  • Agenda:
    • Linda - Survey Results 
    • Dan - Publicity update
    • Brendan - Preliminary longitudinal analysis
    • Tom/Ellen/Brendan - Data Management Plan
  • Survey Results(Linda):
    • Response rate = 15.1% overall.  Surpassed Ithaka's
      • Lowest - Undergrad: 13.8%
      • Highest - Graduate: 16.6%
    • Problems with expired surveys in 3rd week. 16 k sessions "started" & expired. David saved the day! We resent the survey as a correction
    • Nicole requested we try not to use the Provost's name too much.  The fixed email went out with Bob's name attached.
SK: Have to be careful to not use the Provost's name too much.

SK: Expiration dates matter! Can't backdate on Qualtrics.

Publicity Summary (Dan):
  • Collecting publicity locations & social media screenshots, storing them in a single location for future reference.
  • Social media may be working to reach out to people. There were bumps in the response rates on the days of retweets.
  • For future campaigns, we need non-targeted adverts and images for things like social media.
  • IRB doesn't seem to mind about the effect of social media skewing who's taking the survey.
  • Next time we should try to get onto email lists and student newsletter.

SK: We should have a general image targeted for mass social media.

SK: Medical campus is not good for postering.  There are digital displays though.

Undergraduate results (Brendan):
  • All data captured up to 3/30.
  • Adjunct data was very low. Called "Part time contingent".
  • Changes made to libraries have definite effect on numbers.
  • Brendan will add highlights later
  • Less library instruction may have overarching effects on perception about the libraries.
-Brendan: To pull out following data-
    • Frequency of online use of resources by school/other demographics.
    • Instruction by School

Graduate results (Brendan):
  • Lots of Ph.Ds
  • Decrease in journals -> looking at archives and other resources now?
  • General trend of journals being less important, books being more important.
        -Brendan: To pull out following data-
    • Contributions by Schools
    • Resource importance by School
    • Nonimportance of Resources School
    • Review of new questions
    • Unsatisfaction with hours.
    • Unsatisfaction overall.

Tom - Send out the dmp information again again
Brendan - to meet with Konstantin about comment analysis.
Next Meeting: 4/20 at 12:45 at LAW. Come early for the NISO webinar on Library Assessment.



                                                                                                      BD, TH, DB, LP, KS, EF, RS, SS (telecom), DF

                                                                                                      Next meeting: 4/7 Thursday  12:00pm-1:30pm  


                                                                                                      The survey has successfully launched on 3/15/16 There were just 17 faculty emails that bounced

                                                                                                      Thus far: 4% for undergraduate 680  average duration 20 minutes.Graduate students 5%  1100   (the top responders are GRS and Medical school) Faculty 5%  250 

                                                                                                      There is a concern that we won’t be able to reach the same 30% we did last time. A positive development is that there are more comments than last time

                                                                                                      The launch was staggered to allow for the potential intervention. But everything went everything very smoothly.

                                                                                                      Steve suggested that we should develop a second reminder which politely reminds patrons that they haven’t yet taken the survey.

                                                                                                      The Task Force - we had a discussion how to approach the communication strategies with the staff in regards to the work of the task force. Linda will contact Bob Hudson.


                                                                                                      Every school is covered – digital and physical posters. Reminders will go 3/22 and 3/29

                                                                                                      Posters are on the buses

                                                                                                      Dan is tracking all the social media. Five postings on Facebook and four on the Blog, and nine tweets.

                                                                                                      News announcements on some of the library web sites.

                                                                                                       Idea – can we use mascots (BU Red) in the next survey to boost our response rates.


                                                                                                      DATA MANAGEMENT

                                                                                                      Google Drive

                                                                                                      One of the folders is labeled “Data Management”

                                                                                                      Tom suggested two approaches – a number of meeting devoted to this topic or to work together on Google Drive and then have just a few meetings. 

                                                                                                      We are waiting for the response from Bob Hudson regarding the survey that we would like to put out to the library staff about data management

                                                                                                      Data sets and the issue of de-identifying the data - can we release de-identified our data. We can certainly release the survey instrument and reports.

                                                                                                      TO DO:

                                                                                                      ·       Review the Google Doc – do everything up to the policy (the first three questions)

                                                                                                      ·       Review Shirley’s comments



                                                                                                      BD, TH, DB, LP, KS, EF, RS, SS, DF

                                                                                                      • Minutes, announcements, changes to the agenda
                                                                                                        • Dan is presenting a poster on MINES at the 2016 Library Assessment Conference. If you want to go, talk to Dan
                                                                                                        • Brendan and Tom went to the Software Carpentry workshop, focused on using terminal and associated resources to analyze data. Upcoming workshop on Data Carpentry: http://www.datacarpentry.org if anyone is interested 
                                                                                                      • User Surveys
                                                                                                        • Contact in Provost’s Office
                                                                                                          • no contact as of yet, Bob has been in touch this week and we should have one soon
                                                                                                        • Pilot of
                                                                                                          • Mary McCabe, exempt application coordinator at IRB says we can go ahead and pilot after we submit the application. 
                                                                                                          • Question: Can we put in the communications from the Provost’s office without IRB having reviewed them, and then change in real time as needed? 
                                                                                                          • Qualtrics distribution convenience sampling testing still scheduled for 2/1
                                                                                                        • Demographics—will not use data from registrar b/c can get from provost 
                                                                                                        • Publicity
                                                                                                          • Images are fine and do not need to be changed.
                                                                                                          • We can do one poster each week on Warren towers and will try to put a lot up on the busses. 
                                                                                                          • Dan will get in touch with each school and find out who controls digital display boards in lobby
                                                                                                          • Training
                                                                                                          • Everyone needs to complete the financial conflict of interest (check your email) by 1/29
                                                                                                      • Changes to surveys
                                                                                                        • Change 2015 to 2016
                                                                                                        • change Marlene Alderman to Ronald E. Wheeler, wheelerr@bu.edu
                                                                                                        • iPad mini3 TO iPad mini4
                                                                                                        • BU Libraries Search should NOT be all capitalized because of Law search (generic name, not proper name)
                                                                                                        • Library Usage and Resources—ask “during current year” grad and undergrad are same, except undergrad omits “accessing scanners”—> no reason for this difference, so add to undergrad?
                                                                                                        • Task force will not be a surprise. Provost’s Office will add it to the invitation, so we do not need to put it in the methodology 
                                                                                                        • Brendan to make changes & corrections to surveys
                                                                                                        • Brendan will make a read-only archive of all our things for the sake of preservation
                                                                                                        • Ellen to read all four survey instruments for typos after Brendan does overlay 
                                                                                                      • Other
                                                                                                        • We will be having Zepheira using an ALMA feed to transform our catalog records into linked data. It will have geotagging and google analytics so we can track things, to make the data in our records discoverable through Google. We should plan to survey before and after. 
                                                                                                        • Will this boost circulation and usage data? 
                                                                                                        • Mike and Jack will host a conversation on this initiative soon—please attend!

                                                                                                      Minutes - Assessment Committee – 12/11/2015

                                                                                                      Next meeting: Thursday, January 7 at 12pm -  Mugar Admin Conference Room
                                                                                                      No Announcements
                                                                                                      Present: Dan (minutes), Tom H, Tom C, Sarah, Konstantin, Ellen, Linda (chair), Brendan, David, Steve

                                                                                                      TechQual results at BU, and a coming National Survey of Student Engagement - Tom C

                                                                                                      • The library related question on TechQual is question number 17.  The results were pretty typical, but the idea is to score better next time.  This is a gap measurement tool (minimal vs perceived vs desired level of service).
                                                                                                      • Tracey of IS&T plans to run this again every two years.
                                                                                                      • There were 6 pages of comments collected, and Tom C will send them to the AC via email.
                                                                                                      • Ellen will get a copy and also we decided the DAG should receive them too.
                                                                                                      • National Survey of Student Engagement, this is past the planning phase at BU, but Tom has been invited to revisit the library related questions on it by Denise Mooney, as it will likely be given again in the future.

                                                                                                      Applying a Data Management Plan to MINES report material - Tom H, Ellen, Brendan

                                                                                                      • package up data from MINES survey for public use
                                                                                                      • idea would be to facilitate further insights for library or other colleagues
                                                                                                      • should include a readme.txt file that describes data formats and methods
                                                                                                      • Jupyter was mentioned as a tool that helps to capture workflow
                                                                                                      • Action item: Brendan will go through MINES data material to be made public.
                                                                                                      • Tom will move forward with this, and look to bring an example back to the committee

                                                                                                      Staff survey on storing and sharing data - Tom H, Ellen, Brendan

                                                                                                      • Idea is to survey staff about their data storing and sharing activity
                                                                                                      • LP- who will answer this survey and why?  All staff, for all work output?
                                                                                                      • TH - outcomes could come in form of AC recommendations to Bob H.
                                                                                                      • could ask questions such as: what is your essential function?, are you a supervisor or individual contributor? do you produce training or tutorial material?  do you have departmental guidance on where to store data output? does the data you collect end up going into a national survey (arl stats)?
                                                                                                      • Results will start conversations about best practices, especially in the situation that someone leaves the job and needs to pass on data processes to other folks
                                                                                                      • should the survey be anonymous or not?
                                                                                                      • Tom will move forward with this, consider methodology and bring thoughts back to the committee

                                                                                                      Item for next agenda: Linda

                                                                                                      • More opportunities to present, publish, and share the MINES Report and data REPORT

                                                                                                      Minutes - Assessment Ctte – 10/30/2015

                                                                                                      Present: Tom, Konstantin, Ellen, Linda (chair), Brendan, David, Steve (minutes)

                                                                                                      Logistic of 11/19 all-staff meeting:

                                                                                                      • Location: Law 525
                                                                                                      • Bob Hudson will welcome, with possible introductory comments by Marlene or Russ Sweet
                                                                                                      • Joanne Pierce, Learning Assessment, Provost Office will be invited by Linda
                                                                                                      • Refreshments will be arranged by Cathy M.
                                                                                                      • Ellen will arrange room & technology
                                                                                                      • Bob will ask that new staff be introduced (by supervisor)
                                                                                                      • Action Item: Assessment ctte members should plan to arrive at 1:30, spread out and welcome people; we should encourage our staff and colleagues to attend

                                                                                                      MINES REPORT

                                                                                                      • Acknowledgments: Will use titles rather than affiliation; please give David your preferred title
                                                                                                      • Action item: Send your title that you would like used in the acknowledgment section.
                                                                                                      • LP will provide affiliations for Terry Plum, Brinley Franklin, and Martha Kyrillidou, and for the ITS folks to acknowledgments
                                                                                                      • David will work up an executive summary
                                                                                                      • Change ‘to March’ to ‘through March 3rd’ or ‘through February’
                                                                                                      • David will work on the library visits portion of the “Where” conclusions
                                                                                                      • Tighten up ‘What is an online section,” second paragraph
                                                                                                      • Consistent use of ‘chart’ and ‘figure’ (i.e., there may be multiple figures within a single chart)
                                                                                                      • Delete the sentence intro descriptions for “Why” (top of p. 5) and “Where” (p. 6) and “When”; or move some of the description of these up from the conclusions
                                                                                                      • Strengthen 2nd paragraph under “Who” recommendation           
                                                                                                      • Action item: Give the report a final read through, make sure everybody is thanked/acknowledged.


                                                                                                      Minutes - Assessment Committee - 10/22/15

                                                                                                      Attendees: DF, RS, LP (chair), EF (minutes), BD, DB, TH, SS


                                                                                                      ·         Harvard assessment librarians will be visiting Monday, November 2 at 11 AM

                                                                                                      ·         Tentatively scheduling all-staff meeting to present MINES report November 19th from 2-3, but not advertising yet

                                                                                                      Action items: Linda, Brendan, and Tom will read the appendix of the MINES report and bring feedback. David will rewrite the conclusions section based on today’s feedback, in a cohesive and coherent style, and send around a new draft for feedback

                                                                                                      Next Meeting: 10/30 in Law Library Conference Room (third floor) 2-3:30PM. 

                                                                                                      MINES Conclusions Discussion--general

                                                                                                      ·         Primarily discussion of compiled comments available here

                                                                                                      ·         Will remove “use of library resources for course work is closely tied to the academic  calendar, while use for research is more constant” because it doesn’t really get us anything 

                                                                                                      ·         KS had suggested quotes from Franklin & Plum article. Decided not to use actual quote, but incorporate ideas in these and Tom Casserly’s quote on library use, and note that it is in line with what other schools have found (undergraduates are using library extensively, but they are not accessing online resources from within the library)

                                                                                                      ·         Also need to change “the libraries will” to “the libraries shall” because it’s a recommendation, not a mandate or promise

                                                                                                      ·         All of TH comments are included in LP comments, as are EF’s so majority of discussion focused on LP’s suggestions. 


                                                                                                      ·         We will incorporate Dan’s feedback on use of the term “online sessions” re grad students to avoid inferring overall use

                                                                                                      ·         Mines data does not tell us how heavily resources are being used. This is an indicator of library use, similar to gate counts, but there is limited ability to extrapolate overall use from these user sessions—need to reword title “number of online sessions initiated in year” in figure

                                                                                                      ·         We can tell who is using things, but not comfortable saying “heavy” or “significant” use because who can tell if it really is?

                                                                                                      ·         Change emphasis on use to who, rather than what—all members of community are using resources in what we would consider significant ways. Overall, all users are using print and electronic collections—no one group has been left behind by shift from print to e. 

                                                                                                      ·         Is there a way to include ARL data so that we can demonstrate how we rank in reference to our peers? (i.e., to determine how heavy our usage is) This would be difficult, because the data would need to be normalized (FTE and volume count)

                                                                                                      ·         “From the number of online session, we roughly infer usage”—> then include that all groups are making use of the resources (as opposed to roughly estimating amounts of usage by group), which also gets us away from qualitative statements about amount of use. 

                                                                                                      ·         Collections are part of our services, we should also discuss services.  

                                                                                                      ·         May also need to examine services to graduate students

                                                                                                      ·         “online journal collections” should be changed to “online collections.” Also need to include “online services” in this.

                                                                                                      ·         Reword to something like “library’s preferred mode of acquisitions is serving all patron groups equally well/satisfactory” b/c presently, implications don’t actually deal with WHO.

                                                                                                      ·         Also add that because grad students are heaviest user group, should consider/examine their services and needs to make sure we are adequately meeting them. 

                                                                                                      ·         TH: Who should have two conclusions and implications: one discussing everyone, and then also a second conclusion and implication on graduate students. 

                                                                                                      ·         Suggested text: Currently, collection building is largely focused on faculty, which serves undergrad demand (through course reserves). Unlike UT and UW, we don’t have separate undergrad and grad libraries, so question of what are we doing to satisfy graduate needs isn’t currently getting answered. Worth investigating, although ultimately may not be a big deal. 


                                                                                                      ·         RS: suggests focusing energies on getting greater integration b/w library resources and primo and course management systems

                                                                                                      ·         Need to rewrite conclusion to be clearer that it includes both types of students (grad and undergrad).

                                                                                                      ·         Also need to include that faculty secondary use is preparing for teaching.

                                                                                                      ·         Graduate student teaching use number (1%) makes sense, when you consider what and who is teaching—law, med, business don’t teach, usually teaching canned courses so not necessarily research for teaching. 

                                                                                                      ·         Break into three conclusions and implications: WHY coursework, WHY teaching, and WHY research (w/in research, tease out funded vs unfunded research). 

                                                                                                      ·         undergrad coursework implication: information literacy instruction. DF to pull from survey to determine whether or not they’re having trouble, how we can better support their efforts 

                                                                                                      BD did analysis of how WHY varies by location—tasks groups are likely to be doing in library vs other locations. Implications for services—need to make them available at points of access. Services based on research and scholarship should take this into account, especially for undergrads, since they’re not doing research when they come to the library, and by the time they are doing research, they’re no longer coming to the library. Determined to be interesting, but not worth putting into report. Save for internal use. 


                                                                                                      ·         MINES data and survey data conflict re value of space à Illustrates disconnect between library space use and library collection use.

                                                                                                      ·         If not using library space for access to collections, what are they using it for, and how well are we doing it? à External survey data brought in to help answer this.

                                                                                                      ·         MINES shows one side of this (use of collections)—> not saying library space is not important, just saying it’s not being used for what we thought it was being used for

                                                                                                      ·         Add to table on library resource use: library gate count to help illustrate that just because resources are not being used in the library, does not mean the library is not getting used. Once gate count is added, we can bring in survey data on importance of library. 

                                                                                                      ·         Suggest adding note on non-homogeneity of populations between libraries, because some use of e-resources comes from within library. 

                                                                                                      ·         Suggest adding to conclusions that just because students use online resources, they are still looking for human services—reference help, etc. 

                                                                                                      ·         Implication: need physical version of MINES to issue short survey to every 100th person coming in to the library. 

                                                                                                      ·         Need to include comments and conclusions on use of physical space to include professional students/schools

                                                                                                      Minutes - Assessment Committee -  October 15, 2015 

                                                                                                      Attendees: Linda (chair), Brendan (minutes), David, Russ, Virtual Konstantin, Dan, Tom, Steve, and Sarah

                                                                                                      Notifications and Changes to Agenda - 
                                                                                                      • University Assessment Workshop happening October 16th, 9am-12pm.  At least Linda & David attending.
                                                                                                      • Harvard assessment librarians planning a visit to discuss creating a library-wide assessment plan, they would like our input
                                                                                                      MINES Publicity Timeline
                                                                                                      • Goal is to send to Bob ASAP.  Bob will send them to Provost and to staff.  Upload to website shortly after that.
                                                                                                      • Plan for a staff-wide meeting around Thursday November 19th, perhaps 2-3:30. Will cover findings, methodology, conclusions and next steps at least.
                                                                                                      • Staff-wide meeting planning to happen a week or two before presentation date.
                                                                                                      • David agreed to do main presentation; Linda volunteers to voluntell us to present if more presenters are necessary.

                                                                                                      Discussion and Review of Final Draft – Conclusion and Appendix

                                                                                                      • Tom C.’s thoughts:
                                                                                                        •  Move the importance of MINES paragraph to the top of the introduction since it is most important.
                                                                                                        • "A lot of online use by undergraduates is in library, and this is a significant figure to quote for 'library-as-space' purposes"
                                                                                                      • General:
                                                                                                        • Our conclusions should each lead to implications. We can split up conclusions, for example, by the W’s of the report, or by user group.
                                                                                                        • Be wary of drawing transient or dubious conclusions.  Does the data really lead to implication?  Are statements about user groups reflective of reality (e.g. off “campus” use for faculty in MED campus)?
                                                                                                        • It’s okay to mention our results are surprising. It adds weight to implications and our analysis.
                                                                                                      • Discussion regarding library resources vs collections.  David will change resources to collections where appropriate and 
                                                                                                        • “Collections are a major resource of libraries” maybe in the conclusion?
                                                                                                      • Minimize clunkiness of first and second conclusions if we’re keeping 
                                                                                                      • Considering whether to have an executive summary.
                                                                                                      • Maybe pull in data from previous surveys/statistics.  Ex: “In the 2012 survey, this was the satisfaction of the grads with our online journals. Coupled with the MINES data…”
                                                                                                      •  Libraries aren’t *only* used for our resources.  Accessing our website is almost exclusively for our resources.  We shouldn’t make too many location extrapolations.
                                                                                                      • Expand on what we would change for our next MINES run-through.
                                                                                                      • Conclusion/implication suggestions:
                                                                                                        • Grad students are the heaviest users -> We should focus acquisition efforts on graduate students? Is that possible?
                                                                                                        • “Repurpose space for undergraduate use in library” – check with Bob first.
                                                                                                        • We should bring in how the data points to an increasing demand for online services, hence our changes to staffing.
                                                                                                        • Maybe a statement about improving resource accessibility, or help finding and interpreting our online resources. 
                                                                                                        • Current purpose conclusion : ‘Graduates use resources most for coursework, and then research -> We should expand (in-class ?) information literacy in context of the courses, or we need collaboration efforts with teaching faculty to ensure they have available and needed tools, or provide/supply support for research services such as data.
                                                                                                      • Fill out suggestions for conclusions and implications by the next meeting on the Google Doc: MINES Report - Conclusion Suggestions

                                                                                                      NEXT MEETING: 10/22/2015 at 2:00, location TBD

                                                                                                      Minutes - Assessment Committee -  October 8, 2015

                                                                                                      Attendees: Linda, Russell, Konstantin, Dan, David, Brendan, Tom, Steve, Ellen, and  Sarah

                                                                                                      Minutes: Tom

                                                                                                      1. Minutes, changes to the agenda, announcements - None
                                                                                                      2. MINES Report Discussion - 
                                                                                                        1. General Feedback - Everyone thought the new draft looked very polished, it was well done, and there were a lot of positive remarks. The following items were small edits to the draft:
                                                                                                          1. Add difference between sessions and use - make it clear in the introduction that sessions are not equivalent to use. Example of gate counts could be helpful to tell about use .
                                                                                                          2. Lack of BMC category - We will look at adding this to our next survey.
                                                                                                          3. Add full-text library downloads to introduction page - remove library web page views 
                                                                                                          4. Add gate count to the "where" section or introduction from ARL stats - potentially with language about library as space
                                                                                                          5. Look at the 6 paragraph in the introduction - maybe pulling that paragraph up to start the report 
                                                                                                          6. Perhaps review the conclusions to highlight 
                                                                                                          7. Inconsistent use of gradients in graphs 
                                                                                                          8. Check BU Libraries Assessment Committee in report title and throughout the document
                                                                                                          9. Review the chart colors and their look in black and white 
                                                                                                          10. Add paragraph about what online resources we are talking about in the introduction and remove Top Online Destinations
                                                                                                        2. Conclusions - we will look at the conclusions of the MINES report at our next meeting.

                                                                                                      Our next meeting is: Thursday, October 15 from 2:00-3:30 in the Pardee Conference Room

                                                                                                      Assessment Committee Minutes - September 4, 2015

                                                                                                      Attendees: Linda, Russ, Dan, David, Brendan, Tom, Steve, Ellen, Konstantin

                                                                                                      Minutes: Tom

                                                                                                      Reporting for the ARL annual statistics has begun. The deadline is Oct. 1 for ARL and about a week prior for BU. Linda will be coordinating these efforts and will be contacting people appropriately.

                                                                                                      MINES Survey Report:
                                                                                                      We had a great conversation about the David's initial draft for our MINES Survey Report. The group all agreed the initial report was an excellent start. We spent the remainder of the meeting discussing the document's structure, language, and overall purpose.

                                                                                                      Some general thoughts:
                                                                                                      - The group would like to conduct more analysis in research funding section of the document. This analysis might include a breakdown of research funding and library usage by audience and school.
                                                                                                      - We discussed that we need to be mindful that there is not a 1:1 relationship between online sessions and use. Additionally, we should clarify how the MINES data supports and supplements the data we already gather about usage of library resources.
                                                                                                      - We looked at restructuring the sections as: who (use), what (top online), when (use over time), where (location), why (purpose)
                                                                                                      - We determined that any charts information about the print collection should be symmetrical with the MINES data (same audience breakdown, same scale, and legend)

                                                                                                      Potential "high-level" recommendations:
                                                                                                      - Our conversion to "e first" is not leaving any group behind and that we should continue building services around our electronic holdings
                                                                                                      - Library services should assume that patrons are going to access our holdings online and outside of the library's walls
                                                                                                      - Coursework drives undergraduate and graduate use of the library and we should prioritize outreach efforts appropriately
                                                                                                      - We should continue prioritizing staff efforts and distribution around our electronic holdings
                                                                                                      Our next meeting is: Thursday, September 17 from 12:00-1:30 in the large Mugar Conference Room

                                                                                                      Assessment Committee Minutes 7/30/15
                                                                                                      Attendees - Linda (chair), Brendan (minutes), Dan, David, Ellen, Steve, Tom

                                                                                                      Next meeting: Thursday, August 6 from 1:00-2:00 pm (Report Review
                                                                                                      NEXT-next meeting: Friday, September 4 at 2pm

                                                                                                      To do-
                                                                                                      Linda to add Tom to assess-l,
                                                                                                                 to forward him the contact for research assessment at the Provost's office,
                                                                                                             & to research whether an evaluation of (inter)national library surveys has been done.
                                                                                                      Brendan to get circulation statistics for Lib Use Report.
                                                                                                      Dan to get webhit numbers for David
                                                                                                      David to contact Mike Ward about Primo searches.
                                                                                                      Ellen to forward link to Assessment Cookbook notification.
                                                                                                      Everyone to continue working on their initiatives.

                                                                                                      Assessing social media impact (Dan)

                                                                                                      Plan includes a scholarly lit review, student feedback, env scans of peer institutions, and approaches to mgmt.
                                                                                                      Include a collection of example posts.
                                                                                                      Finish gathering by November this year.
                                                                                                      Interviewing to finish March next year. (contact Mary McGowan, Ellen to share law contacts)
                                                                                                      Complete report by next year summer..
                                                                                                      Output: explanation of best practices and local policy recommendations.
                                                                                                              Dan mentions BrowZine, a tool for journal browsing.  Expensive but may be useful for faculty.
                                                                                                      MINES/Library Collection Use Report (David)
                                                                                                              Next meeting will be devoted to group review of the draft.
                                                                                                              Dan, Brendan, and Steve to assist with collecting statistics for the report.
                                                                                                              Looking to finish the report by second or so week of September.
                                                                                                      Data Security (Tom, Ellen, Brendan)
                                                                                                      we need an index of who's collecting what, and contact data.
                                                                                                              output: How libraries organize our data.
                                                                                                      Linda suggests putting our data in our institutional repository.

                                                                                                      Data Visualization (Tom, Brendan, Konstantin*)
                                                                                                              Efforts made to visualize SEL headcount data. Preliminary graphs will be made more interactive.
                                                                                                      Usage Stats (Steve)
                                                                                                      We need to set standards for how to put data into Alma.
                                                                                                      Collab between Steve, Ellen, maybe Jennifer and Tim.
                                                                                                      Will start working over the fall.
                                                                                                      Lots of stuff (Ellen)
                                                                                                      ACRL sent out request for "Assessment cookbook".  Due date is 8/6/15.  Ellen to forward link to all of us.
                                                                                                      Idea: putting together a professional reading collection.
                                                                                                      Steve - why not make a lib guide for them?
                                                                                                      Cornell U Library research and assessment unit - Check it out.
                                                                                                      InCites (Linda)
                                                                                                      Provost decided not to renew InCites.  They like Academic Analytics (strictly faculty).
                                                                                                      Library may push to get access to Academic Analytics.  We may be able to work with the company to get lib-interesting data.
                                                                                                      New Idea - Look at other national/international surveys (only a few) and institutions sharing the same assessment genealogy. Establish reasons 
                                                                                                                  for why we have assessed the way we have.
                                                                                                      Linda talked with Jillian "director of learning assessment for BU". Expect more communication in the future.
                                                                                                      Timeline, realistically start after mid October. output a report like MINES and recommendations.
                                                                                                      David - This should be published if it hasn't been done before.
                                                                                                      Data Storytelling Workshop (Tom & Brendan)- 
                                                                                                      Designed for journalism fields interested in data. It's a new field meaning we're not behind the trend.
                                                                                                      Showed off techniques for getting leads on stories, searching for datasets, cleaning data with tools.
                                                                                                      Most important lesson: keep focus on people. Add stories to the data to engage.
                                                                                                      Libraries stand to inform people about data storytelling ourselves, workshops, guides (timed with seminar papers maybe).
                                                                                                      Journalism students may be a useful asset for us.


                                                                                                      Assessment Committee Minutes 5/29/15
                                                                                                      Attendees - Linda (chair), Dan (minutes), Brendan, David, Tom H, Steve

                                                                                                      Next meeting: Thursday, June 18 at 2pm

                                                                                                      Announcements: none

                                                                                                      1.  Data management (Brendan & Tom)

                                                                                                      Brendan and Tom presented a slideshow about the Data Management initiative. They examined at data classifications, possible data roles and overall goals at the Libraries.  They are planning to construct data policies that encourage folks to work with the Libraries on data issues.  Data permissions, training from IS&T for OneDrive, and centralizing library data were also discussed.

                                                                                                      2.  InCites (Linda & Steve) 

                                                                                                      Linda and Steve lead the discussion of this tool.

                                                                                                      Initiative: Exploration of InCites as an assessment tool
                                                                                                      A report about InCites addressed to the Library’s Assessment Committee will be written this summer. It will include a purpose statement, a description of InCites, an estimate of the current usage, and the potential of InCites for the Assessment Committee. The report will include recommendations for the Assessment Committee and the Library’s outreach efforts.
                                                                                                      Target Due Date:
                                                                                                      Preliminary Outline

                                                                                                      • Purpose
                                                                                                      • Description of Resource
                                                                                                      • Target Audiences
                                                                                                      • Limitations
                                                                                                      • Current Usage at BU
                                                                                                      • Environmental Scan of Use by Other Universities/Libraries
                                                                                                         - Literature review
                                                                                                         - Questions posted to assessment listservs, with follow up and summary about use (ARL, BLC)
                                                                                                      • Recommendations

                                                                                                      3.  Assessing Social Media Impact (Dan)

                                                                                                      This project will be a preliminary look into social media use best practices for libraries, and will result in a document that includes:

                                                                                                      • a scholarly literature review,
                                                                                                      • examples and feedback from students,
                                                                                                      • an environmental scan,
                                                                                                      • and approaches at various BU Libraries.

                                                                                                      "Being active in the social media sphere is important because it provides innovative ways for us to connect with users we may never see face to face on a personal and meaningful level." Fichter, Darlene, and Jeff Wisniewski. "How To Measure the Results of Your Activity on Social Media Sites." Marketing Library Services 23.2 (2009). 


                                                                                                      Assessment Committee Minutes 5/5/15
                                                                                                      Attendees - Ellen, Dan, Konstantin, Brendan, David, Tom, Steve, Linda (chair)

                                                                                                      Next meeting: Friday, May 29 at 1pm



                                                                                                      1.       Minutes, announcements, changes to the agenda

                                                                                                      2.       FY16 Initiatives

                                                                                                      1.        Organize and describe our data sources. (Ellen, Brendan, Tom)
                                                                                                      2.        MINES reports (David) 
                                                                                                      3.            Data visualization (Brendan & Tom)
                                                                                                      4.        InCites (Linda & eventually Steve)  
                                                                                                      5.        Impact of social media (Dan) 
                                                                                                      6.        Ongoing tracking  of usage statistics for e-resources- U-Stats (Steve)
                                                                                                       Next meeting  

                                                                                                       The Assessment Committee welcomed a new committee member, Tom Hohenstein.

                                                                                                       The following discussion has taken place on goals, timeline and scope FY16 initiatives. We've discussed data usage (f.ex. CRC vs. Medical, Law databases, etc.), data services (reference instructions, floor plans, hours, etc.), sources for data (Google forms, LibGuides, IPADS survey)

                                                                                                       Updates on ezProxy, Web of Science, Academic Analytics (not yet available to BU Libraries).

                                                                                                       Other topics we brainstormed:  comparison of our data to outside sets (f.ex. rationalization and normalization of our data in the context of outside data); collection of data (f.ex. focus groups - see also Primo Advisory Group, which right now focuses on usability testing); audience and appropriate products; resources and tools at BU (Hariri Institute for Computing and Computational Science and Engineering, Digital Initiatives, etc.)

                                                                                                      In the next 15 minutes the group brainstormed ideas for the FY16 Initiatives (see list above)

                                                                                                      1.       Organize and describe our data sources. (Ellen, Brendan, Tom)

                                                                                                      2.       MINES Reports (David)

                                                                                                      3.       Data visualization (Brendan & Tom)

                                                                                                      4.       InCites (Linda & eventually Steve)

                                                                                                      5.       Imapct of Social Media (Dan)

                                                                                                      6.       Counter complianbt denials

                                                                                                      7.       Ongoing tracking  fo Usage Statisitics for e-resources- UStats (Steve)

                                                                                                      8.       Environmental Scan of Data being collected by BU Libraries

                                                                                                      Reminder: Anita Green Scholars - starts sooner this year

                                                                                                      Assessment Committee Minutes 4/17/15
                                                                                                      Attendees—BD, TH, SS, JR, JA, RS, EF, DF, KS (via skype), MW, LP, TL

                                                                                                      Next meeting: Monday, May 4 at 2PM. Focus: substantive conversation on what type of data we want to use.


                                                                                                      1. Brendan to report re EZProxy Logs meeting with Scott Macomber
                                                                                                      2. Linda re Eddie from Sourcing on use of Tableau
                                                                                                      3. Discussion re use of priorities for next yea


                                                                                                      Brendan re Scott Macomber & EZ-Proxy Logs

                                                                                                      • EZProxy records two types of files— .log (records everything) and .spu (records first place they land after login)
                                                                                                      • Mines captures URL data approximately every 100th person.
                                                                                                      • Not totally clear what EZproxy captures, Scott is working with OCLC to sort this out. 
                                                                                                      • Brendan has 2 files—log for Jan 1 has 86,000 logs. BD discussed cleanup process for .log files—there is a lot to be done, so we would need to limit amount we asked for/looked at
                                                                                                      • .log files show username, so needs to be removed. .log captures all webpages opened/any motion by user id (i.e., when they click the back button)
                                                                                                      • .spu captures first session through EZProxy i.e., if you went to multiple BU resources in one session, it only captures the first one
                                                                                                      • Brendan will compare host files to MINES and see if there is anything we can pull. Also going to tie to Google Analytics data to assess search path based on timestamp to educate us on how to improve our search capabilities. 

                                                                                                      Should we really do this? Is it worth it? What questions are we trying to answer?

                                                                                                      • BD recommends one year, max b/c of time involved.
                                                                                                      • JA recommends running a call with Python to pull user group before stripping username, then TH recommends scripting to automate
                                                                                                      • BD does not recommend options listed on OCLC website due to age and ease of use, so we are likely looking at scripting something by hand
                                                                                                      • Side note: JA is not sure how long we will have EZProxy, b/c depends on vendor requirements
                                                                                                      • Can’t get user group info from elsewhere, but what do we get out of that? 
                                                                                                      • SS is not sure how he would use resource info for collection development.
                                                                                                      • Ustats does already give us stats on what is downloaded. 
                                                                                                      • How are people finding our resources? Google Scholar? Primo? 
                                                                                                      • User behavior is interesting, but could this be gathered through snapshots of log, rather than analysis of all the data 
                                                                                                      • RS: we don’t have COUNTER-compliant data, so this would help us some representation of what is being used for renewal decisions; would also help with ARL questions re usage stats. LP—we need COUNTER compliant data. RS—OCLC Parse holds itself out as COUNTER-compliant. JR—reports themselves are not COUNTER-compliant, they just use COUNTER recommendations/format

                                                                                                      BD, DB, and LP met with Eddie from Sourcing re Tableau

                                                                                                      • He is willing to be a reference person for our use of Tableau
                                                                                                      • We don’t know the other products for data visualization, so not recommended we make any decisions just yet
                                                                                                      • Looking at this for presenting information in meetings with staff, university administration, etc.
                                                                                                      • It is very expensive, although it can be setup to grab data from multiple sources, but we may not neneed this functionality 
                                                                                                      • Meeting at 2PM Tuesday on Tableau, can be viewed in Mugar Admin Conference Room if interested 
                                                                                                      • BD to ask Rachel Lewellen of UMass Amherst if she will share some of her MINES/Tableau data 
                                                                                                      • We will hold off on purchasing Tableau for now because it has a steep learning curve, and we don't currently have an exact need for its functionality 

                                                                                                      Key focus for next year: data and getting it out to users


                                                                                                      Assessment Meeting Minutes 
                                                                                                      Ellen, Dan, Konstantin, Brendan, David, Sarah (minutes), Steve, Linda (chair)

                                                                                                      Next meeting 4/17 (Friday) 2-3:30pm. Linda will send out room info.

                                                                                                      Eddie from Sourcing is using Tableau, coming to meet with some of group 4/7/15 2pm small conf rm

                                                                                                      Reviewed what was covered in last mtg

                                                                                                      ARL SPEC kits – Konstantin
                                                                                                      ~4/year on hot topics are accepted. Written by ARL member libraries generally. All ARL libraries are asked to fill in survey, then data compiled and report written by authoring institution. Access to 2003-current will require a subscription. Earlier years are FT in Hathi trust
                                                                                                      2015 topics include one on scholarly output assessment activities
                                                                                                      Steve notes that in the past the examples/best practices included in SPEC reports were one of the more useful parts. So just the TOCs can be useful to know who might have examples (that may be available for free from their sites).
                                                                                                      May be worthwhile to see what assessment ones have been done.

                                                                                                      David – Analytics
                                                                                                      Look at patron searches in BULS, can this inform reference and instruction?
                                                                                                      Mike Ward says google analytics of page are currently better than Primo/ex Libris built in ones. No examples or models exist that DF found to assess the queries (how to do it, or if it’s useful). DAGS may be using some small parts of this for improving searches/results/displays
                                                                                                      DF doesn’t think it’s a high priority now, might be interesting to look again in a year when ex Libris analytics may be better.

                                                                                                      Brendan – MINES
                                                                                                      “We downloaded all the data from MINES, and there’s a lot of it” - over 7K responses. Completion rate around 91% last time it was calculated. BD cleaned up some bad data (probably statistically insignificant anyway). Added some databases hit based on connection URL for Ebsco. Discussed having BD create a couple other graphs. DF, BD, and LP will work on how to create reports from this data. DF- audiences: brief report for outside/above library, more detailed for internal use or sharing with other libraries. Potentially present to grad research council. Adding in data from outside survey? (# of potential users per school, etc). need context for reader in reports. What would we do differently if we ran MINES again? Longitudinal snapshot?
                                                                                                      Should BD work to draw out more detailed specific databases from aggregators? – consensus was no b/c of potential quality of data issues.

                                                                                                      Dan –Tableau Trial
                                                                                                      Uploaded some spreadsheets (ebrary stats Steve had sent out) and then explored some graphical display options. Lots of online support. Dan did look at a few example videos, but mostly was just exploring. Showed us 3 visualizations he’d worked on. Definite interest in learning more about it.

                                                                                                      Assessment Meeting Minutes 03/09/2015      
                                                                                                      Present: Linda (head), Brendan (minutes), Dan, David, Ellen, Sarah, Konstantin, Russ, Steve

                                                                                                      Next Meeting: Monday 03/23/15, Mugar Conference Rm.

                                                                                                      Action Items:

                                                                                                      • Linda: Ensure student permissions for archived publicity files.
                                                                                                      • Sarah: Send out fact sheet files and publicity contact list.
                                                                                                      • Brendan: Get EZProxy logs from Scott Macomber and fiddle with the data.
                                                                                                      • Everyone: Think more about the future of the committee.

                                                                                                      • Nothing new from Provost -> 2015 Library Survey will be postponed until 2016.  The surveys and how to implement them are being saved to a read-only dropbox archive for future reference.
                                                                                                      • Brendan showed the structure and contents of the archive.  Showed the contents of the survey manual.

                                                                                                      Individual Research Topics
                                                                                                      • Tableau and Data Warehouses [Dan]
                                                                                                        • Tableau is more visualization-driven than other data analyzers.
                                                                                                        • Rachel Lewellen of UMass Amherst does a lot with Tableau.  Uses it for presentations.
                                                                                                        • It's potentially another tool people will need to learn to use and manage among all the other tools people should learn to use, but don't.  A data warehouse may be wasted effort, and moreover money.
                                                                                                        • People don't want unprocessed data.  Give them facts and graphs that are useful by themselves.
                                                                                                        • Individualized dashboards would be the most effective, but they're labor intensive.
                                                                                                        • The Anita Greene Scholars are great graphic designers, but we need to schedule time with them since they're in high demand.
                                                                                                        • BU Libraries don't have a single go-to person for statistics and data (like Rachel Lewellen).  We should reconsider this.
                                                                                                      • Fact Sheets [Sarah]
                                                                                                        • Some fact sheets tell a story instead of numbers.  This is a good model for public fact sheets
                                                                                                        • We can compare library rankings and university rankings to highlight library contributions to BU's success.
                                                                                                        • Consider tying our delivered stats to the University strategic plan.
                                                                                                        • What data we share and how we share it depends on the audience.  A statistic filled fact sheet for sharing with libraries and provost; applied department-oriented figures for colleagues; fun and informative infographics for the users.
                                                                                                        • Maybe a newsletter would be useful for internal audiences.
                                                                                                      • EZProxy Usage Stats [Russ]
                                                                                                        • Usage is logged, but not analyzed.  "Does it log each link, or only each login?"
                                                                                                        • Program "EZParse" could be useful for analysis.
                                                                                                        • We may be able to get the total logs from Scott Macomber, or capture logs from a set of library computers.
                                                                                                        • Usable as a check against the MINES data.
                                                                                                        • Some concern that our EZProxy format will be difficult to analyze by database or journal.
                                                                                                        • Brendan will look into the logs by next meeting.
                                                                                                      • U-Stats and Cost per Use Data [Ellen & Steve]
                                                                                                        • Cost per use difficult to measure; depends on how we define use.
                                                                                                        • What data should we be collecting and how should we store it?
                                                                                                        • Consider creating a "best practices" procedure for data.
                                                                                                        • We should look at how other schools handle data.
                                                                                                        • There's a problem porting data between Ustats and Alma.  Alma Analytics may be the intermediary.
                                                                                                      • OUT OF TIME!  Next week: Konstantin with the ARL Spec Kit, David with "Analytics and Us", and a MINES analysis update from Linda and Brendan.  Next meeting: 3/23 in the Mugar conference room.

                                                                                                      Assessment Meeting Minutes 02/23/2015     
                                                                                                      Present: Linda (head), Dan (minutes), Brendan, David, Ellen, Sarah, Konstantin, Russ
                                                                                                      Next Meeting:
                                                                                                      Monday 03/09/15, Admin Conf Rm.


                                                                                                      • The opt-out open access policy passed Univ. Council.
                                                                                                      • Russ is leaving the committee; Ellen will be continuing to represent Law.
                                                                                                      Action Items:
                                                                                                      1. All - monitor for 2015 Survey developments as appropriate.
                                                                                                      2. All - be prepared at the next meeting to discuss future directions as assigned below.

                                                                                                      Update on Survey 2015

                                                                                                      We are starting to run low on time to accomplish the 2015 survey this Spring.  If we are not able to accomplish it, our new target would become Spring 2016.  As this is a distinct possibility, all of our materials are now well archived (thanks Brendan) in Dropbox and other locations.  There is also a "readme" file that explains what documents and materials can be found in the archive, and describes their purpose.

                                                                                                      Discussion of future work
                                                                                                      At the university level, there is a renewed focus on assessment systems for the evaluation of learning outcomes, and the library will be participating in this in the near future.  In light of this Committee’s charge and goals and the Libraries’ strategic plans, what other assessment activities would improve library services and resources.
                                                                                                      • [Ellen] - U-Stats.... cost per use data....  (discuss with Steve, and Jennifer from Law)
                                                                                                      • [Konstantin] - group of specialists to respond to the ARL Spec Survey > ARL Spec Kit
                                                                                                      • [Dan] - centralized point of access for all things library assessment (data warehouse?  Tableau? see * below)
                                                                                                      • [Sarah] - creation of Fact Sheet(s) for university constituents, such as bu.edu/research/  (needs examples, environmental scan)
                                                                                                      • [Linda] - MINES - now one year old at BU
                                                                                                      • [Russ] - investigate what ezproxy logs can tell us about use (and possibly save to U-Stats)
                                                                                                      • [David] - what can google analytics tell us about our library website users?  about our BU Libraries Search users?  about collection use (and possibly save these to U-Stats)?
                                                                                                      • Social Media outreach efforts - tracking twitter, facebook accounts
                                                                                                      • Know just what we do collect and should let decision-makers know vs. what should we be collecting and should let decision-makers know
                                                                                                      • leadership role for all library assessment activities
                                                                                                      • assessment tied to GPA trend
                                                                                                      • focus groups

                                                                                                      * From the ARL-ASSESS listserv:

                                                                                                      Data Management and Visualization Webcasts

                                                                                                      Data visualization brings library stories to life. Using Tableau software, libraries may better harness, analyze, and report their data to internal and external stakeholders. Join us for a three-part series on data management with Tableau, presented by experienced assessment librarians.

                                                                                                      Sarah Murphy, Ohio State University

                                                                                                      Tuesday, March 3, 2015, 2:00–2:30 p.m. EST

                                                                                                      Register online
                                                                                                      free of charge

                                                                                                      The Ohio State University (OSU) has used Tableau since 2012 to support a number of assessment projects. Examples include a series of interactive dashboards that query, analyze, and deliver transactional data to subject librarians to support their collection development and engagement activities. To deliver library data of interest to the broader academic community, the Libraries has also started to embed interactive, downloadable Tableau dashboards via select OSU Libraries websites.

                                                                                                      Sarah Anne Murphy is coordinator of assessment for The Ohio State University Libraries. She earned an MLS from Kent State University in 2000 and an MBA from The Ohio State University in 2008. She has published two books and several papers related to library assessment.

                                                                                                      Jeremy Buhler, University of British Columbia
                                                                                                      Tuesday, March 10, 2015, 2:00–2:30 p.m. EDT
                                                                                                      Register online
                                                                                                      free of charge

                                                                                                      The University of British Columbia (UBC) Library began using Tableau in 2013 to analyze longitudinal LibQUAL+ results. The resulting online display highlights changes in survey results over time and facilitates the comparison of responses across several user groups. Building on the success of this visualization, UBC Library is now using Tableau to encourage library staff to engage with, think about, and find new applications for library metrics.

                                                                                                      Jeremy Buhler, MLIS, is assessment librarian at the University of British Columbia Library, where he has worked since 2011.

                                                                                                       Rachel Lewellen, University of Massachusetts Amherst
                                                                                                      Tuesday, March 17, 2015, 2:00–2:30 p.m. EDT

                                                                                                      Register online
                                                                                                      free of charge

                                                                                                      Tableau helps the University of Massachusetts (UMass) Amherst Libraries staff to both analyze and understand data. Selectors may access purchasing and use data without having to maintain or manipulate spreadsheets. Using Tableau for the analysis of a consortial e-book project minimizes duplication of effort and provides a common perspective for understanding the data. Interactive visualizations expedite access to live data from MINES (Measuring the Impact of Networked Electronic Services), and make it easier for staff to use circulation, service desk, and gate-count data for operational purposes.

                                                                                                      Rachel Lewellen is assessment librarian at the UMass Amherst Libraries. She has been using Tableau since 2012

                                                                                                      Tuesday, April 21, 2015

                                                                                                      2:00–3:00 p.m. EDT

                                                                                                      Webcast—Data Management and Visualization with Tableau

                                                                                                      Discussion and Q&A with Sarah Murphy, Jeremy Buhler, and Rachel Lewellen


                                                                                                      Brendan, Dan, David (chair pro tem), Ellen, Russ, Steve (minutes)

                                                                                                      Next meeting: 2pm, 2/9, Mugar conference room

                                                                                                      To do:

                                                                                                      -Steve will serve as Sarah's backup to resolve the capitalization of Library, Libraries, library, libraries through the three surveys

                                                                                                      Bob Hudson updated the committee on the approval status, that the Provost has been given copies of the surveys to review.

                                                                                                      Survey pilots: Went well, with almost all 'piloteers' taking and completing the survey.

                                                                                                      The committee then reviewed comments from the pilot (see Brendan's 'Review of Encountered Pilot Errors and their Fixes' handout) and corrected errors as necessary.


                                                                                                      Dan, Russ, Brendan, Steve, Ellen, Konstantin (minutes), David, Linda (chair)

                                                                                                      Next meeting: 2pm, 1/26, Mugar conference room

                                                                                                      To Do: 

                                                                                                      - Konstantin and Ellen - identify 2 staff who are willing to participate in the pilot

                                                                                                      - collect and review responses from pilot testers

                                                                                                      -Sarah still needs to resolve the capitalization of Library, Libraries, library, libraries through the three surveys

                                                                                                      We’ve reviewed display option on the iPad.  It looks OK.

                                                                                                       Discussion of remaining issues:

                                                                                                      QF15, QG2a: Open Access -  we’ve agreed to the changes for points one and two regarding articles published

                                                                                                      Resolved the wording for library search – FQ5 -  copy from FQ5, GQ7 and UQ6 - 

                                                                                                      FQ6, GQ6, UQ6 -  our agreement: “Research assistance …”  “General assistance”

                                                                                                      FQ9 and QG13 - “archiving papers and other publications in digital archives” – include SSRN

                                                                                                      “Dear Faculty …” text is finalized

                                                                                                      We’ve finalized the email list for pilot participants.

                                                                                                      Include some staff to participate in the pilot – Linda will email the link


                                                                                                      Dan, Russ, Brendan, Steve, Ellen, Linda (chair), David (minutes)

                                                                                                      Next meeting: 2pm, 1/20, Mugar conference room

                                                                                                      To Do

                                                                                                      • Ellen: Check survey on iPad
                                                                                                      • Russ: Propose wording for questions about subject librarians/staff assistance/research assistance


                                                                                                      Linda will be on vacation Jan. 26th - Feb. 8th

                                                                                                      Should know Provost decision Jan 22nd.


                                                                                                      Looked at Linda’s proposed emails for survey recipients. Ellen had minor edit that was accepted and made by Brendan.

                                                                                                      For pilot, Brendan recommends that we we just use a non-individualized link for pilot. Can test panels separately amongst ourselves.

                                                                                                      Brendan looked at display logic affecting data; there should be no real problem.

                                                                                                      There was some editing of Linda’s emails for pilot participants, recorded by Brendan.

                                                                                                      Which surveys should staff see in the pilot? All of them.

                                                                                                      Linda brought up possible issue with viewing survey on iPad? David couldn’t reproduce on his iPad. Ellen will check it on hers.

                                                                                                      Went over various comments and suggestions for surveys. Brendan captured changes made.

                                                                                                      Russ will propose wording for Subject librarians/staff assistance/research assistance.

                                                                                                      Will use student workers for undergrad pilot.

                                                                                                      Next meeting is all about the pilot, and we plan to send it out after meeting.


                                                                                                      Linda (chair), Brendan (minutes), Steve, Dan, Sarah, Ellen, David , Konstantin, Russ

                                                                                                      Next mtg: 01/12/15 2:00-3:30, Lg. conf. rm.

                                                                                                      To do

                                                                                                      • Sarah: Ensure Libraries is capitalized/punctuated properly across surveys
                                                                                                      • David: Check for wording consistency across surveys.
                                                                                                      • Dan: Respond to Ken about our decision on splitting up teaching/research/clinical.
                                                                                                      • Linda: Draft up emails sent out with the pilot and the real survey.
                                                                                                      • Brendan: Review survey options, flow, and distribution plan.  Determine if display logic will mess up analysis.  Prepare pilot panels.  Modify surveys for pilot commenting.
                                                                                                      • EVERYONE!: Review ALL three surveys, and try to find 3 undergrads, grads, and faculty each (as appropriate), and email their info to Brendan.

                                                                                                      AGENDA: 1. Review previous meeting 2. Plan pilot


                                                                                                          BD finished changing the surveys.  Questions have been updated as discussed.

                                                                                                          Linda made chart of each survey's section and question flow.  It all looks good and makes sense.


                                                                                                          Linda likes convenient sampling.  “Try 3-5 each"

                                                                                                          We need people from both campuses.

                                                                                                          Steve mentions that data may be difficult to parse due to display logic.

                                                                                                              Brendan will check if this applies to us.

                                                                                                          Panels will be created for the reviewers

                                                                                                          For undergrads, don't use student workers, but their friends are okay.

                                                                                                          We should mention we’re collecting demos in the intro.

                                                                                                          Show Marlene, Mary, and Amy the survey before pilot.  “We thought you’d like to see this before it goes out.”

                                                                                                          Everyone’s Spring recess ends after March 15th. Target release date: April 2nd.

                                                                                                      Advertising Update

                                                                                                          Sarah showed off the adverts.  Very nice.  Still some tinkering here and there.  Lots of places we’re advertising, still waiting on even more locations.

                                                                                                          Russ: "Can we modify our websites for advertising the survey?"  Linda: “IRB says about 1 advert in each library is fine.  More than one will shift our user group         toward the patrons."

                                                                                                      Scheduling: Mondays 2-3:30 in the big conference room for the next 5 weeks, EXCEPT January 20th, 2-3:30 instead of January 19th. 

                                                                                                      Meeting adjourned.  


                                                                                                      Steve, Dan, Brendan, Sarah (minutes), Ellen, David (Chair), Konstantin

                                                                                                      next mtg: 1/5/15 2pm conf rm.


                                                                                                      Pilot Surveys

                                                                                                      OA Questions

                                                                                                      SDS presented new wording for OA questions on grad and fac (FQ36, GQ2a)

                                                                                                      Regarding articles you have published: [Respond Yes/No/Unsure]

                                                                                                        • Have you published an article in an open access journal?
                                                                                                        • Have you paid an additional fee to make an article open access?
                                                                                                        • Have you deposited an article in OpenBU (BU's institutional repository)?
                                                                                                        • Have you deposited an article in a discipline-based repository (e.g., PubMed Central, SSRN, arXiv)?

                                                                                                      On the faculty survey, the open access factor on the question "How important are the following factors in your decision on where to publish journal articles?" will read: 'Journal allows open access (immediately or after an embargo)'.

                                                                                                      DF suggested a small wording change: move “have you” after the “regarding articles you have published” and start each question with the verb (published, paid, deposited).

                                                                                                      Info lit Questions

                                                                                                      DB reports from conversation w/ Ken Liss

                                                                                                      already implemented wording on guides/tutorials

                                                                                                      separating teaching from clinical/research purpose (FQ 32,33,43,45) - KL interested, we were not open to making this large a change at this time

                                                                                                      consider in survey keepers- separating teaching from research from fac, or teaching from study for grads.

                                                                                                      Streaming Media

                                                                                                      GQ17 (could be added to fac FQ 45 as well) Streaming Media

                                                                                                      SDS - yes wants this question

                                                                                                      fac interested in JOVE; Krasker DVD based, is there interest in other?; Lots of audio, not much video currently

                                                                                                      BD has concern about “streaming media” wording could be interpreted different ways (synchonous vs asynchronous, live streaming vs netflix style)

                                                                                                      switch to “online video and audio resources”?

                                                                                                      “streaming audio and video” 

                                                                                                      decided on wording:

                                                                                                      “Access to streaming audio and video resources” for both GQ17 & FQ45

                                                                                                      Review of Questions/issues from BD’s email 12/15

                                                                                                      discussed, BD will make changes

                                                                                                      Assessment Meeting Minutes 12/15/14

                                                                                                      Attendees:  Linda (head), Dan, Brendan, David, Ellen, Steve, Sarah, Konstantin (minutes), Russ, Vika (called-in)

                                                                                                      Next Meeting: Monday, December 22, 2pm Admin. Conf. Rm.

                                                                                                      NEXT MEETING: Agenda

                                                                                                      1.            Question “streaming media” 
                                                                                                      2.        Report from Open Access group  (Sarah and Steve)
                                                                                                      3.         Discuss Ken’s second suggestion
                                                                                                      4.        Respond to Brendan’s email (all)

                                                                                                      12/15/14 Agenda

                                                                                                      1.       Minutes (thanks Ellen for the terrific minutes), announcements, changes to the agenda?

                                                                                                      2.       Discussion with Vika about our OA questions (Sarah)

                                                                                                      3.       Brendan’s overview presentation and outstanding questions (Brendan)

                                                                                                      4.       Discussion about Ken’s feedback on the survey’ Information Literacy questions (Dan)

                                                                                                      5.       Publicity files, Brendan’s suggestion about publicity

                                                                                                      6.       Do we want to add an augmented services question to the undergrad survey?

                                                                                                      7.       Pilot

                                                                                                      Brendan’s overview:

                                                                                                      Brendan walked us through all the major and minor changes. (see Brendan’s power point presentation). 

                                                                                                      The group discussed question “During that past academic year at BU have you …. 4th option: “Worked on a research project including data”.  We’ve decided to keep the wording as is.

                                                                                                      Double check if IS&T term is clear to the medical users -  include BU/BUMC IS&T ? …. (Konstantin)

                                                                                                      We’ve discussed the advantage of mentioning “Blackboard” in a question which asks about “course system”

                                                                                                      Sarah brought up the point that Ken’s suggestion that library guides need to be reincorporated in the undergrad and graduate surveys.

                                                                                                      We’ve discussed open issues concerning service augmentation.  Russ suggested to change the word “augmented” to “enhance”. We all agreed.

                                                                                                      We’ve decided to discuss the potential of removing “media streaming” during the next meeting.


                                                                                                      Discussion with Vika about OA questions:

                                                                                                      Sarah summarized Vika’s concerns and Vika clarified that we shouldn’t focus just on “open access journals” vs. open access in general.

                                                                                                      Vika suggested to formulate a separate question that asks about Gold and Green access.

                                                                                                      Discussion followed. Steve argued that we’re trying to find out if our patrons are making their work open access rather than publishing in the “open access” journal.

                                                                                                      We’ve decided to modify the question so that we capture “immediately or after embargo” wording.

                                                                                                      Question “Have you ever submitted an article to: ….” -  We’ve decided that we have to consider the wording in the context of Vika’s suggestion.

                                                                                                      Sarah emailed Vika’s suggested wording.

                                                                                                      Ken's suggestions

                                                                                                      Dan discussed Ken's suggestions, but we still need to revisit them.

                                                                                                      Augmented Services for the Undergraduates

                                                                                                      Russ suggested that we substitute the word "augmented" with "enhanced". The group unanimously agreed.


                                                                                                      Assessment Meeting Minutes 12/8/14

                                                                                                      Attendees: RS, ER (minutes), LP, BD, DB, SS, DF (for last half)

                                                                                                      Next Meeting: Monday, December 15, Admin. Conf. Rm.

                                                                                                      We focused on a group review of the completed graduate and faculty surveys

                                                                                                      -Grad & Faculty Surveys will NOT have a “demographics” bloc header—decision is that is it not necessary to prep users to answer personal questions.

                                                                                                      Sarah is setting up a phone in with Vika for next Monday’s meeting to discuss the Open Access wording on the survey and what data she would like us to collect.

                                                                                                      Things to cover at next meeting: Open Access questions and language with Vika, input from Ken on Information Literacy, conventions/consistency of language across all three surveys with Brendan, review the three surveys in their entirety, data management questions with David, and discuss the pilot


                                                                                                      On satisfaction and importance questions, IMPORTANCE rating will be asked first, and then SATISFACTION rating

                                                                                                      -Brendan is looking at alternative phrasing for “information literacy” and also looking at the three surveys together to see where language should consistent, but isn’t—he will present these at 12/15 meeting

                                                                                                      Faculty Survey

                                                                                                      -David is working on language for Research Data questions

                                                                                                      -“Research data” option has been added to question about “during the past academic year, have you…” and set up as display logic, so only survey takers who have worked with research data will see questions about storage and sharing

                                                                                                      -Question on bibliographic managers may be moved to different block, but this was not yet decided

                                                                                                      -Question on assigned readings and how students access—to be reviewed with Steve and Konstantin at 12/15 meeting

                                                                                                      -Dan is following up with Ken on the language for the information literacy question (FQ30)—this language must be to us by 12/15

                                                                                                      FQ32—discussed switching assessment and importance (to importance and assessment) but decided not too, since it would be leading, but are aware we may now be comparing apples and oranges

                                                                                                      -“Information literacy” is library speak, but may have to remain

                                                                                                      Grad Survey

                                                                                                      -We agreed to remove “other” from the importance & satisfaction re library resources questions

                                                                                                      -We agreed to add an option on current awareness to our question on augmenting library services—this question will also be copied to the faculty survey

                                                                                                      -We changed the question on library use and space (group and individual study space, access to computers) to focus on HOW graduate students use the library resourceà this change needs to be copied over to the undergraduate survey

                                                                                                      Assessment Meeting Minutes 11/17/2014     
                                                                                                      Present: Linda (head), Dan (minutes), Brendan, David, Ellen, Steve, Konstantin, Russ
                                                                                                      Next Meeting:
                                                                                                      Monday, 11/24, Admin Conf Rm.


                                                                                                      • Discussion of the Information Literacy questions with Ken Liss is delayed for one week.  Ken will join us next week and he and Sarah will lead that discussion.
                                                                                                      • The 2015 Undergrad Survey is tentatively complete.
                                                                                                      Action Items:
                                                                                                      1. Faculty Survey Subgroup should note the discussions in the high level review below and make necessary edits.
                                                                                                      2. Undergrad Survey Subgroup should be prepared to walk the group through a preview of that survey at the next meeting.

                                                                                                      High Level Review of Faculty Survey 
                                                                                                      • On Q32 (importance of resources) - The subgroup should remember to add the satisfaction scale to these (but not the use scale, as this is the faculty survey).
                                                                                                      • We decided to move the section on publishing & OA to near the end, but not past the last 'Overall' section.  Brendan accomplished this during the meeting. 
                                                                                                      • On Q43 (importance of services) - The subgroup decided to add Q44 (the satisfaction with same services) to this question, positioned horizontally next to it.  Not yet accomplished.
                                                                                                      • On Q45 (augmenting services) - We added a data management "statement" (note: this is the Qualtrics term for a row within a question).  The subgroup will work out the examples given and match the wording of the question on the grad survey.
                                                                                                      • For the Info Lit block of questions, the subgroup (actually, all subgroups) will monitor the results of our coming conversation(s) with Ken.  We did note it would be great to fold these 4 questions into 2.  Also noted the 'Providing correct citations' statement became 2 statements for the 2012 grad survey, and really could be 3 (providing appropriate citations, providing correctly formatted citations, and using information in an ethical manner).
                                                                                                      • On Q47 (overall contributions) - The scale needs to change to match the grad survey's scale.  We also added an "overall career success" statement.
                                                                                                      • At the end of the survey we considered other questions (Brendan noted these in Q51):
                                                                                                        • Put Bib manager question on all surveys (see undergrad).    
                                                                                                        • Take Faculty version of reserves question. "How do you direct students to resources you assign?"
                                                                                                        • Don't need first part of bib manager questions. Reevaluate options.


                                                                                                      Assessment Committee Meeting, Monday, November 10, 2014

                                                                                                      1.       Minutes, announcements, changes to the agenda

                                                                                                      a.       No changes were made to the minutes of the last full committee meeting submitted by Ellen

                                                                                                      b.      The following announcements:

                                                                                                                                                                     i.      Brendan noted a possible completion rate problem in the MINES survey, but he observed—and the rest of us agreed—that it did not represent a significant problem.  Even we had to accept the lower rate, it was still quite high and not worth intervention.

                                                                                                                                                                   ii.      Brendan also noted that in Qualtrics we can make notes and comments within the draft questions themselves, allowing us to keep all of the committee’s work product there and make it more convenient to read everything in one place, not only now but in the future.

                                                                                                                                                                  iii.      Linda announced that Ron Yeany and his team at IS&T have offered to look into whether they can help in extracting useful usage data from Exproxy.

                                                                                                      c.       No changes were suggested to the agenda proposed by Linda.


                                                                                                      2.       Continue discussion about usage, importance, and satisfaction and what changes we should make on our surveys.

                                                                                                      a.       Continued and reviewed the ongoing discussion we have had about whether to use usage or satisfaction in our survey. David and Steve quickly reprised the issues for us.

                                                                                                      b.      Konstantin registered his concerns about using importance and offered the white paper from the Customer Satisfaction Research Institute, along with four other relevant articles in the literature, as food for thought in our consideration when considering the use of importance in our survey.

                                                                                                      c.       Steve noted that if we moved away from importance we would lose the ability for longitudinal study, since we used importance in the first round of surveys.

                                                                                                      d.      We returned to the question if we could not use all three factors in the survey as suggested in the last meeting.  Russ queried if we could use the pilot period to test if the three factors could all be used in the survey without making the question too long and cumbersome, confusing the test taker.

                                                                                                      e.      Linda noted that we may have more room in the surveys for a few longer questions given that we are hoping to collect demographic data from the Registrar and Human Resources, rather than by asking the respondents.

                                                                                                      f.        Sarah observed that if we used all three factors that would give additional insight into patron feedback.

                                                                                                      g.       Consensus:   all three elements will be used in the pilot on questions related to services.


                                                                                                      3.       Demonstration/review of the Graduate Student Library Survey

                                                                                                      a.       Graduate Student subcommittee:  have had a high level discussion and are now having a more targeted discussion.  The subcommittee has met twice and needs additional time to discuss issues, but that can follow this discussion by the full committee.

                                                                                                      b.      A question was raised whether we can use the panels in Qualtrics for the distance education and online learners.

                                                                                                      c.       Question 8:  Brendan wondered if the wording is loaded; Linda did not believe it was; Steve suggested using “Please rate the importance” as a way to resolve the possible loaded nature of the question.  The question was also raised if the elements surveyed are really separate or do they represent a skillset (or many different skillsets).  Sarah suggested getting Ken Liss’s insights on this.   She also recommended inviting him to our next meeting to get his thoughts here as well as on other info literacy questions.

                                                                                                      d.      Question 9: on first line it was recommended to use just BU Library Search and then on the second line use “Databases” and search tools.  Russ and Konstantin also recommended using the examples of HeinOnline and PubMed, as was similarly done on the faculty survey.  We also considered whether we needed new examples of Social Media beyond Twitter and blogs, or need any at all.  The consensus was to keep the two existing examples.

                                                                                                      e.      Question 17:  Sarah wondered if we should not change the wording to be more like UW’s analog question #8, to be more user-centric.  We agreed that the subcommittee in its next meeting would investigate this further and make recommendations to the full committee.


                                                                                                      4.       Progress report on the Faculty Library Survey

                                                                                                      a.       We have had a high level committee discussion and one subcommittee meeting discussion.  Issues in the subcommittee, principally about the use of usage or importance, were then discussed by the full committee at the last meeting as well as in the early portion of this meeting.


                                                                                                      5.       High level review of the Faculty Library Survey

                                                                                                      a.       Additional high level discussions need to occur.


                                                                                                      6.       Next steps?

                                                                                                      a.        Undergrad survey is very close to being finished.  Needs an additional half hour of the full committee’s time.

                                                                                                      b.      Grad survey needs discussion in two more meetings.

                                                                                                      c.       Faculty needs two more meetings, although we may be able to shorten that time inasmuch as we are utilizing language developed in the new grad survey in corresponding sections in the faculty survey.

                                                                                                      d.      Should we include post-docs in our survey, and how?



                                                                                                      Assessment Meeting Minutes 11/3/14

                                                                                                      Present: BD, DF, LP, ER (minutes), SDS, DB, RS, KS

                                                                                                      Next Meeting(s):

                                                                                                      Graduate Subcommittee: Friday, November 7, 1-2:30, Mugar Small Admin Conf. Room

                                                                                                      Undergrad Subcommittee: Monday, November 10, 12:30-2, Administrative Offices, Mugar, small conference room

                                                                                                      Full Assessment Committee meeting, Monday, November 10, 2-3:30, Administrative Offices, Mugar, large conference room


                                                                                                      Brendan gave an update on the post-survey information we will use to capture entries for the gift cards, and he will send it around.

                                                                                                      There have been no changes to the undergraduate survey since our reserves discussion last week, and the undergraduate subcommittee will update us at the next meeting.

                                                                                                      Action Items:

                                                                                                      Whole committee: need to look at the larger information literacy question issues with Ken before we can decide on which word to use

                                                                                                      SRS, ER, KS: Add question about subject librarians to grad student survey

                                                                                                      All subgroups: make survey questions closer/more parallel across the board (see slide 11 where DF has compared the same questions across the three surveys, highlighting the different language we use)

                                                                                                      Undergrad survey/undergrad subcommittee: we have lumped together questions on physical things and services, and these should probably be two separate categories (see slide 15)

                                                                                                      Discussion re David’s presentation on usage vs satisfaction vs importance:

                                                                                                      Presentation (with DF’s recommendations on what word to use with which question) is on the wiki

                                                                                                      Overall: we are considering adding satisfaction questions to the faculty survey, but will ask undergrads only usage questions.

                                                                                                      Whether to continue distinguishing print vs ebooks? Key points:

                                                                                                      ·        Really two separate questions: (a) how important are these sources (print vs ebook) to you and (b) how satisfied are you with the selection we have at BU?

                                                                                                      ·        Do we care about responses if it will not change our format selection? (i.e., we are an e-preferred library)

                                                                                                      ·        Are we focusing on satisfaction with the format of our materials or our collection (e.g., “you don’t have what I want” vs “you have what I want, but not in my preferred format)

                                                                                                      o   Decided are trying to get more at satisfaction with the collection and so will combine the formats, which is what we did when asking about our journals collection

                                                                                                      o   May also add the word “collection” to the question to really get people to focus in on it

                                                                                                      ·        This question will also have a comments box

                                                                                                      ·        Determined there are no questions about e vs print materials we want to add, as we already know what we need to on that topic

                                                                                                      Importance vs Usage re: Library Services

                                                                                                      ·        Satisfaction questions are good for giving us longitudinal data, but are not particularly useful for capturing information from just one survey

                                                                                                      ·        How to capture usage?

                                                                                                      o   If faculty member is on sabbatical and we ask about the past year—will they say they have not used anything, when ordinarily, they heavily rely on our resources?

                                                                                                      §  Alternatively, will faculty pay attention to “current year” or look at their normal course of business?

                                                                                                      §  But “current year” should definitely be kept for ugrad survey, as it helps us to look at what resources each class uses

                                                                                                      §  Grant cycle is 3 years—if faculty are not writing grants in this current academic year, will usage questions underreport on resource usage/importance?

                                                                                                      o   Use of importance AND satisfaction questions together could give us a better idea of where we need to improve (e.g., less heavily-used resource may still be important, especially with varying workloads over the course of a faculty member’s career. Should we not limit to the current year for faculty, as it may provide an inaccurate snapshot?)

                                                                                                      o   What can this data really tell us? How would we tell if something is low use because they are dissatisfied or if it is low use because it is not an important resource?

                                                                                                      ·        Faculty use in past year vs importance to faculty—

                                                                                                      o   How to capture resources that are important but may not have been used in the past year and/or are not necessarily used on an annual basis?

                                                                                                      o   Issue: probably don’t want to ask faculty all three questions (is it important, do you use it/how often do you use it, are you satisfied with it?)

                                                                                                      ·        Issues with deriving usage stats from satisfaction data—e.g., on the undergrad survey, they reported satisfaction higher than their reported usage, but can also be important—the discrepancy itself provides useful information

                                                                                                      ·        “use is use”—usage questions wouldn’t tell us the type or quality of use (i.e., getting help with a paper vs asking us where the bathroom is)

                                                                                                      ·        Should we ask all three questions, but cut down the number of services we’re asking about to five or so?

                                                                                                      ·        Services for which usage is declining may still be important (i.e., ILL)

                                                                                                      ·        Currently, the usage question is a yes/no, not a likert scale. What is the value of asking it this way?

                                                                                                      o   If we give up the usage question, we give up being able to report usage

                                                                                                      o   Likert scale is also misleading, because what might be frequent for one user wouldn’t be frequent for another

                                                                                                      ·        Tip people off at the beginning with a header like “for the following services, we will ask you to think about whether you use these services, how important they are to you, and how satisfied you are with them” – makes people more likely to answer questions, rather than getting bogged down by never ending questions and quitting the survey?

                                                                                                      ·        Can usage be pulled from other areas so we don’t have to ask the question (i.e., get ILL stats from ILL librarians)?

                                                                                                      o   This question helps capture the perception of services they think they use

                                                                                                      o   Capturing usage in the survey helps us correlate use and importance in a way that cannot be captured with other data sources on use

                                                                                                      ·        Degree of use—do we care? à kind of an open question

                                                                                                      ·        Importance might also measure unique, non-repetitive events, as opposed to regular use of an itemà potential for skewed/misleading results

                                                                                                      Overall: committee members agree on main points with the exception of the use vs importance questions

                                                                                                      Suggestions for resolving this issue:

                                                                                                      ·        Make our ideal survey, then edit

                                                                                                      o   Need to be careful we’re not making an entirely new survey and losing comparability to prior surveys

                                                                                                      ·        Create a matrix of Use, Satisfaction, and Importance with two categories: Facilities and Services

                                                                                                      ·        Is there a more elegant way to ask about what library you use (that would help eliminate the need for separate facilities questions, so we only need to ask about services)?

                                                                                                      Can we bring in some topical questions (i.e., data management) that we can change each time?                    




                                                                                                      Assessment Meeting Minutes 10/26/14
                                                                                                      Present: SDS, DF, KS, ER, LP, BD, SRS (minutes), RS, DB 


                                                                                                      Brendan went to Google Analytics workshop on GA for libraries in Washington DC, and reported back on some of what he learned:
                                                                                                      We should make analytics user friendly for our  librarians to access/use
                                                                                                      See if IT planning on upgrading to newest version
                                                                                                      Suggested consolidating “properties” (Primo and BU Libs website) to track use better (also Law, Theo and Med sites?)
                                                                                                      Implementing goals (eg what subpages are branches wanting to push), setting goals given the BU libraries organization and heterogeneous users/uses may be a challenge
                                                                                                      Using existing dashboard templates for new dashboards
                                                                                                      Additional webanalytics tools to augment those analytics available from GA (eg clicktale.com, usertesting.com, qualaroo.com, compete.com)
                                                                                                      LP - Post-survey we should form a subgroup to look at implementing these suggestions

                                                                                                      DAG is looking to do some usability work specific to the BULS searchbox
                                                                                                      Subgroup for Faculty Survey review will meet this Wed 10/29

                                                                                                      UG survey:
                                                                                                      Intro needs provost/IRB review but otherwise done
                                                                                                      Minor wording and ordering changes on Page 1
                                                                                                      Reserves Question
                                                                                                      Reviewed 41a and 41b, two approaches to this question
                                                                                                       a - percentage use (tho may not add to 100)  
                                                                                                       b - last use snapshot 

                                                                                                      Broadly, we preferred 41a. We established that the scale we want is use-frequency, though there are concerns about how students will interpret that.

                                                                                                      Existing subquestions 1 and 2 (print and online reserves) can be combined, but for “find library resources on your own”  separate into "from library" vs "purchase"
                                                                                                      1. Reserves 
                                                                                                      2. Blackboard 
                                                                                                      3. go find your own readings (from library)
                                                                                                      4. purchase

                                                                                                      Wording note: Must make textbooks inclusion/exclusion unambiguous

                                                                                                      May not want this exact Q on grad survey b/c of multiple roles grad students may have (as both teachers and students), and because law/med/theo's approaches to collection/reserves may differ from Mugar & branches' (eg Law buys textbooks)

                                                                                                      **PILOT this question early and revise as needed

                                                                                                      Individual libraries at BU section
                                                                                                      Law name change – CHECK ALL SURVEYS FOR THIS (RS/ER)
                                                                                                      Skip logic magic was demonstrated
                                                                                                      Changes: seating/desks, dropped security, added outlets

                                                                                                      Whirlwind update from Grad Survey review group


                                                                                                      Assessment Meeting Minutes 10/6/2014

                                                                                                      Present: Linda (head), Brendan (minutes), David, Ellen, Steve, Konstantin, Sarah
                                                                                                      Next Meetings:
                                                                                                      • Undergraduate Subcommittee: Friday, October 17th, 12:30-2:00 @ SEL 
                                                                                                      • Graduate Subcommittee: Thursday, October 16th, 2:00-3:30 @ Law Conference Room 
                                                                                                      • Assessment Committee: ???

                                                                                                      Action Items:
                                                                                                      1. Find things to improve about the graduate/faculty surveys.
                                                                                                      2. Review undergraduate survey post-edits [[ ignore bracketed sections ]]
                                                                                                      3. ???

                                                                                                      Review of Changes to Undergraduate Survey 
                                                                                                      • Linda agrees to work on the introductions for the surveys. Shooting for ”5 Ipad minis for undergrads, 4 for grads, and 3 for faculty.”
                                                                                                      • Put display logic on a single “Individual Libraries” page so it pops up in page and makes the survey feel more condensed. 
                                                                                                        • POST-MEETING UPDATE: This isn’t possible due to display logic glitching. We’d have to change our format to make it work.
                                                                                                      • We should double check the surveys for the naming conventions for the libraries. With Law being renovated, should we change its name in the survey? It seems like we’ve been using the colloquial names rather than the official names in some places.
                                                                                                      • Section headers were added and embiggened.
                                                                                                      • Included pop-up question for people who have used a bibmanager, asking which ones specifically.
                                                                                                      • Added Steve’s reserve question. Undergrad subcommittee will iron out the wording/options/likert format.
                                                                                                      • Do we want a question about mobile devices? 
                                                                                                      • Question: Will the Qualtrics mailer be blocked as spam? 
                                                                                                      Graduate Survey Collective Critique 
                                                                                                      • Check whether distance ed. Is capturable from the grad. info. The rest of the demographics can be removed.
                                                                                                      • “Are we asking open access questions for graduate students too?”
                                                                                                      • Remove primary discipline since we get that information.
                                                                                                      • Brendan and David agree to help the other subcommittees for consistency between surveys.
                                                                                                      • Consider separating BU Library Search from other databases (see Q39).
                                                                                                      • Maybe we should separate out service/facilities questions.
                                                                                                      • Remodel the favorite library questions after the Undergraduate Survey. Sarah: “Study Carrels” 
                                                                                                      Publicity Update – Sarah

                                                                                                      • Meeting with Mary McGowan’s Anita Greene Scholars. We can get both vertical and horizontal designs from them. We’ll try to get them on the bus too. Posters should come up first, and then emails will go out. And they’ll have them ready by Thanksgiving and the students will be available for rebranding through intersession. Design style is “like a comic book, but not like that dark stuff”. 

                                                                                                      General Notes and Miscellany 
                                                                                                      • Brendan and David agree to help the other subcommittees for consistency between surveys. 
                                                                                                      • We should consider looking for a new 9th committee member to replace Lisa.
                                                                                                      • Konstantin: “Should we start gathering reference use data for grads/faculty in the surveys?” 

                                                                                                      Assessment Meeting Minutes 9/29/14

                                                                                                      Present: Brendan, Dan, Steve, Lisa, Ellen, Russel, Linda (chair), David, Sarah, Konstantin (minutes), Ken Liss
                                                                                                      Next meeting: Monday 10/6, 2pm-3:30pm, Mugar Administrative Conference Room

                                                                                                      Action Items:

                                                                                                               1. Minutes, announcements, changes to the agenda?

                                                                                                                2.   Publicity update (Sarah)

                                                                                                               3.   Full committee high-level review of the Graduate Student Library Survey

                                                                                                               4.    Discussion about Undergraduate Student Library Survey (Undergrad sub-group)

                                                                                                               5.    Next steps?

                                                                                                      • Timeline
                                                                                                        • We should submit to the IRB in mid-January
                                                                                                        • Before we submit we should receive approval from the Provost office for ALL files (i.e. publicity, conflict of interest, surveys, etc.)
                                                                                                      • Weekly meetings
                                                                                                        • We will be meeting weekly, not biweekly
                                                                                                        • The following groups will be responsible for reports
                                                                                                          • Faculty survey: Steve, Russ, David
                                                                                                          • Grad survey: Sarah, Konstantin, Ellen
                                                                                                          • Undergrad survey: Dan, Brendan, Linda
                                                                                                      • Report from the Undergrad survey group
                                                                                                        • Brendan shared the file "Suggested changes for the Undegrad Survey"
                                                                                                        • All agreed that we should review Intro, exclude undergrads >18 years old by email distribution, reorder sections (Resources, Services, Individual libraries)
                                                                                                      • All agreed that we should follow up with focus groups

                                                                                                      Assessment Meeting Minutes 09/15/2014

                                                                                                      Present: Brendan, Dan, Steve, Lisa, Ellen, Russell, Linda (chair), David (minutes), Sarah
                                                                                                      Next Meeting: Monday, 9/29, 2pm-3:30pm, Mugar Administrative Conference Room 
                                                                                                      Following Meetings: For rest of year, will be meeting every other Monday at 2pm, except for one Friday: 9/29, 10/17 (Friday), 10/27, 11/10, 11/24, 12/8, 12/22

                                                                                                      Action Items:
                                                                                                      Sarah: Remind reference librarians about the ongoing MINES survey
                                                                                                      Linda: Ask for input from Ken Liss on questions about instruction and services (Q8 & Q9 in undergrad survey).
                                                                                                      Steve: Propose question to capture undergraduate use of "reserve" materials
                                                                                                      David: Talk to JD about question on bibliographic managers, and which ones to list in question.
                                                                                                      Brendan, Dan, and Lisa: Work on undergrad survey for next meeting, including, comparing to MIT and UW surveys, looking at qualitative data from last survey, and proposing any needed wording changes.
                                                                                                      Sarah, Ellen & ???: Similar work (for future meeting) for grad survey.


                                                                                                      Dan announced improvements to the websites Google Analytics, which now track searches from across site, including branches.

                                                                                                      Russell introduced our newest member, Ellen Richardson, Assistant Librarian for Administration at the law library.

                                                                                                      MINES Update

                                                                                                      Brendan showed the current state of the survey. We have had almost 3,500 results so far, with an extremely high completion rate. Brendan has downloaded the data (so far) into Excel to do some preliminary analysis, including parsing the URL to let us see what resource is being visited. Extremely impressive.

                                                                                                      Brendan will share data with anyone on committee who is interested. We can look at it and use it, but remember it is preliminary and hasn't yet undergone complete analysis.

                                                                                                      Next Survey

                                                                                                      Linda reported that until we hear from Provost (nothing yet), we still don’t know for certain whether we will do everyone in next survey, or only faculty. We will finish work on updating undergrad survey; then do the faculty survey update, in case that is the only survey we administer.

                                                                                                      Decided that for each survey we will do pass as a group to make sure we have right questions, without wordsmithing. Then break into groups to look at wording and compare to other surveys.

                                                                                                      Undergraduate Survey

                                                                                                      Sarah suggested we may want to speak to Ken Liss to find out if the questions on instruction and services (Q8 & Q9) are really the metrics he wants to track. Linda will do it.

                                                                                                      For the questions about facilities (Q13), we need to rethink our categories. "Furniture" can be confusing. Seating vs. Chairs. Outlets. Lighting.

                                                                                                      On Q18 regarding course reserves, Steve brought up wanting to know about where students are getting resources we traditionally think of as e-reserves (e.g., the library's e-reserves system, Blackboard, faculty web site, etc.). There was a lively discussion on whether undergrads (as opposed to faculty) are the right ones to ask, and whether a question can be written that captures what we want to know from undergrads. Steve will work on possible wording for question.

                                                                                                      Sarah remembered there being useful feedback on survey in qualitative data. When we break into groups, groups should look at qualitative data for their survey for comments about the survey.

                                                                                                      Q21: The question was changed from Yes/No to a scale; need to change wording of question to reflect change.

                                                                                                      Decided that if undergrads answer "Yes" to Q22 about using bibliographic manager, there will be a further question asking which one(s) they use. David will talk to JD about what should be on list.

                                                                                                      Because the wording for the comments boxes refers to the page titles, we need to add header to pages, or change wording of comment question.

                                                                                                      Some time was spent on the perennial use vs. importance vs. satisfaction problem, with no solution.

                                                                                                      Brendan, Dan, and Lisa volunteered to work on detailed wording for undergrad survey by next meeting. This will include comparing to MIT and UW surveys, and looking at qualitative data from last survey

                                                                                                      Sarah & Ellen volunteered to look at wording for Grad survey (for future meeting).

                                                                                                      Brief discussion of a replacement for the irreplaceable Lisa, who is leaving us. We are a large group, so we don't have to replace her. Linda suggests we consider needed skills in making decision.


                                                                                                      Assessment Meeting Minutes for 07/31/14

                                                                                                      Present: David (temp chair), Dan (minutes), Steve, Brendan, Lisa
                                                                                                      Next meeting: TBD - Doodle; Administrative Conference Room Mugar

                                                                                                      Action Items:.
                                                                                                      David will send a Doodle out to help schedule our next meeting due to August vacations.
                                                                                                      Brendan will continue editing the undergraduate survey draft on Qualtrics, according to decisions made today.

                                                                                                      Dan will share the archival surveys on Qualtrics, but set permissions so that no one can edit them.

                                                                                                      All can look at the undergrad draft to identify issues, better wording, missing questions/options/themes etc.

                                                                                                      - no announcements

                                                                                                      Discussion of 2015 Library User Survey Question Development
                                                                                                      Lisa lead discussion of the last sheet in this Google Doc, called "Comparing Under Q BU, UW, MIT". 
                                                                                                       - not much different from our to UW
                                                                                                      's survey: their survey is nice and very short
                                                                                                       - Answer rows on MIT survey Q2 are nicely specific.
                                                                                                       - We also looked at MIT Q11's layout (aware | importance).  We considered how this might apply for us, considering we often ask about use/importance/satisfaction.  We looked at our questions 9, 25, and 26 in comparison.
                                                                                                       - We have something very similar to MIT's Q12.
                                                                                                       - MIT Q16 + Q18 include library instruction outside of a class; probably not that useful for BU (could mean HGARC events etc?).
                                                                                                       - MIT did not ask overall importance of libraries
                                                                                                       - MIT Q24, the $100 to spend question.  Poorly worded and makes respondent add - not good for fatigue.
                                                                                                       - Lisa also noted that there may be some Qs from MIT not represented at all on the spreadsheet. She did not include them because there were no comparable Qs on our survey.

                                                                                                      Discussion of editable
                                                                                                      surveys on Quatrics
                                                                                                      David began this with the question of when, exactly, might it be favorable to combine the data coming from undergrads, grads, and fac.  There seem to be very few questions where we would want to do this, and without weighting the numbers for response rate, the numbers would be skewed.  We decided to begin composing Qs on three separate surveys.
                                                                                                      As we will begin editing the Qualtrics surveys, Dan will share the archival surveys on Qualtrics, but set permissions so that no one can edit them.
                                                                                                      We began looking at the faculty survey questions on Qualtrics.  We considered the frequency of physical and virtual visits, which were combined in 2010 but later separated, and we added a "during the current academic year" phrase as well.  We looked at the undergrad survey in comparison, and realized we should edit that first as it was the latest, and would need the least editing.
                                                                                                      Do we need to ask about majors again?  Used those to send comments to specific librarians, etc.
                                                                                                      Brendan noted the prevalence of the N/A option on many questions, and we noted the consultants advised having this "way out" for respondents.
                                                                                                      Our question 13 - piped text from previous question to set up to work in Qualtrics.  Steve is satisfied with the options on Q19, to look at changes over time.
                                                                                                      MIT asks about USE of more specific sources or collections than we do. We considered setting up our Q differently, but we do have many other statistical sources on collection usage, so we decided NOT to add any about specific USE other than the options we already have listed in the survey.
                                                                                                      Brendan suggested that we could create a numbering system that would indicate which questions are longitudinally useful.
                                                                                                      We completed an initial run-through of the questions on the undergrad survey, with some edits left for Brendan to finish up later, as he was "driving" the Pardee room computer and editing on the fly.

                                                                                                      Scheduling next meeting
                                                                                                      We looked at scheduling, but could not get around vacation schedules.  David will send a Doodle out to help schedule it.

                                                                                                      Assessment Meeting Minutes for 7/14/2014

                                                                                                      Present: Linda (chair), Brendan, Konstantin (minutes). Lisa, Russell, Steve, David

                                                                                                      Next meeting: Thursday, 7/31/14 at 12pm (at Pardee)

                                                                                                      Action Items:

                                                                                                      - Ask Bob Hudson to talk to Provost (Linda)

                                                                                                      - Look at the questions (David)

                                                                                                      - Discuss Brendan's action plan (all)

                                                                                                      - Take a look at Konstantin's analysis (all)

                                                                                                      - Update numbers on the spreadsheet (Dan)

                                                                                                      - Create the Google doc (David)

                                                                                                      - Wording, style, etc. - comparison of MIT and U of W - tabs: faculty (Konstantin), grad (Helen/Russ), undergrad (Lisa)


                                                                                                      Russ suggested adding Helen Richardson for membership on the Assessment Committee. (all agreed)

                                                                                                      Dan talked about Google Analytics - workshops - comparison functionality, bounce rate, exit rate.

                                                                                                      Dan reported on survey transfer. To access surveys on Qualtrics: go to bu.edu/qualtrics

                                                                                                      Discussion followed in regards to our survey plan

                                                                                                      Brendan's report: meeting with Pam Andrews (Lauren Hess replaced Pam in 7/14). Creation of panels. We discussed the advantages and disadvantages of working with the Provost office.

                                                                                                      Konstantin raised a question if BMC affiliates will be included in the panel.

                                                                                                      Assessment Meeting Minutes for 6/2/2014

                                                                                                      Present: Linda (chair), Brendan (helped with minutes), Konstantin, Lisa, Dan, Russell, Steve, Sarah, David (minutes)

                                                                                                      Next meeting: Thursday, 6/19/2014 at 2pm

                                                                                                      Action Items:

                                                                                                      • Dan: Transfer surveys
                                                                                                      • Lisa: Talk to Ken about Ithaka Q15.2 and 16.1 and Q4(?)
                                                                                                      • Konstatin: Looking at categories for each question. Talk to colleagues on medical campus about survey needs, and data management
                                                                                                      • Brendan: Working on Google Analytics
                                                                                                      • David: Help with display logic, ask JD and Jenna for input on data management question
                                                                                                      • Sarah: Ithaka Q 22.2: Sarah will compare to our faculty Q35 to see if there is anything we want from Ithaka. Look at reports to assess importance of disciplines
                                                                                                      • Russ: Examples of where WA survey wording better than ours

                                                                                                      Brendan: MIT Survey:

                                                                                                      • Some questions are much more specifically oriented. We’ve been trying to stay general with our questions but perhaps now we should start considering using more targeted questions.
                                                                                                      • Linda: “We should start keeping track of the “follow-up questions” like that one.
                                                                                                      • MIT Q13 (where to start looking for a specific ebook) could be useful as a simple statistic/longitudinal stat. Ithaka has a similar question.
                                                                                                      • Q24 (“How would you allocate $100 to improve library system”) Brendan likes a lot. “It’s fun!” Pretty easy to analyze too. Problems: many people can’t add up to 100, and the options may not be open ended enough.
                                                                                                      • Q25’s second question (“Are there other library services you think MIT should be offering”) is good in Linda’s eyes. It focuses on what the users want, not what we think they want.

                                                                                                      Dan: Transferring Surveys

                                                                                                      • Skip Logic causing problems.
                                                                                                      • Display Logic ALSO causing problems.
                                                                                                      • Dan will continue to tinker with help of David/Brendan.

                                                                                                      Russ: Comparing UW grad & undergrad surveys

                                                                                                      • “Is it typical for us to combine graduate and professional students?”
                                                                                                      • Sarah brings up concerns about using qualtrics for this survey if we’re having so much difficulty with skip/display logic. (“Well we have SurveyMonkey for another year if we need it”).
                                                                                                      • We really need to focus on the wording.
                                                                                                      • Possible wording: “Have you used the library’s services for an internship or sponsorship?” , or something to that effect.
                                                                                                        • “We followed UW’s model.”
                                                                                                        • We should include Marlene if we’re going to continue to discuss wording for Law users.

                                                                                                      Lisa: Ithaka Survey

                                                                                                      • Ithaka Q 7.3: possibly useful for follow-up to survey
                                                                                                      • Q 13.4: Possible springboard for question regarding publishing Steve: compare faculty attitudes towards publishers vs. libraries for publishing advice, etc. Russ: should library be responsible for some of these issues?
                                                                                                      • Q 15.2: Question about whose responsibility it is to teach information literacy
                                                                                                      • Q 16.1: Bring Ken into discussion about whether we want something like 15.2 and 16.1
                                                                                                      • Q 22.2: Sarah will compare to our faculty Q35 to see if there is anything we want from Ithaka.
                                                                                                      • Q 6.5 and 8.3: David will ask Jenna and JD to look at data management questions and give advice

                                                                                                      Assessment Meeting Minutes for 05/19/14

                                                                                                      Present: Linda (chair), Brendan (minutes), Russell, Dan, Sarah, Konstantin, Steve, David, 
                                                                                                      Next meeting: Monday, June 2nd, 1:00-2:30; Administrative Conference Room Mugar

                                                                                                      Action Items:
                                                                                                       - Russ: Compare UW grad & undergraduate surveys.
                                                                                                       - David: Compare undergrad & grad BU reports.
                                                                                                       - Dan: Transfer all other BU surveys
                                                                                                       - Konstantin: Compare the other BU surveys to each other.
                                                                                                       - Brendan: Compare the other BU surveys to each other & troubleshoot Qualtrics issues.
                                                                                                       - Sarah: Determine any problems with ceasing to collect discipline information & survey keeper

                                                                                                       - Brendan received a promotion to Assessment Data Specialist (or something like that).  Now we can offload all our projects onto him!

                                                                                                      Plans for New Survey:
                                                                                                       Linda brings up two important questions:
                                                                                                      1. How will we go about keeping track of our ideas/things to update or edit?
                                                                                                        • If we use the minutes, we'll need to be really organized.
                                                                                                        • Survey Keepers (see Wiki's side bar) are used to keep notes about conventions we've agreed upon, past-encountered problems, and survey decisions to avoid in the future.
                                                                                                        • General consensus to use minutes.
                                                                                                      2. Should we do one per year, or a three-in-one survey every three years?
                                                                                                        • Steve: Our response rates may change if we don't actively target a single user group.
                                                                                                        • David: If we put out a single report at the end for all three, it won't be targeted enough.  We should put out individual reports for separate user groups and perhaps an executivesummary of all three.
                                                                                                        • Sarah: We can filter the survey by user group.
                                                                                                        • Longitudinal statistics casually discussed.  If we want to start recording longitudinal stats, we need to make the changes with this report.  They have been used by other libraries and yielded interesting results.
                                                                                                        • If we do one survey per year, we're putting out one report per year as well.  Worries about survey and analysis burnout float around.
                                                                                                        • David: We could also have the home page link separate surveys for other user groups (everyone likes this idea a lot)
                                                                                                        • We could try using display logic in place of skip logic to bypass the skip issues.
                                                                                                        • General agreement on 3 surveys in one year, each tailored and targeted toward the specific user group, linked from a main survey page, and individual reports instead of a comprehensive report.
                                                                                                      Dan's update - Uploading Old Survey to Qualtrics:
                                                                                                      • Differences between the two:
                                                                                                        • The introduction transferred over into Q1.  Not noticeable from user's screen.
                                                                                                        •  Problem where the respondent cannot use the back button over any logically skipped survey content.
                                                                                                      David's update - Reviewing Survey Reports:
                                                                                                      • The reports referenced every question, but there wasn't a lot of deep analysis.
                                                                                                      • "Part-time" faculty wasn't a particularly useful demographic.  We should exclude it next time.
                                                                                                      • Question: are adjuncts considered part-time?
                                                                                                      • Q3 wasn't particularly useful either (whether faculty taught or used federal aid)
                                                                                                      • Perhaps we can simplify by not collecting department information.  Sometimes departments are fuzzy with self reporting anyway.
                                                                                                      • Brendan: "Can we extract info from email?  MIT collected no demographics and used email addresses to create demographics."  IRB would have a conniption fit if we started collecting personal data like that.  The anonymity would not exist anymore.  Not worth the time.
                                                                                                      • We can use conditional questioning to work around the parts people find confusing (e.g. with Med & BMC/school related fields)
                                                                                                      • David: "I want to ask about data management in the open access section"
                                                                                                      • Ranking importance is fine, but what we really could use is frequency data. 
                                                                                                      • We should add "in the last year" and drop "academic year" in place of "last year" so the data is current and not related to past experiences.
                                                                                                      Russ - Analyzing UW Survey:
                                                                                                      • "UW's survey is about the survey taker.  Ours is about what we think is important to them."
                                                                                                      • We need to package questions from their vantage point.
                                                                                                      • UW's survey is clean and minuses.  Ours can be wordy at times.  It has its plusses & minuses.
                                                                                                      • They included phrasings like "Thinking back to your last..." kind of questions.  May be more useful for infrequent services.
                                                                                                      Out of time. Next meeting decided for June 2nd.

                                                                                                      Assessment Meeting Minutes for 04/03/14

                                                                                                      Present: Linda (chair), Dan (minutes), Russell, Brendan, Lisa, David, Konstantin
                                                                                                      Next meeting: Thursday, May 1st 1-2:30; Administrative Conference Room Mugar

                                                                                                      Action Items:.
                                                                                                      Linda will renew Survey Monkey.
                                                                                                      Brendan will create a google doc in case groups wish to post comparison info before meetings.

                                                                                                      Dan will transfer the prior BU faculty survey from Survey Monkey to Qualtrics (2 copies).

                                                                                                      All can begin looking at other survey instruments (UW, MIT, Ithaka) for helpful comparisons.

                                                                                                      - Dan announced that Feb 27th marks the 1st full day of external link tracking through Google Analytics (GA).  One can view stats on internal and external links for any page on our site, though those numbers exist in two separate places within GA.
                                                                                                      - Linda announced that she and Tom C., Steve, Stephanie, and possibly David and Brendan will be attending a session at the Hyatt Regency on Academic Analytics.  The session will specifically be aimed at librarians.

                                                                                                      OCLC Assessment event update
                                                                                                      Linda will serve on a panel at an OCLC meeting at Brandeis on April 22nd called "Getting the Right Fit: Tailoring Assessment Strategies for your Library."  More info is available here.

                                                                                                      MINES is proceeding very smoothly now.  We have had a 0% drop out rate.  We need to think about how to back up the data periodically.  David believes we can download the data at any time, and we agreed that perhaps a schedule should be followed to download and backup the data periodically, although IS&T seems to trust the cloud.

                                                                                                      Faculty Survey planning
                                                                                                      We began planning for a faculty survey, not in exact order of the agenda, but the info is here below.
                                                                                                      Game plan- general, but not unanimous, consensus to perform surveys individually instead of grouped. We also decided to renew Survey Monkey (Linda will do this) once more as the deadline is fast approaching and it may come in handy
                                                                                                      Developing the survey - will follow these steps
                                                                                                      comparing to other surveys - Small groups will do initial comparison work and then bring suggestions to an AC meeting for all to discuss.  Brendan will create a google doc in case groups wish to post something before meetings.
                                                                                                      a.  Dan will transfer the prior BU faculty survey from Survey Monkey to Qualtrics and look for any differences (particularly in skip logic).  Dan will create 2 copies on Qualtrics - one to edit and one for posterity.
                                                                                                      b. Linda and Konstantin will then look over comparisons from our own grad and undergrad surveys
                                                                                                      c. Next, Lisa and Brendan will look at comparisons to the MIT survey
                                                                                                      d. Lastly, Russ and possibly others will compare to UW and Ithaka instruments
                                                                                                      2. final pass thoughts - examine possibilities of skip logic, questions or other items to eliminate to keep the survey short, and look through our survey keepers list
                                                                                                      iii.     Timeline - We're thinking about Spring 2015.  Russ expressed that we should be mindful of the timing, as Law faculty will be relocated during Spring 2015. Survey fatigue
                                                                                                      IRB - Linda will accomplish
                                                                                                      v.     Publicity/Recruitment - poster from Anita Greene group and emails
                                                                                                      Launch - nothing noted
                                                                                                      Analysis, quantitative - nothing noted
                                                                                                      Analysis, qualitative - have no one currently for this. there is the possibility of going back to tag previous surveys.
                                                                                                      Report writing - will likely rely on in-house talent.
                                                                                                      Staff involvement in changes - we need to examine this more.


                                                                                                      Assessment Meeting Minutes for 01/29/14


                                                                                                      Present: Linda (chair), Brendan (minutes), Russell, Lisa, David, Steve, Konstantin, Sarah, Dan

                                                                                                      Next meeting: Friday, Feb. 14th (Valentine's Day) 2-3:30; Administrative Conference room Mugar

                                                                                                      Action Items:
                                                                                                      - Notify Tim that Pubmed (and other resources?) should be placed behind EZproxy
                                                                                                      - LINDA: Call Amy about list being proxied. (DONE 2/3/14- LP)


                                                                                                      - Russell will be joining us henceforth as a new member of the Assessment Committee. Welcome!

                                                                                                      Tuesday, 1/21 Phone Call w/ UMass Amherst’s Rachel Lewellen

                                                                                                      Present were David, Steve, and Sarah and Linda.  Overall, a very positive, informative call.  UMA started out with a year long implementation and collected data from all users who started Ezproxy sessions during two, 2 hr. periods per month. In UMA's second year-long implementation of MINES, they switched to n = 140, and then reduced finally to n = 120. It’s important not to change n too far along so that data won’t have to be excluded, although Terry Plum implied that collection data can be adjusted if a change to N is needed during an implementation. Over 70% of surveyed users completed the survey at UMass (!!!), no complaints registered, not even about frequency. UMA did not have a comment box, but did have an email link. They only received one email. UMA included a question about why survey taker is using the resource.  Shared graphs are promising; of note the URL collection stats of those surveyed generally reflects the URL collection stats of the entire user-base (ie statistically relevant data).  Very promising overall!

                                                                                                      Let’s apply this to our MINES! 

                                                                                                      - We can use the URL data that will be gathered, but it will involve more involved analysis than other sources of data.

                                                                                                      - Konstantin: “We should use ‘BMC Resident’ for primary status question”.  Everyone agrees with the tactful word choice.

                                                                                                      - Google Chrome is being problematic for everyone with scrolling issues (only for most recent update).  It may interact with survey if someone tries to get to bottom of primary status list. “Maybe we can change the number of visible items on the dropdown list” suggests David.  Brilliance!  Later, “Alas, we cannot do that” informs David.  Foiled by technology again.  We shall investigate during test period whether Chrome will be problematic

                                                                                                      - “High school student shouldn’t be at top of list. It’s not ordered by age (Brendan grumbles), so we should order it by the groups most likely to use it, ie put High school students below staff on the drop down list.  General consensus, Brendan dissents vehemently.   Too bad for him.

                                                                                                      - David’s suggestion to get the sponsored question to drop down works.  An excellent solution to the problem we were having.

                                                                                                      - “ask@bu.edu” to be changed to “assess-l@bu.edu” so assessment committee members can personally field survey questions.

                                                                                                      - On criteria for assigning survey-testing: “Perfect for those who have miffed you!”
                                                                                                      - Linda: “We can’t go live until we’re sure the Med campus and Pubmed work.  We need to get Pubmed behind EZproxy.  And guide writers will need to update PubMed links.  Also many Law resources won’t be EZproxied.

                                                                                                      - Thoughts on n:

                                                                                                         * Larger student base and greater usage means n can be larger (or smaller? No, larger… we think).

                                                                                                         * We can change n within the first month if our number doesn’t suffice.

                                                                                                         * Linda: “Let’s go around and see what everyone thinks is reasonable.” Crickets

                                                                                                         * After some deliberation, Brendan: “100 or whatever is reasonable”.

                                                                                                         * Number-crunching Steve: “100 gets us just enough data with 70% expectation rate.”


                                                                                                      MINES Testing

                                                                                                      Directions will be on the site.  Mugar to have two demo sessions on Friday and Monday, one in the morning, and one in the evening.  Everyone else to schedule demos for the other branches. Remember, we want to test on as many platforms as possible, even mobile ones.  Also we need to test for blank survey results.  Proposal: Go live on 11th @ 12:00? … eh, maybe not.  We’ll see.



                                                                                                      -       Steve, on n: “Yes means no, and no means yes”

                                                                                                      -       Russ, on difference between Law Library and other libraries: “We actually provide service” (excellent smack talk, if I may say so)


                                                                                                      NEXT MEETING?

                                                                                                      -       Linda: “How does Valentines day work for everyone?”  All concur. There will probably be Valentine’s goodies, and Brendan will probably bring something (he didn’t mention this though). 

                                                                                                      Meeting adjourned.

                                                                                                      Assessment Meeting Minutes for 01/16/14

                                                                                                      Present: Linda Plunket (chair), Dan Benedetti (minutes), Russell Sweet, Brendan DeRoo, Lisa Philpotts, David Fristrom, Tim Lewontin
                                                                                                      Next meeting: Wednesday, Jan. 29th 2-3:30; Administrative Conference room Mugar

                                                                                                      Action Items:.
                                                                                                      Linda will work out what staff should test which databases (refnews/alphabetical?)
                                                                                                      Linda will change wording on MINES survey as outlined in 'B' below
                                                                                                      Sarah will be writing up the instructions for staff testing
                                                                                                      Konstantin will answer this question: If asked to give their primary status, would faculty on the Med campus be confused between a 'faculty' option and an 'affiliate' option?
                                                                                                      David will look at Qualtrics closely to see if we can easily get the URLs patrons will be forwarded to on completion of the MINES survey.
                                                                                                      All who can should attend a call with Rachel Lewellen, Assessment Librarian from Umass Amherst, Tuesday (01.21, 10am)

                                                                                                      - Linda reported the library wide meeting was a good success.
                                                                                                      - We've been invited to an undergrad faculty council, and a presentation will be given by Tom C. and David.
                                                                                                      - Brendan was asked by Dan to help with the Google Analytics initiative.
                                                                                                      - Today's meeting will be about MINES, so read this data visualization article in preparation for the next meeting:
                                                                                                      Azzam, T., Evergreen, S., Germuth, A.A., & Kistler, S.J. (2013). Data visualization and evaluation. In T. Azzam & S. Evergreen (Eds.), Data Visualization, part 1. New Directions for Evaluation, 139, 7-32.

                                                                                                      MINES Initiative updates
                                                                                                      Nearly ready to implement, but need to set up staff testing on a test website.  We will be looking for any minor issues, probably with 'refnews' staff, with Linda working out which staff should test what databases.  Sarah will be writing up the instructions for staff testing, and all should review them.  Russ also notes we should start the surveying early enough before finals, or alternatively wait until early June.
                                                                                                      A) Some institutions are putting open access databases behind ezproxy in order to track through MINES.  David reminded us that many patrons go straight thru google to resources, so we won't be able to get them.  Tim pointed out that we don't send pubmed thru ezproxy because of the way their website used to work, but we can now.  Because of possible complications with redirects, we should carefully test the performance of very important databases like pubmed. 
                                                                                                      B) We worked on the wording of the MINES questions.  Terry Plum had two suggestions on wording: 1-that we don't need an 'other' on q2.CORRECTION BY LINDA: TERRY'S COMMENT WAS NOT THAT WE DIDN'T NEED IT BUT ASKED IF RESPONDENTS WOULD KNOW THAT OTHER REFERRED TO OTHER-AT-BU.  2- on the 'purpose' question, to include 'research unfunded (faculty)', because it is primarily faculty that do research and she wanted to distinguish this from scholarship.   We decided to change this to "unfunded research and scholarship" and not include '(faculty)' or '(students)' in answer options.  We also decided to add an answer option for 'library staff'.  Brendan caught an incorrect plural of "affiliates'.  Finally, a question for Konstantin came up: If asked to give their primary status, would faculty on the Med campus be confused between a 'faculty' option and an 'affiliate' option?
                                                                                                      C) Two places are trying to get answers on what resources patrons are using, thru MINES: Umass Amherst and Univ of Toronto.  There will be a call with Rachel Lewellen, Assessment Librarian from Umass Tuesday (01.21, 10am) that all who can should attend.  Umass is also increasing the number of times the survey will be filled out, by having the survey pop up more often.  We decided that we should also track resources, and David will look at Qualtrics closely to see if we can get this easily, as it seems should be possible.