Minutes

  • Please enter minutes on this page, with the newest at the top. Put a line between entries. Highlight action items in RED with a name/initials indicating who's responsible for the task and a due date, if there is one. For minutes prior to 2014, see the Archive of work through 2013.


Assessment Committee Minutes 4/17/15
Attendees—BD, TH, SS, JR, JA, RS, EF, DF, KS (via skype), MW, LP, TL


Next meeting: Monday, May 4 at 2PM. Focus: substantive conversation on what type of data we want to use.

Agenda:

  1. Brendan to report re EZProxy Logs meeting with Scott Macomber
  2. Linda re Eddie from Sourcing on use of Tableau
  3. Discussion re use of priorities for next yea

 

Brendan re Scott Macomber & EZ-Proxy Logs

  • EZProxy records two types of files— .log (records everything) and .spu (records first place they land after login)
  • Mines captures URL data approximately every 100th person.
  • Not totally clear what EZproxy captures, Scott is working with OCLC to sort this out. 
  • Brendan has 2 files—log for Jan 1 has 86,000 logs. BD discussed cleanup process for .log files—there is a lot to be done, so we would need to limit amount we asked for/looked at
  • .log files show username, so needs to be removed. .log captures all webpages opened/any motion by user id (i.e., when they click the back button)
  • .spu captures first session through EZProxy i.e., if you went to multiple BU resources in one session, it only captures the first one
  • Brendan will compare host files to MINES and see if there is anything we can pull. Also going to tie to Google Analytics data to assess search path based on timestamp to educate us on how to improve our search capabilities. 

Should we really do this? Is it worth it? What questions are we trying to answer?

  • BD recommends one year, max b/c of time involved.
  • JA recommends running a call with Python to pull user group before stripping username, then TH recommends scripting to automate
  • BD does not recommend options listed on OCLC website due to age and ease of use, so we are likely looking at scripting something by hand
  • Side note: JA is not sure how long we will have EZProxy, b/c depends on vendor requirements
  • Can’t get user group info from elsewhere, but what do we get out of that? 
  • SS is not sure how he would use resource info for collection development.
  • Ustats does already give us stats on what is downloaded. 
  • How are people finding our resources? Google Scholar? Primo? 
  • User behavior is interesting, but could this be gathered through snapshots of log, rather than analysis of all the data 
  • RS: we don’t have COUNTER-compliant data, so this would help us some representation of what is being used for renewal decisions; would also help with ARL questions re usage stats. LP—we need COUNTER compliant data. RS—OCLC Parse holds itself out as COUNTER-compliant. JR—reports themselves are not COUNTER-compliant, they just use COUNTER recommendations/format

BD, DB, and LP met with Eddie from Sourcing re Tableau

  • He is willing to be a reference person for our use of Tableau
  • We don’t know the other products for data visualization, so not recommended we make any decisions just yet
  • Looking at this for presenting information in meetings with staff, university administration, etc.
  • It is very expensive, although it can be setup to grab data from multiple sources, but we may not neneed this functionality 
  • Meeting at 2PM Tuesday on Tableau, can be viewed in Mugar Admin Conference Room if interested 
  • BD to ask Rachel Lewellen of UMass Amherst if she will share some of her MINES/Tableau data 
  • We will hold off on purchasing Tableau for now because it has a steep learning curve, and we don't currently have an exact need for its functionality 

Key focus for next year: data and getting it out to users

 



Assessment Meeting Minutes 
4/6/15
Present: 
Ellen, Dan, Konstantin, Brendan, David, Sarah (minutes), Steve, Linda (chair)

Next meeting 4/17 (Friday) 2-3:30pm. Linda will send out room info.

Announcement:
Eddie from Sourcing is using Tableau, coming to meet with some of group 4/7/15 2pm small conf rm

Reviewed what was covered in last mtg

New 
ARL SPEC kits – Konstantin
~4/year on hot topics are accepted. Written by ARL member libraries generally. All ARL libraries are asked to fill in survey, then data compiled and report written by authoring institution. Access to 2003-current will require a subscription. Earlier years are FT in Hathi trust
2015 topics include one on scholarly output assessment activities
Steve notes that in the past the examples/best practices included in SPEC reports were one of the more useful parts. So just the TOCs can be useful to know who might have examples (that may be available for free from their sites).
May be worthwhile to see what assessment ones have been done.

David – Analytics
Look at patron searches in BULS, can this inform reference and instruction?
Mike Ward says google analytics of page are currently better than Primo/ex Libris built in ones. No examples or models exist that DF found to assess the queries (how to do it, or if it’s useful). DAGS may be using some small parts of this for improving searches/results/displays
DF doesn’t think it’s a high priority now, might be interesting to look again in a year when ex Libris analytics may be better.

Brendan – MINES
“We downloaded all the data from MINES, and there’s a lot of it” - over 7K responses. Completion rate around 91% last time it was calculated. BD cleaned up some bad data (probably statistically insignificant anyway). Added some databases hit based on connection URL for Ebsco. Discussed having BD create a couple other graphs. DF, BD, and LP will work on how to create reports from this data. DF- audiences: brief report for outside/above library, more detailed for internal use or sharing with other libraries. Potentially present to grad research council. Adding in data from outside survey? (# of potential users per school, etc). need context for reader in reports. What would we do differently if we ran MINES again? Longitudinal snapshot?
Should BD work to draw out more detailed specific databases from aggregators? – consensus was no b/c of potential quality of data issues.

Dan –Tableau Trial
Uploaded some spreadsheets (ebrary stats Steve had sent out) and then explored some graphical display options. Lots of online support. Dan did look at a few example videos, but mostly was just exploring. Showed us 3 visualizations he’d worked on. Definite interest in learning more about it.


Assessment Meeting Minutes 03/09/2015      
Present: Linda (head), Brendan (minutes), Dan, David, Ellen, Sarah, Konstantin, Russ, Steve

Next Meeting: Monday 03/23/15, Mugar Conference Rm.

Action Items:

  • Linda: Ensure student permissions for archived publicity files.
  • Sarah: Send out fact sheet files and publicity contact list.
  • Brendan: Get EZProxy logs from Scott Macomber and fiddle with the data.
  • Everyone: Think more about the future of the committee.

Agenda
  • Nothing new from Provost -> 2015 Library Survey will be postponed until 2016.  The surveys and how to implement them are being saved to a read-only dropbox archive for future reference.
  • Brendan showed the structure and contents of the archive.  Showed the contents of the survey manual.

Individual Research Topics
  • Tableau and Data Warehouses [Dan]
    • Tableau is more visualization-driven than other data analyzers.
    • Rachel Lewellen of UMass Amherst does a lot with Tableau.  Uses it for presentations.
    • It's potentially another tool people will need to learn to use and manage among all the other tools people should learn to use, but don't.  A data warehouse may be wasted effort, and moreover money.
    • People don't want unprocessed data.  Give them facts and graphs that are useful by themselves.
    • Individualized dashboards would be the most effective, but they're labor intensive.
    • The Anita Greene Scholars are great graphic designers, but we need to schedule time with them since they're in high demand.
    • BU Libraries don't have a single go-to person for statistics and data (like Rachel Lewellen).  We should reconsider this.
  • Fact Sheets [Sarah]
    • Some fact sheets tell a story instead of numbers.  This is a good model for public fact sheets
    • We can compare library rankings and university rankings to highlight library contributions to BU's success.
    • Consider tying our delivered stats to the University strategic plan.
    • What data we share and how we share it depends on the audience.  A statistic filled fact sheet for sharing with libraries and provost; applied department-oriented figures for colleagues; fun and informative infographics for the users.
    • Maybe a newsletter would be useful for internal audiences.
  • EZProxy Usage Stats [Russ]
    • Usage is logged, but not analyzed.  "Does it log each link, or only each login?"
    • Program "EZParse" could be useful for analysis.
    • We may be able to get the total logs from Scott Macomber, or capture logs from a set of library computers.
    • Usable as a check against the MINES data.
    • Some concern that our EZProxy format will be difficult to analyze by database or journal.
    • Brendan will look into the logs by next meeting.
  • U-Stats and Cost per Use Data [Ellen & Steve]
    • Cost per use difficult to measure; depends on how we define use.
    • What data should we be collecting and how should we store it?
    • Consider creating a "best practices" procedure for data.
    • We should look at how other schools handle data.
    • There's a problem porting data between Ustats and Alma.  Alma Analytics may be the intermediary.
  • OUT OF TIME!  Next week: Konstantin with the ARL Spec Kit, David with "Analytics and Us", and a MINES analysis update from Linda and Brendan.  Next meeting: 3/23 in the Mugar conference room.



Assessment Meeting Minutes 02/23/2015     
Present: Linda (head), Dan (minutes), Brendan, David, Ellen, Sarah, Konstantin, Russ
Next Meeting:
Monday 03/09/15, Admin Conf Rm.

Announcements:

  • The opt-out open access policy passed Univ. Council.
  • Russ is leaving the committee; Ellen will be continuing to represent Law.
Action Items:
1. All - monitor for 2015 Survey developments as appropriate.
2. All - be prepared at the next meeting to discuss future directions as assigned below.

Update on Survey 2015

We are starting to run low on time to accomplish the 2015 survey this Spring.  If we are not able to accomplish it, our new target would become Spring 2016.  As this is a distinct possibility, all of our materials are now well archived (thanks Brendan) in Dropbox and other locations.  There is also a "readme" file that explains what documents and materials can be found in the archive, and describes their purpose.

Discussion of future work
At the university level, there is a renewed focus on assessment systems for the evaluation of learning outcomes, and the library will be participating in this in the near future.  In light of this Committee’s charge and goals and the Libraries’ strategic plans, what other assessment activities would improve library services and resources.
  • [Ellen] - U-Stats.... cost per use data....  (discuss with Steve, and Jennifer from Law)
  • [Konstantin] - group of specialists to respond to the ARL Spec Survey > ARL Spec Kit
  • [Dan] - centralized point of access for all things library assessment (data warehouse?  Tableau? see * below)
  • [Sarah] - creation of Fact Sheet(s) for university constituents, such as bu.edu/research/  (needs examples, environmental scan)
  • [Linda] - MINES - now one year old at BU
  • [Russ] - investigate what ezproxy logs can tell us about use (and possibly save to U-Stats)
  • [David] - what can google analytics tell us about our library website users?  about our BU Libraries Search users?  about collection use (and possibly save these to U-Stats)?
  • Social Media outreach efforts - tracking twitter, facebook accounts
  • Know just what we do collect and should let decision-makers know vs. what should we be collecting and should let decision-makers know
  • leadership role for all library assessment activities
  • assessment tied to GPA trend
  • focus groups

* From the ARL-ASSESS listserv:

Data Management and Visualization Webcasts

Data visualization brings library stories to life. Using Tableau software, libraries may better harness, analyze, and report their data to internal and external stakeholders. Join us for a three-part series on data management with Tableau, presented by experienced assessment librarians.


Sarah Murphy, Ohio State University

Tuesday, March 3, 2015, 2:00–2:30 p.m. EST

Register online
free of charge

The Ohio State University (OSU) has used Tableau since 2012 to support a number of assessment projects. Examples include a series of interactive dashboards that query, analyze, and deliver transactional data to subject librarians to support their collection development and engagement activities. To deliver library data of interest to the broader academic community, the Libraries has also started to embed interactive, downloadable Tableau dashboards via select OSU Libraries websites.

Sarah Anne Murphy is coordinator of assessment for The Ohio State University Libraries. She earned an MLS from Kent State University in 2000 and an MBA from The Ohio State University in 2008. She has published two books and several papers related to library assessment.


Jeremy Buhler, University of British Columbia
Tuesday, March 10, 2015, 2:00–2:30 p.m. EDT
Register online
free of charge
 

The University of British Columbia (UBC) Library began using Tableau in 2013 to analyze longitudinal LibQUAL+ results. The resulting online display highlights changes in survey results over time and facilitates the comparison of responses across several user groups. Building on the success of this visualization, UBC Library is now using Tableau to encourage library staff to engage with, think about, and find new applications for library metrics.

Jeremy Buhler, MLIS, is assessment librarian at the University of British Columbia Library, where he has worked since 2011.

 Rachel Lewellen, University of Massachusetts Amherst
Tuesday, March 17, 2015, 2:00–2:30 p.m. EDT

Register online
free of charge
 

Tableau helps the University of Massachusetts (UMass) Amherst Libraries staff to both analyze and understand data. Selectors may access purchasing and use data without having to maintain or manipulate spreadsheets. Using Tableau for the analysis of a consortial e-book project minimizes duplication of effort and provides a common perspective for understanding the data. Interactive visualizations expedite access to live data from MINES (Measuring the Impact of Networked Electronic Services), and make it easier for staff to use circulation, service desk, and gate-count data for operational purposes.

Rachel Lewellen is assessment librarian at the UMass Amherst Libraries. She has been using Tableau since 2012

Tuesday, April 21, 2015

2:00–3:00 p.m. EDT

Webcast—Data Management and Visualization with Tableau

Discussion and Q&A with Sarah Murphy, Jeremy Buhler, and Rachel Lewellen



1/26/15

Brendan, Dan, David (chair pro tem), Ellen, Russ, Steve (minutes)

Next meeting: 2pm, 2/9, Mugar conference room

To do:

-Steve will serve as Sarah's backup to resolve the capitalization of Library, Libraries, library, libraries through the three surveys


Bob Hudson updated the committee on the approval status, that the Provost has been given copies of the surveys to review.

Survey pilots: Went well, with almost all 'piloteers' taking and completing the survey.

The committee then reviewed comments from the pilot (see Brendan's 'Review of Encountered Pilot Errors and their Fixes' handout) and corrected errors as necessary.



1/20/15

Dan, Russ, Brendan, Steve, Ellen, Konstantin (minutes), David, Linda (chair)

Next meeting: 2pm, 1/26, Mugar conference room

To Do: 

- Konstantin and Ellen - identify 2 staff who are willing to participate in the pilot

- collect and review responses from pilot testers

-Sarah still needs to resolve the capitalization of Library, Libraries, library, libraries through the three surveys


We’ve reviewed display option on the iPad.  It looks OK.

 Discussion of remaining issues:

QF15, QG2a: Open Access -  we’ve agreed to the changes for points one and two regarding articles published

Resolved the wording for library search – FQ5 -  copy from FQ5, GQ7 and UQ6 - 

FQ6, GQ6, UQ6 -  our agreement: “Research assistance …”  “General assistance”

FQ9 and QG13 - “archiving papers and other publications in digital archives” – include SSRN

“Dear Faculty …” text is finalized

We’ve finalized the email list for pilot participants.

Include some staff to participate in the pilot – Linda will email the link


1/12/2015

Dan, Russ, Brendan, Steve, Ellen, Linda (chair), David (minutes)

Next meeting: 2pm, 1/20, Mugar conference room

To Do

  • Ellen: Check survey on iPad
  • Russ: Propose wording for questions about subject librarians/staff assistance/research assistance


Annoucements:

Linda will be on vacation Jan. 26th - Feb. 8th

Should know Provost decision Jan 22nd.


Discussion

Looked at Linda’s proposed emails for survey recipients. Ellen had minor edit that was accepted and made by Brendan.

For pilot, Brendan recommends that we we just use a non-individualized link for pilot. Can test panels separately amongst ourselves.


Brendan looked at display logic affecting data; there should be no real problem.

There was some editing of Linda’s emails for pilot participants, recorded by Brendan.

Which surveys should staff see in the pilot? All of them.

Linda brought up possible issue with viewing survey on iPad? David couldn’t reproduce on his iPad. Ellen will check it on hers.

Went over various comments and suggestions for surveys. Brendan captured changes made.

Russ will propose wording for Subject librarians/staff assistance/research assistance.

Will use student workers for undergrad pilot.

Next meeting is all about the pilot, and we plan to send it out after meeting.


01/05/2015

Linda (chair), Brendan (minutes), Steve, Dan, Sarah, Ellen, David , Konstantin, Russ

Next mtg: 01/12/15 2:00-3:30, Lg. conf. rm.

To do

  • Sarah: Ensure Libraries is capitalized/punctuated properly across surveys
  • David: Check for wording consistency across surveys.
  • Dan: Respond to Ken about our decision on splitting up teaching/research/clinical.
  • Linda: Draft up emails sent out with the pilot and the real survey.
  • Brendan: Review survey options, flow, and distribution plan.  Determine if display logic will mess up analysis.  Prepare pilot panels.  Modify surveys for pilot commenting.
  • EVERYONE!: Review ALL three surveys, and try to find 3 undergrads, grads, and faculty each (as appropriate), and email their info to Brendan.


AGENDA: 1. Review previous meeting 2. Plan pilot

Review

    BD finished changing the surveys.  Questions have been updated as discussed.

    Linda made chart of each survey's section and question flow.  It all looks good and makes sense.

Pilot

    Linda likes convenient sampling.  “Try 3-5 each"

    We need people from both campuses.

    Steve mentions that data may be difficult to parse due to display logic.

        Brendan will check if this applies to us.

    Panels will be created for the reviewers

    For undergrads, don't use student workers, but their friends are okay.

    We should mention we’re collecting demos in the intro.

    Show Marlene, Mary, and Amy the survey before pilot.  “We thought you’d like to see this before it goes out.”

    Everyone’s Spring recess ends after March 15th. Target release date: April 2nd.

Advertising Update

    Sarah showed off the adverts.  Very nice.  Still some tinkering here and there.  Lots of places we’re advertising, still waiting on even more locations.

    Russ: "Can we modify our websites for advertising the survey?"  Linda: “IRB says about 1 advert in each library is fine.  More than one will shift our user group         toward the patrons."

Scheduling: Mondays 2-3:30 in the big conference room for the next 5 weeks, EXCEPT January 20th, 2-3:30 instead of January 19th. 

Meeting adjourned.  


12/22/14

Steve, Dan, Brendan, Sarah (minutes), Ellen, David (Chair), Konstantin

next mtg: 1/5/15 2pm conf rm.

AGENDA:

Pilot Surveys


OA Questions

SDS presented new wording for OA questions on grad and fac (FQ36, GQ2a)

Regarding articles you have published: [Respond Yes/No/Unsure]

    • Have you published an article in an open access journal?
    • Have you paid an additional fee to make an article open access?
    • Have you deposited an article in OpenBU (BU's institutional repository)?
    • Have you deposited an article in a discipline-based repository (e.g., PubMed Central, SSRN, arXiv)?

On the faculty survey, the open access factor on the question "How important are the following factors in your decision on where to publish journal articles?" will read: 'Journal allows open access (immediately or after an embargo)'.

DF suggested a small wording change: move “have you” after the “regarding articles you have published” and start each question with the verb (published, paid, deposited).


Info lit Questions

DB reports from conversation w/ Ken Liss

already implemented wording on guides/tutorials

separating teaching from clinical/research purpose (FQ 32,33,43,45) - KL interested, we were not open to making this large a change at this time

consider in survey keepers- separating teaching from research from fac, or teaching from study for grads.


Streaming Media

GQ17 (could be added to fac FQ 45 as well) Streaming Media

SDS - yes wants this question

fac interested in JOVE; Krasker DVD based, is there interest in other?; Lots of audio, not much video currently

BD has concern about “streaming media” wording could be interpreted different ways (synchonous vs asynchronous, live streaming vs netflix style)

switch to “online video and audio resources”?

“streaming audio and video” 

decided on wording:

“Access to streaming audio and video resources” for both GQ17 & FQ45


Review of Questions/issues from BD’s email 12/15

discussed, BD will make changes


Assessment Meeting Minutes 12/15/14

Attendees:  Linda (head), Dan, Brendan, David, Ellen, Steve, Sarah, Konstantin (minutes), Russ, Vika (called-in)

Next Meeting: Monday, December 22, 2pm Admin. Conf. Rm.

NEXT MEETING: Agenda

  1.            Question “streaming media” 
  2.        Report from Open Access group  (Sarah and Steve)
  3.         Discuss Ken’s second suggestion
  4.        Respond to Brendan’s email (all)


12/15/14 Agenda

1.       Minutes (thanks Ellen for the terrific minutes), announcements, changes to the agenda?

2.       Discussion with Vika about our OA questions (Sarah)

3.       Brendan’s overview presentation and outstanding questions (Brendan)

4.       Discussion about Ken’s feedback on the survey’ Information Literacy questions (Dan)

5.       Publicity files, Brendan’s suggestion about publicity

6.       Do we want to add an augmented services question to the undergrad survey?

7.       Pilot


Brendan’s overview:

Brendan walked us through all the major and minor changes. (see Brendan’s power point presentation). 

The group discussed question “During that past academic year at BU have you …. 4th option: “Worked on a research project including data”.  We’ve decided to keep the wording as is.

Double check if IS&T term is clear to the medical users -  include BU/BUMC IS&T ? …. (Konstantin)

We’ve discussed the advantage of mentioning “Blackboard” in a question which asks about “course system”

Sarah brought up the point that Ken’s suggestion that library guides need to be reincorporated in the undergrad and graduate surveys.

We’ve discussed open issues concerning service augmentation.  Russ suggested to change the word “augmented” to “enhance”. We all agreed.

We’ve decided to discuss the potential of removing “media streaming” during the next meeting.

 

Discussion with Vika about OA questions:

Sarah summarized Vika’s concerns and Vika clarified that we shouldn’t focus just on “open access journals” vs. open access in general.

Vika suggested to formulate a separate question that asks about Gold and Green access.

Discussion followed. Steve argued that we’re trying to find out if our patrons are making their work open access rather than publishing in the “open access” journal.

We’ve decided to modify the question so that we capture “immediately or after embargo” wording.

Question “Have you ever submitted an article to: ….” -  We’ve decided that we have to consider the wording in the context of Vika’s suggestion.

Sarah emailed Vika’s suggested wording.


Ken's suggestions

Dan discussed Ken's suggestions, but we still need to revisit them.


Augmented Services for the Undergraduates

Russ suggested that we substitute the word "augmented" with "enhanced". The group unanimously agreed.



Minutes

Assessment Meeting Minutes 12/8/14

Attendees: RS, ER (minutes), LP, BD, DB, SS, DF (for last half)

Next Meeting: Monday, December 15, Admin. Conf. Rm.

We focused on a group review of the completed graduate and faculty surveys

-Grad & Faculty Surveys will NOT have a “demographics” bloc header—decision is that is it not necessary to prep users to answer personal questions.

Sarah is setting up a phone in with Vika for next Monday’s meeting to discuss the Open Access wording on the survey and what data she would like us to collect.

Things to cover at next meeting: Open Access questions and language with Vika, input from Ken on Information Literacy, conventions/consistency of language across all three surveys with Brendan, review the three surveys in their entirety, data management questions with David, and discuss the pilot

General

On satisfaction and importance questions, IMPORTANCE rating will be asked first, and then SATISFACTION rating

-Brendan is looking at alternative phrasing for “information literacy” and also looking at the three surveys together to see where language should consistent, but isn’t—he will present these at 12/15 meeting

Faculty Survey

-David is working on language for Research Data questions

-“Research data” option has been added to question about “during the past academic year, have you…” and set up as display logic, so only survey takers who have worked with research data will see questions about storage and sharing

-Question on bibliographic managers may be moved to different block, but this was not yet decided

-Question on assigned readings and how students access—to be reviewed with Steve and Konstantin at 12/15 meeting

-Dan is following up with Ken on the language for the information literacy question (FQ30)—this language must be to us by 12/15

FQ32—discussed switching assessment and importance (to importance and assessment) but decided not too, since it would be leading, but are aware we may now be comparing apples and oranges

-“Information literacy” is library speak, but may have to remain

Grad Survey

-We agreed to remove “other” from the importance & satisfaction re library resources questions

-We agreed to add an option on current awareness to our question on augmenting library services—this question will also be copied to the faculty survey

-We changed the question on library use and space (group and individual study space, access to computers) to focus on HOW graduate students use the library resourceà this change needs to be copied over to the undergraduate survey

_______________________________________________________
Assessment Meeting Minutes 11/17/2014     
Present: Linda (head), Dan (minutes), Brendan, David, Ellen, Steve, Konstantin, Russ
Next Meeting:
Monday, 11/24, Admin Conf Rm.


Announcements:

  • Discussion of the Information Literacy questions with Ken Liss is delayed for one week.  Ken will join us next week and he and Sarah will lead that discussion.
  • The 2015 Undergrad Survey is tentatively complete.
Action Items:
1. Faculty Survey Subgroup should note the discussions in the high level review below and make necessary edits.
2. Undergrad Survey Subgroup should be prepared to walk the group through a preview of that survey at the next meeting.

High Level Review of Faculty Survey 
  • On Q32 (importance of resources) - The subgroup should remember to add the satisfaction scale to these (but not the use scale, as this is the faculty survey).
  • We decided to move the section on publishing & OA to near the end, but not past the last 'Overall' section.  Brendan accomplished this during the meeting. 
  • On Q43 (importance of services) - The subgroup decided to add Q44 (the satisfaction with same services) to this question, positioned horizontally next to it.  Not yet accomplished.
  • On Q45 (augmenting services) - We added a data management "statement" (note: this is the Qualtrics term for a row within a question).  The subgroup will work out the examples given and match the wording of the question on the grad survey.
  • For the Info Lit block of questions, the subgroup (actually, all subgroups) will monitor the results of our coming conversation(s) with Ken.  We did note it would be great to fold these 4 questions into 2.  Also noted the 'Providing correct citations' statement became 2 statements for the 2012 grad survey, and really could be 3 (providing appropriate citations, providing correctly formatted citations, and using information in an ethical manner).
  • On Q47 (overall contributions) - The scale needs to change to match the grad survey's scale.  We also added an "overall career success" statement.
  • At the end of the survey we considered other questions (Brendan noted these in Q51):
    • Put Bib manager question on all surveys (see undergrad).    
    • Take Faculty version of reserves question. "How do you direct students to resources you assign?"
    • Don't need first part of bib manager questions. Reevaluate options.

___________________________________________________________________________________________________________________________________________


Assessment Committee Meeting, Monday, November 10, 2014

1.       Minutes, announcements, changes to the agenda

a.       No changes were made to the minutes of the last full committee meeting submitted by Ellen

b.      The following announcements:

                                                               i.      Brendan noted a possible completion rate problem in the MINES survey, but he observed—and the rest of us agreed—that it did not represent a significant problem.  Even we had to accept the lower rate, it was still quite high and not worth intervention.

                                                             ii.      Brendan also noted that in Qualtrics we can make notes and comments within the draft questions themselves, allowing us to keep all of the committee’s work product there and make it more convenient to read everything in one place, not only now but in the future.

                                                            iii.      Linda announced that Ron Yeany and his team at IS&T have offered to look into whether they can help in extracting useful usage data from Exproxy.

c.       No changes were suggested to the agenda proposed by Linda.

 

2.       Continue discussion about usage, importance, and satisfaction and what changes we should make on our surveys.

a.       Continued and reviewed the ongoing discussion we have had about whether to use usage or satisfaction in our survey. David and Steve quickly reprised the issues for us.

b.      Konstantin registered his concerns about using importance and offered the white paper from the Customer Satisfaction Research Institute, along with four other relevant articles in the literature, as food for thought in our consideration when considering the use of importance in our survey.

c.       Steve noted that if we moved away from importance we would lose the ability for longitudinal study, since we used importance in the first round of surveys.

d.      We returned to the question if we could not use all three factors in the survey as suggested in the last meeting.  Russ queried if we could use the pilot period to test if the three factors could all be used in the survey without making the question too long and cumbersome, confusing the test taker.

e.      Linda noted that we may have more room in the surveys for a few longer questions given that we are hoping to collect demographic data from the Registrar and Human Resources, rather than by asking the respondents.

f.        Sarah observed that if we used all three factors that would give additional insight into patron feedback.

g.       Consensus:   all three elements will be used in the pilot on questions related to services.

 

3.       Demonstration/review of the Graduate Student Library Survey

a.       Graduate Student subcommittee:  have had a high level discussion and are now having a more targeted discussion.  The subcommittee has met twice and needs additional time to discuss issues, but that can follow this discussion by the full committee.

b.      A question was raised whether we can use the panels in Qualtrics for the distance education and online learners.

c.       Question 8:  Brendan wondered if the wording is loaded; Linda did not believe it was; Steve suggested using “Please rate the importance” as a way to resolve the possible loaded nature of the question.  The question was also raised if the elements surveyed are really separate or do they represent a skillset (or many different skillsets).  Sarah suggested getting Ken Liss’s insights on this.   She also recommended inviting him to our next meeting to get his thoughts here as well as on other info literacy questions.

d.      Question 9: on first line it was recommended to use just BU Library Search and then on the second line use “Databases” and search tools.  Russ and Konstantin also recommended using the examples of HeinOnline and PubMed, as was similarly done on the faculty survey.  We also considered whether we needed new examples of Social Media beyond Twitter and blogs, or need any at all.  The consensus was to keep the two existing examples.

e.      Question 17:  Sarah wondered if we should not change the wording to be more like UW’s analog question #8, to be more user-centric.  We agreed that the subcommittee in its next meeting would investigate this further and make recommendations to the full committee.

 

4.       Progress report on the Faculty Library Survey

a.       We have had a high level committee discussion and one subcommittee meeting discussion.  Issues in the subcommittee, principally about the use of usage or importance, were then discussed by the full committee at the last meeting as well as in the early portion of this meeting.

 

5.       High level review of the Faculty Library Survey

a.       Additional high level discussions need to occur.

 

6.       Next steps?

a.        Undergrad survey is very close to being finished.  Needs an additional half hour of the full committee’s time.

b.      Grad survey needs discussion in two more meetings.

c.       Faculty needs two more meetings, although we may be able to shorten that time inasmuch as we are utilizing language developed in the new grad survey in corresponding sections in the faculty survey.

d.      Should we include post-docs in our survey, and how?

 

 _________________________________________________________

Assessment Meeting Minutes 11/3/14

Present: BD, DF, LP, ER (minutes), SDS, DB, RS, KS

Next Meeting(s):

Graduate Subcommittee: Friday, November 7, 1-2:30, Mugar Small Admin Conf. Room

Undergrad Subcommittee: Monday, November 10, 12:30-2, Administrative Offices, Mugar, small conference room

Full Assessment Committee meeting, Monday, November 10, 2-3:30, Administrative Offices, Mugar, large conference room

Announcements:

Brendan gave an update on the post-survey information we will use to capture entries for the gift cards, and he will send it around.

There have been no changes to the undergraduate survey since our reserves discussion last week, and the undergraduate subcommittee will update us at the next meeting.

Action Items:

Whole committee: need to look at the larger information literacy question issues with Ken before we can decide on which word to use

SRS, ER, KS: Add question about subject librarians to grad student survey

All subgroups: make survey questions closer/more parallel across the board (see slide 11 where DF has compared the same questions across the three surveys, highlighting the different language we use)

Undergrad survey/undergrad subcommittee: we have lumped together questions on physical things and services, and these should probably be two separate categories (see slide 15)

Discussion re David’s presentation on usage vs satisfaction vs importance:

Presentation (with DF’s recommendations on what word to use with which question) is on the wiki

Overall: we are considering adding satisfaction questions to the faculty survey, but will ask undergrads only usage questions.

Whether to continue distinguishing print vs ebooks? Key points:

·        Really two separate questions: (a) how important are these sources (print vs ebook) to you and (b) how satisfied are you with the selection we have at BU?

·        Do we care about responses if it will not change our format selection? (i.e., we are an e-preferred library)

·        Are we focusing on satisfaction with the format of our materials or our collection (e.g., “you don’t have what I want” vs “you have what I want, but not in my preferred format)

o   Decided are trying to get more at satisfaction with the collection and so will combine the formats, which is what we did when asking about our journals collection

o   May also add the word “collection” to the question to really get people to focus in on it

·        This question will also have a comments box

·        Determined there are no questions about e vs print materials we want to add, as we already know what we need to on that topic

Importance vs Usage re: Library Services

·        Satisfaction questions are good for giving us longitudinal data, but are not particularly useful for capturing information from just one survey

·        How to capture usage?

o   If faculty member is on sabbatical and we ask about the past year—will they say they have not used anything, when ordinarily, they heavily rely on our resources?

§  Alternatively, will faculty pay attention to “current year” or look at their normal course of business?

§  But “current year” should definitely be kept for ugrad survey, as it helps us to look at what resources each class uses

§  Grant cycle is 3 years—if faculty are not writing grants in this current academic year, will usage questions underreport on resource usage/importance?

o   Use of importance AND satisfaction questions together could give us a better idea of where we need to improve (e.g., less heavily-used resource may still be important, especially with varying workloads over the course of a faculty member’s career. Should we not limit to the current year for faculty, as it may provide an inaccurate snapshot?)

o   What can this data really tell us? How would we tell if something is low use because they are dissatisfied or if it is low use because it is not an important resource?

·        Faculty use in past year vs importance to faculty—

o   How to capture resources that are important but may not have been used in the past year and/or are not necessarily used on an annual basis?

o   Issue: probably don’t want to ask faculty all three questions (is it important, do you use it/how often do you use it, are you satisfied with it?)

·        Issues with deriving usage stats from satisfaction data—e.g., on the undergrad survey, they reported satisfaction higher than their reported usage, but can also be important—the discrepancy itself provides useful information

·        “use is use”—usage questions wouldn’t tell us the type or quality of use (i.e., getting help with a paper vs asking us where the bathroom is)

·        Should we ask all three questions, but cut down the number of services we’re asking about to five or so?

·        Services for which usage is declining may still be important (i.e., ILL)

·        Currently, the usage question is a yes/no, not a likert scale. What is the value of asking it this way?

o   If we give up the usage question, we give up being able to report usage

o   Likert scale is also misleading, because what might be frequent for one user wouldn’t be frequent for another

·        Tip people off at the beginning with a header like “for the following services, we will ask you to think about whether you use these services, how important they are to you, and how satisfied you are with them” – makes people more likely to answer questions, rather than getting bogged down by never ending questions and quitting the survey?

·        Can usage be pulled from other areas so we don’t have to ask the question (i.e., get ILL stats from ILL librarians)?

o   This question helps capture the perception of services they think they use

o   Capturing usage in the survey helps us correlate use and importance in a way that cannot be captured with other data sources on use

·        Degree of use—do we care? à kind of an open question

·        Importance might also measure unique, non-repetitive events, as opposed to regular use of an itemà potential for skewed/misleading results

Overall: committee members agree on main points with the exception of the use vs importance questions

Suggestions for resolving this issue:

·        Make our ideal survey, then edit

o   Need to be careful we’re not making an entirely new survey and losing comparability to prior surveys

·        Create a matrix of Use, Satisfaction, and Importance with two categories: Facilities and Services

·        Is there a more elegant way to ask about what library you use (that would help eliminate the need for separate facilities questions, so we only need to ask about services)?

Can we bring in some topical questions (i.e., data management) that we can change each time?                    

 

 

 


Assessment Meeting Minutes 10/26/14
Present: SDS, DF, KS, ER, LP, BD, SRS (minutes), RS, DB 

Announcements

Brendan went to Google Analytics workshop on GA for libraries in Washington DC, and reported back on some of what he learned:
We should make analytics user friendly for our  librarians to access/use
See if IT planning on upgrading to newest version
Suggested consolidating “properties” (Primo and BU Libs website) to track use better (also Law, Theo and Med sites?)
Implementing goals (eg what subpages are branches wanting to push), setting goals given the BU libraries organization and heterogeneous users/uses may be a challenge
Using existing dashboard templates for new dashboards
Additional webanalytics tools to augment those analytics available from GA (eg clicktale.com, usertesting.com, qualaroo.com, compete.com)
LP - Post-survey we should form a subgroup to look at implementing these suggestions

DAG is looking to do some usability work specific to the BULS searchbox
Subgroup for Faculty Survey review will meet this Wed 10/29


UG survey:
Intro needs provost/IRB review but otherwise done
Minor wording and ordering changes on Page 1
Reserves Question
Reviewed 41a and 41b, two approaches to this question
 a - percentage use (tho may not add to 100)  
 b - last use snapshot 

Broadly, we preferred 41a. We established that the scale we want is use-frequency, though there are concerns about how students will interpret that.

Existing subquestions 1 and 2 (print and online reserves) can be combined, but for “find library resources on your own”  separate into "from library" vs "purchase"
1. Reserves 
2. Blackboard 
3. go find your own readings (from library)
4. purchase

Wording note: Must make textbooks inclusion/exclusion unambiguous

May not want this exact Q on grad survey b/c of multiple roles grad students may have (as both teachers and students), and because law/med/theo's approaches to collection/reserves may differ from Mugar & branches' (eg Law buys textbooks)

**PILOT this question early and revise as needed

Individual libraries at BU section
Law name change – CHECK ALL SURVEYS FOR THIS (RS/ER)
Skip logic magic was demonstrated
Changes: seating/desks, dropped security, added outlets

Whirlwind update from Grad Survey review group

  


Assessment Meeting Minutes 10/6/2014

Present: Linda (head), Brendan (minutes), David, Ellen, Steve, Konstantin, Sarah
Next Meetings:
  • Undergraduate Subcommittee: Friday, October 17th, 12:30-2:00 @ SEL 
  • Graduate Subcommittee: Thursday, October 16th, 2:00-3:30 @ Law Conference Room 
  • Assessment Committee: ???

Action Items:
1. Find things to improve about the graduate/faculty surveys.
2. Review undergraduate survey post-edits [[ ignore bracketed sections ]]
3. ???


Review of Changes to Undergraduate Survey 
  • Linda agrees to work on the introductions for the surveys. Shooting for ”5 Ipad minis for undergrads, 4 for grads, and 3 for faculty.”
  • Put display logic on a single “Individual Libraries” page so it pops up in page and makes the survey feel more condensed. 
    • POST-MEETING UPDATE: This isn’t possible due to display logic glitching. We’d have to change our format to make it work.
  • We should double check the surveys for the naming conventions for the libraries. With Law being renovated, should we change its name in the survey? It seems like we’ve been using the colloquial names rather than the official names in some places.
  • Section headers were added and embiggened.
  • Included pop-up question for people who have used a bibmanager, asking which ones specifically.
  • Added Steve’s reserve question. Undergrad subcommittee will iron out the wording/options/likert format.
  • Do we want a question about mobile devices? 
  • Question: Will the Qualtrics mailer be blocked as spam? 
Graduate Survey Collective Critique 
  • Check whether distance ed. Is capturable from the grad. info. The rest of the demographics can be removed.
  • “Are we asking open access questions for graduate students too?”
  • Remove primary discipline since we get that information.
  • Brendan and David agree to help the other subcommittees for consistency between surveys.
  • Consider separating BU Library Search from other databases (see Q39).
  • Maybe we should separate out service/facilities questions.
  • Remodel the favorite library questions after the Undergraduate Survey. Sarah: “Study Carrels” 
Publicity Update – Sarah

  • Meeting with Mary McGowan’s Anita Greene Scholars. We can get both vertical and horizontal designs from them. We’ll try to get them on the bus too. Posters should come up first, and then emails will go out. And they’ll have them ready by Thanksgiving and the students will be available for rebranding through intersession. Design style is “like a comic book, but not like that dark stuff”. 

General Notes and Miscellany 
  • Brendan and David agree to help the other subcommittees for consistency between surveys. 
  • We should consider looking for a new 9th committee member to replace Lisa.
  • Konstantin: “Should we start gathering reference use data for grads/faculty in the surveys?” 



Assessment Meeting Minutes 9/29/14

Present: Brendan, Dan, Steve, Lisa, Ellen, Russel, Linda (chair), David, Sarah, Konstantin (minutes), Ken Liss
Next meeting: Monday 10/6, 2pm-3:30pm, Mugar Administrative Conference Room

Action Items:

         1. Minutes, announcements, changes to the agenda?

          2.   Publicity update (Sarah)

         3.   Full committee high-level review of the Graduate Student Library Survey

         4.    Discussion about Undergraduate Student Library Survey (Undergrad sub-group)

         5.    Next steps?


  • Timeline
    • We should submit to the IRB in mid-January
    • Before we submit we should receive approval from the Provost office for ALL files (i.e. publicity, conflict of interest, surveys, etc.)
  • Weekly meetings
    • We will be meeting weekly, not biweekly
    • The following groups will be responsible for reports
      • Faculty survey: Steve, Russ, David
      • Grad survey: Sarah, Konstantin, Ellen
      • Undergrad survey: Dan, Brendan, Linda
  • Report from the Undergrad survey group
    • Brendan shared the file "Suggested changes for the Undegrad Survey"
    • All agreed that we should review Intro, exclude undergrads >18 years old by email distribution, reorder sections (Resources, Services, Individual libraries)
  • All agreed that we should follow up with focus groups




Assessment Meeting Minutes 09/15/2014

Present: Brendan, Dan, Steve, Lisa, Ellen, Russell, Linda (chair), David (minutes), Sarah
Next Meeting: Monday, 9/29, 2pm-3:30pm, Mugar Administrative Conference Room 
Following Meetings: For rest of year, will be meeting every other Monday at 2pm, except for one Friday: 9/29, 10/17 (Friday), 10/27, 11/10, 11/24, 12/8, 12/22

Action Items:
Sarah: Remind reference librarians about the ongoing MINES survey
Linda: Ask for input from Ken Liss on questions about instruction and services (Q8 & Q9 in undergrad survey).
Steve: Propose question to capture undergraduate use of "reserve" materials
David: Talk to JD about question on bibliographic managers, and which ones to list in question.
Brendan, Dan, and Lisa: Work on undergrad survey for next meeting, including, comparing to MIT and UW surveys, looking at qualitative data from last survey, and proposing any needed wording changes.
Sarah, Ellen & ???: Similar work (for future meeting) for grad survey.

Announcements:

Dan announced improvements to the websites Google Analytics, which now track searches from across site, including branches.

Russell introduced our newest member, Ellen Richardson, Assistant Librarian for Administration at the law library.

MINES Update

Brendan showed the current state of the survey. We have had almost 3,500 results so far, with an extremely high completion rate. Brendan has downloaded the data (so far) into Excel to do some preliminary analysis, including parsing the URL to let us see what resource is being visited. Extremely impressive.

Brendan will share data with anyone on committee who is interested. We can look at it and use it, but remember it is preliminary and hasn't yet undergone complete analysis.

Next Survey

Linda reported that until we hear from Provost (nothing yet), we still don’t know for certain whether we will do everyone in next survey, or only faculty. We will finish work on updating undergrad survey; then do the faculty survey update, in case that is the only survey we administer.

Decided that for each survey we will do pass as a group to make sure we have right questions, without wordsmithing. Then break into groups to look at wording and compare to other surveys.

Undergraduate Survey

Sarah suggested we may want to speak to Ken Liss to find out if the questions on instruction and services (Q8 & Q9) are really the metrics he wants to track. Linda will do it.

For the questions about facilities (Q13), we need to rethink our categories. "Furniture" can be confusing. Seating vs. Chairs. Outlets. Lighting.

On Q18 regarding course reserves, Steve brought up wanting to know about where students are getting resources we traditionally think of as e-reserves (e.g., the library's e-reserves system, Blackboard, faculty web site, etc.). There was a lively discussion on whether undergrads (as opposed to faculty) are the right ones to ask, and whether a question can be written that captures what we want to know from undergrads. Steve will work on possible wording for question.

Sarah remembered there being useful feedback on survey in qualitative data. When we break into groups, groups should look at qualitative data for their survey for comments about the survey.

Q21: The question was changed from Yes/No to a scale; need to change wording of question to reflect change.

Decided that if undergrads answer "Yes" to Q22 about using bibliographic manager, there will be a further question asking which one(s) they use. David will talk to JD about what should be on list.

Because the wording for the comments boxes refers to the page titles, we need to add header to pages, or change wording of comment question.

Some time was spent on the perennial use vs. importance vs. satisfaction problem, with no solution.

Brendan, Dan, and Lisa volunteered to work on detailed wording for undergrad survey by next meeting. This will include comparing to MIT and UW surveys, and looking at qualitative data from last survey

Sarah & Ellen volunteered to look at wording for Grad survey (for future meeting).

Brief discussion of a replacement for the irreplaceable Lisa, who is leaving us. We are a large group, so we don't have to replace her. Linda suggests we consider needed skills in making decision.

__________________________________________________________________________________________________________________________


Assessment Meeting Minutes for 07/31/14

Present: David (temp chair), Dan (minutes), Steve, Brendan, Lisa
Next meeting: TBD - Doodle; Administrative Conference Room Mugar

Action Items:.
David will send a Doodle out to help schedule our next meeting due to August vacations.
Brendan will continue editing the undergraduate survey draft on Qualtrics, according to decisions made today.

Dan will share the archival surveys on Qualtrics, but set permissions so that no one can edit them.

All can look at the undergrad draft to identify issues, better wording, missing questions/options/themes etc.

Announcements:
- no announcements

Discussion of 2015 Library User Survey Question Development
 
Lisa lead discussion of the last sheet in this Google Doc, called "Comparing Under Q BU, UW, MIT". 
 - not much different from our to UW
's survey: their survey is nice and very short
 - Answer rows on MIT survey Q2 are nicely specific.
 - We also looked at MIT Q11's layout (aware | importance).  We considered how this might apply for us, considering we often ask about use/importance/satisfaction.  We looked at our questions 9, 25, and 26 in comparison.
 - We have something very similar to MIT's Q12.
 - MIT Q16 + Q18 include library instruction outside of a class; probably not that useful for BU (could mean HGARC events etc?).
 - MIT did not ask overall importance of libraries
 - MIT Q24, the $100 to spend question.  Poorly worded and makes respondent add - not good for fatigue.
 - Lisa also noted that there may be some Qs from MIT not represented at all on the spreadsheet. She did not include them because there were no comparable Qs on our survey.

Discussion of editable
surveys on Quatrics
David began this with the question of when, exactly, might it be favorable to combine the data coming from undergrads, grads, and fac.  There seem to be very few questions where we would want to do this, and without weighting the numbers for response rate, the numbers would be skewed.  We decided to begin composing Qs on three separate surveys.
As we will begin editing the Qualtrics surveys, Dan will share the archival surveys on Qualtrics, but set permissions so that no one can edit them.
We began looking at the faculty survey questions on Qualtrics.  We considered the frequency of physical and virtual visits, which were combined in 2010 but later separated, and we added a "during the current academic year" phrase as well.  We looked at the undergrad survey in comparison, and realized we should edit that first as it was the latest, and would need the least editing.
Do we need to ask about majors again?  Used those to send comments to specific librarians, etc.
Brendan noted the prevalence of the N/A option on many questions, and we noted the consultants advised having this "way out" for respondents.
Our question 13 - piped text from previous question to set up to work in Qualtrics.  Steve is satisfied with the options on Q19, to look at changes over time.
MIT asks about USE of more specific sources or collections than we do. We considered setting up our Q differently, but we do have many other statistical sources on collection usage, so we decided NOT to add any about specific USE other than the options we already have listed in the survey.
Brendan suggested that we could create a numbering system that would indicate which questions are longitudinally useful.
We completed an initial run-through of the questions on the undergrad survey, with some edits left for Brendan to finish up later, as he was "driving" the Pardee room computer and editing on the fly.

Scheduling next meeting
We looked at scheduling, but could not get around vacation schedules.  David will send a Doodle out to help schedule it.
_____________________________________________________________________________________

Assessment Meeting Minutes for 7/14/2014


Present: Linda (chair), Brendan, Konstantin (minutes). Lisa, Russell, Steve, David


Next meeting: Thursday, 7/31/14 at 12pm (at Pardee)


Action Items:


- Ask Bob Hudson to talk to Provost (Linda)


- Look at the questions (David)


- Discuss Brendan's action plan (all)


- Take a look at Konstantin's analysis (all)


- Update numbers on the spreadsheet (Dan)


- Create the Google doc (David)


- Wording, style, etc. - comparison of MIT and U of W - tabs: faculty (Konstantin), grad (Helen/Russ), undergrad (Lisa)



Summary:


Russ suggested adding Helen Richardson for membership on the Assessment Committee. (all agreed)


Dan talked about Google Analytics - workshops - comparison functionality, bounce rate, exit rate.


Dan reported on survey transfer. To access surveys on Qualtrics: go to bu.edu/qualtrics


Discussion followed in regards to our survey plan


Brendan's report: meeting with Pam Andrews (Lauren Hess replaced Pam in 7/14). Creation of panels. We discussed the advantages and disadvantages of working with the Provost office.


Konstantin raised a question if BMC affiliates will be included in the panel.





Assessment Meeting Minutes for 6/2/2014

Present: Linda (chair), Brendan (helped with minutes), Konstantin, Lisa, Dan, Russell, Steve, Sarah, David (minutes)

Next meeting: Thursday, 6/19/2014 at 2pm

Action Items:

  • Dan: Transfer surveys
  • Lisa: Talk to Ken about Ithaka Q15.2 and 16.1 and Q4(?)
  • Konstatin: Looking at categories for each question. Talk to colleagues on medical campus about survey needs, and data management
  • Brendan: Working on Google Analytics
  • David: Help with display logic, ask JD and Jenna for input on data management question
  • Sarah: Ithaka Q 22.2: Sarah will compare to our faculty Q35 to see if there is anything we want from Ithaka. Look at reports to assess importance of disciplines
  • Russ: Examples of where WA survey wording better than ours

Brendan: MIT Survey:

  • Some questions are much more specifically oriented. We’ve been trying to stay general with our questions but perhaps now we should start considering using more targeted questions.
  • Linda: “We should start keeping track of the “follow-up questions” like that one.
  • MIT Q13 (where to start looking for a specific ebook) could be useful as a simple statistic/longitudinal stat. Ithaka has a similar question.
  • Q24 (“How would you allocate $100 to improve library system”) Brendan likes a lot. “It’s fun!” Pretty easy to analyze too. Problems: many people can’t add up to 100, and the options may not be open ended enough.
  • Q25’s second question (“Are there other library services you think MIT should be offering”) is good in Linda’s eyes. It focuses on what the users want, not what we think they want.

Dan: Transferring Surveys

  • Skip Logic causing problems.
  • Display Logic ALSO causing problems.
  • Dan will continue to tinker with help of David/Brendan.

Russ: Comparing UW grad & undergrad surveys

  • “Is it typical for us to combine graduate and professional students?”
  • Sarah brings up concerns about using qualtrics for this survey if we’re having so much difficulty with skip/display logic. (“Well we have SurveyMonkey for another year if we need it”).
  • We really need to focus on the wording.
  • Possible wording: “Have you used the library’s services for an internship or sponsorship?” , or something to that effect.
    • “We followed UW’s model.”
    • We should include Marlene if we’re going to continue to discuss wording for Law users.

Lisa: Ithaka Survey

  • Ithaka Q 7.3: possibly useful for follow-up to survey
  • Q 13.4: Possible springboard for question regarding publishing Steve: compare faculty attitudes towards publishers vs. libraries for publishing advice, etc. Russ: should library be responsible for some of these issues?
  • Q 15.2: Question about whose responsibility it is to teach information literacy
  • Q 16.1: Bring Ken into discussion about whether we want something like 15.2 and 16.1
  • Q 22.2: Sarah will compare to our faculty Q35 to see if there is anything we want from Ithaka.
  • Q 6.5 and 8.3: David will ask Jenna and JD to look at data management questions and give advice

Assessment Meeting Minutes for 05/19/14

Present: Linda (chair), Brendan (minutes), Russell, Dan, Sarah, Konstantin, Steve, David, 
Next meeting: Monday, June 2nd, 1:00-2:30; Administrative Conference Room Mugar

Action Items:
 - Russ: Compare UW grad & undergraduate surveys.
 - David: Compare undergrad & grad BU reports.
 - Dan: Transfer all other BU surveys
 - Konstantin: Compare the other BU surveys to each other.
 - Brendan: Compare the other BU surveys to each other & troubleshoot Qualtrics issues.
 - Sarah: Determine any problems with ceasing to collect discipline information & survey keeper


Announcements:
 - Brendan received a promotion to Assessment Data Specialist (or something like that).  Now we can offload all our projects onto him!


Plans for New Survey:
 Linda brings up two important questions:
  1. How will we go about keeping track of our ideas/things to update or edit?
    • If we use the minutes, we'll need to be really organized.
    • Survey Keepers (see Wiki's side bar) are used to keep notes about conventions we've agreed upon, past-encountered problems, and survey decisions to avoid in the future.
    • General consensus to use minutes.
  2. Should we do one per year, or a three-in-one survey every three years?
    • Steve: Our response rates may change if we don't actively target a single user group.
    • David: If we put out a single report at the end for all three, it won't be targeted enough.  We should put out individual reports for separate user groups and perhaps an executivesummary of all three.
    • Sarah: We can filter the survey by user group.
    • Longitudinal statistics casually discussed.  If we want to start recording longitudinal stats, we need to make the changes with this report.  They have been used by other libraries and yielded interesting results.
    • If we do one survey per year, we're putting out one report per year as well.  Worries about survey and analysis burnout float around.
    • David: We could also have the home page link separate surveys for other user groups (everyone likes this idea a lot)
    • We could try using display logic in place of skip logic to bypass the skip issues.
    • General agreement on 3 surveys in one year, each tailored and targeted toward the specific user group, linked from a main survey page, and individual reports instead of a comprehensive report.
Dan's update - Uploading Old Survey to Qualtrics:
  • Differences between the two:
    • The introduction transferred over into Q1.  Not noticeable from user's screen.
    •  Problem where the respondent cannot use the back button over any logically skipped survey content.
David's update - Reviewing Survey Reports:
  • The reports referenced every question, but there wasn't a lot of deep analysis.
  • "Part-time" faculty wasn't a particularly useful demographic.  We should exclude it next time.
  • Question: are adjuncts considered part-time?
  • Q3 wasn't particularly useful either (whether faculty taught or used federal aid)
  • Perhaps we can simplify by not collecting department information.  Sometimes departments are fuzzy with self reporting anyway.
  • Brendan: "Can we extract info from email?  MIT collected no demographics and used email addresses to create demographics."  IRB would have a conniption fit if we started collecting personal data like that.  The anonymity would not exist anymore.  Not worth the time.
  • We can use conditional questioning to work around the parts people find confusing (e.g. with Med & BMC/school related fields)
  • David: "I want to ask about data management in the open access section"
  • Ranking importance is fine, but what we really could use is frequency data. 
  • We should add "in the last year" and drop "academic year" in place of "last year" so the data is current and not related to past experiences.
Russ - Analyzing UW Survey:
  • "UW's survey is about the survey taker.  Ours is about what we think is important to them."
  • We need to package questions from their vantage point.
  • UW's survey is clean and minuses.  Ours can be wordy at times.  It has its plusses & minuses.
  • They included phrasings like "Thinking back to your last..." kind of questions.  May be more useful for infrequent services.
Out of time. Next meeting decided for June 2nd.

__________________________________________________________________________________________________________________________
Assessment Meeting Minutes for 04/03/14

Present: Linda (chair), Dan (minutes), Russell, Brendan, Lisa, David, Konstantin
Next meeting: Thursday, May 1st 1-2:30; Administrative Conference Room Mugar

Action Items:.
Linda will renew Survey Monkey.
Brendan will create a google doc in case groups wish to post comparison info before meetings.

Dan will transfer the prior BU faculty survey from Survey Monkey to Qualtrics (2 copies).

All can begin looking at other survey instruments (UW, MIT, Ithaka) for helpful comparisons.

Announcements:
- Dan announced that Feb 27th marks the 1st full day of external link tracking through Google Analytics (GA).  One can view stats on internal and external links for any page on our site, though those numbers exist in two separate places within GA.
- Linda announced that she and Tom C., Steve, Stephanie, and possibly David and Brendan will be attending a session at the Hyatt Regency on Academic Analytics.  The session will specifically be aimed at librarians.

OCLC Assessment event update
 
Linda will serve on a panel at an OCLC meeting at Brandeis on April 22nd called "Getting the Right Fit: Tailoring Assessment Strategies for your Library."  More info is available here.

MINES
update
MINES is proceeding very smoothly now.  We have had a 0% drop out rate.  We need to think about how to back up the data periodically.  David believes we can download the data at any time, and we agreed that perhaps a schedule should be followed to download and backup the data periodically, although IS&T seems to trust the cloud.

Faculty Survey planning
We began planning for a faculty survey, not in exact order of the agenda, but the info is here below.
Tasks
i.    
Game plan- general, but not unanimous, consensus to perform surveys individually instead of grouped. We also decided to renew Survey Monkey (Linda will do this) once more as the deadline is fast approaching and it may come in handy
ii.   
Developing the survey - will follow these steps
1. 
comparing to other surveys - Small groups will do initial comparison work and then bring suggestions to an AC meeting for all to discuss.  Brendan will create a google doc in case groups wish to post something before meetings.
a.  Dan will transfer the prior BU faculty survey from Survey Monkey to Qualtrics and look for any differences (particularly in skip logic).  Dan will create 2 copies on Qualtrics - one to edit and one for posterity.
b. Linda and Konstantin will then look over comparisons from our own grad and undergrad surveys
c. Next, Lisa and Brendan will look at comparisons to the MIT survey
d. Lastly, Russ and possibly others will compare to UW and Ithaka instruments
2. final pass thoughts - examine possibilities of skip logic, questions or other items to eliminate to keep the survey short, and look through our survey keepers list
iii.     Timeline - We're thinking about Spring 2015.  Russ expressed that we should be mindful of the timing, as Law faculty will be relocated during Spring 2015. Survey fatigue
iv.    
IRB - Linda will accomplish
v.     Publicity/Recruitment - poster from Anita Greene group and emails
vi.    
Launch - nothing noted
vii.    
Analysis, quantitative - nothing noted
viii.    
Analysis, qualitative - have no one currently for this. there is the possibility of going back to tag previous surveys.
ix.    
Report writing - will likely rely on in-house talent.
x.    
Staff involvement in changes - we need to examine this more.

__________________________________________________________________________________________________________________________

Assessment Meeting Minutes for 01/29/14

 

Present: Linda (chair), Brendan (minutes), Russell, Lisa, David, Steve, Konstantin, Sarah, Dan

Next meeting: Friday, Feb. 14th (Valentine's Day) 2-3:30; Administrative Conference room Mugar


Action Items:
- Notify Tim that Pubmed (and other resources?) should be placed behind EZproxy
- LINDA: Call Amy about list being proxied. (DONE 2/3/14- LP)


Announcements:

- Russell will be joining us henceforth as a new member of the Assessment Committee. Welcome!


Tuesday, 1/21 Phone Call w/ UMass Amherst’s Rachel Lewellen

Present were David, Steve, and Sarah and Linda.  Overall, a very positive, informative call.  UMA started out with a year long implementation and collected data from all users who started Ezproxy sessions during two, 2 hr. periods per month. In UMA's second year-long implementation of MINES, they switched to n = 140, and then reduced finally to n = 120. It’s important not to change n too far along so that data won’t have to be excluded, although Terry Plum implied that collection data can be adjusted if a change to N is needed during an implementation. Over 70% of surveyed users completed the survey at UMass (!!!), no complaints registered, not even about frequency. UMA did not have a comment box, but did have an email link. They only received one email. UMA included a question about why survey taker is using the resource.  Shared graphs are promising; of note the URL collection stats of those surveyed generally reflects the URL collection stats of the entire user-base (ie statistically relevant data).  Very promising overall!


Let’s apply this to our MINES! 

- We can use the URL data that will be gathered, but it will involve more involved analysis than other sources of data.

- Konstantin: “We should use ‘BMC Resident’ for primary status question”.  Everyone agrees with the tactful word choice.

- Google Chrome is being problematic for everyone with scrolling issues (only for most recent update).  It may interact with survey if someone tries to get to bottom of primary status list. “Maybe we can change the number of visible items on the dropdown list” suggests David.  Brilliance!  Later, “Alas, we cannot do that” informs David.  Foiled by technology again.  We shall investigate during test period whether Chrome will be problematic

- “High school student shouldn’t be at top of list. It’s not ordered by age (Brendan grumbles), so we should order it by the groups most likely to use it, ie put High school students below staff on the drop down list.  General consensus, Brendan dissents vehemently.   Too bad for him.

- David’s suggestion to get the sponsored question to drop down works.  An excellent solution to the problem we were having.

- “ask@bu.edu” to be changed to “assess-l@bu.edu” so assessment committee members can personally field survey questions.

- On criteria for assigning survey-testing: “Perfect for those who have miffed you!”
- Linda: “We can’t go live until we’re sure the Med campus and Pubmed work.  We need to get Pubmed behind EZproxy.  And guide writers will need to update PubMed links.  Also many Law resources won’t be EZproxied.

- Thoughts on n:

   * Larger student base and greater usage means n can be larger (or smaller? No, larger… we think).

   * We can change n within the first month if our number doesn’t suffice.

   * Linda: “Let’s go around and see what everyone thinks is reasonable.” Crickets

   * After some deliberation, Brendan: “100 or whatever is reasonable”.

   * Number-crunching Steve: “100 gets us just enough data with 70% expectation rate.”

 

MINES Testing

Directions will be on the site.  Mugar to have two demo sessions on Friday and Monday, one in the morning, and one in the evening.  Everyone else to schedule demos for the other branches. Remember, we want to test on as many platforms as possible, even mobile ones.  Also we need to test for blank survey results.  Proposal: Go live on 11th @ 12:00? … eh, maybe not.  We’ll see.

 

Miscelany

-       Steve, on n: “Yes means no, and no means yes”

-       Russ, on difference between Law Library and other libraries: “We actually provide service” (excellent smack talk, if I may say so)

 

NEXT MEETING?

-       Linda: “How does Valentines day work for everyone?”  All concur. There will probably be Valentine’s goodies, and Brendan will probably bring something (he didn’t mention this though). 

Meeting adjourned.

_________________________________________________________________________________________________________________________
Assessment Meeting Minutes for 01/16/14

Present: Linda Plunket (chair), Dan Benedetti (minutes), Russell Sweet, Brendan DeRoo, Lisa Philpotts, David Fristrom, Tim Lewontin
Next meeting: Wednesday, Jan. 29th 2-3:30; Administrative Conference room Mugar

Action Items:.
Linda will work out what staff should test which databases (refnews/alphabetical?)
Linda will change wording on MINES survey as outlined in 'B' below
Sarah will be writing up the instructions for staff testing
Konstantin will answer this question: If asked to give their primary status, would faculty on the Med campus be confused between a 'faculty' option and an 'affiliate' option?
David will look at Qualtrics closely to see if we can easily get the URLs patrons will be forwarded to on completion of the MINES survey.
All who can should attend a call with Rachel Lewellen, Assessment Librarian from Umass Amherst, Tuesday (01.21, 10am)

Announcements:
- Linda reported the library wide meeting was a good success.
- We've been invited to an undergrad faculty council, and a presentation will be given by Tom C. and David.
- Brendan was asked by Dan to help with the Google Analytics initiative.
- Today's meeting will be about MINES, so read this data visualization article in preparation for the next meeting:
Azzam, T., Evergreen, S., Germuth, A.A., & Kistler, S.J. (2013). Data visualization and evaluation. In T. Azzam & S. Evergreen (Eds.), Data Visualization, part 1. New Directions for Evaluation, 139, 7-32.

MINES Initiative updates
Nearly ready to implement, but need to set up staff testing on a test website.  We will be looking for any minor issues, probably with 'refnews' staff, with Linda working out which staff should test what databases.  Sarah will be writing up the instructions for staff testing, and all should review them.  Russ also notes we should start the surveying early enough before finals, or alternatively wait until early June.
A) Some institutions are putting open access databases behind ezproxy in order to track through MINES.  David reminded us that many patrons go straight thru google to resources, so we won't be able to get them.  Tim pointed out that we don't send pubmed thru ezproxy because of the way their website used to work, but we can now.  Because of possible complications with redirects, we should carefully test the performance of very important databases like pubmed. 
B) We worked on the wording of the MINES questions.  Terry Plum had two suggestions on wording: 1-that we don't need an 'other' on q2.CORRECTION BY LINDA: TERRY'S COMMENT WAS NOT THAT WE DIDN'T NEED IT BUT ASKED IF RESPONDENTS WOULD KNOW THAT OTHER REFERRED TO OTHER-AT-BU.  2- on the 'purpose' question, to include 'research unfunded (faculty)', because it is primarily faculty that do research and she wanted to distinguish this from scholarship.   We decided to change this to "unfunded research and scholarship" and not include '(faculty)' or '(students)' in answer options.  We also decided to add an answer option for 'library staff'.  Brendan caught an incorrect plural of "affiliates'.  Finally, a question for Konstantin came up: If asked to give their primary status, would faculty on the Med campus be confused between a 'faculty' option and an 'affiliate' option?
C) Two places are trying to get answers on what resources patrons are using, thru MINES: Umass Amherst and Univ of Toronto.  There will be a call with Rachel Lewellen, Assessment Librarian from Umass Tuesday (01.21, 10am) that all who can should attend.  Umass is also increasing the number of times the survey will be filled out, by having the survey pop up more often.  We decided that we should also track resources, and David will look at Qualtrics closely to see if we can get this easily, as it seems should be possible.

Comments