Evaluating Collaboratory Cultures
Evaluating Collaboratory Cultures
Evidence-based Tools & Practices for Interdisciplinary Research in the Collaboratory
Evidence-based Tools & Practices for Interdisciplinary Research in the Collaboratory
Embedded Files
Project Summary
Project Summary
The purpose of the ECC project is to explore and test the collaboratory as an interdisciplinary infrastructure. We will study the landscape and drivers of collaboration on campus, and using the School of Data Science Collaboratories as a testbed, we will develop an evidence-based framework of methods, metrics, best practices, and institutional policy recommendations that advance the collaboratory and related concepts, such as interdisciplinarity, convergence research, and open science. This transformative project will introduce concepts, tools, standards, and metrics that we anticipate will 1) facilitate transparency, which will help sponsors and other stakeholders across the research ecosystem identify needs and assess effectiveness within the collaborative process; 2) improve communication between cross-functional teams; 3) broaden participation in science with tools and methods that improve access to and visibility of collaborative opportunities; and 4) deepen the impact of scientific investment by optimizing the Collaboratory Cultures framework as a standardized approach to open science research.
Background
Background
The disciplinary silos in Higher Education were organized around defined fields of study in order to facilitate student instruction—law students receive their education in the Law College, medical students at the School of Medicine, etc. But these traditional arrangements are not appropriate for 21st century science, which is organized around research relationships in order to optimize diversity and accelerate innovation. The collaboratory structure characterized by the School of Data Science Collaboratories strikes a balance between UVA’s pride in tradition and its Great and Good strategic plan to celebrate its tricentennial as the world’s best public university.
But does the collaboratory provide a measurable improvement to traditional research infrastrucures? How do collaboratory practices facilitate future trends in science, such as convergence research with international teams and cross-functional capabilities? How can UVA students, faculty, trustees, and administration use insights into their collaboratory functioning to improve decisions? Looking forward to 2030, how can UVA leverage its unique advantage in collaboratory expertise toward becoming the nation’s premier Collaboratory Research Center with training, certification, and services for industry, academia, civil society, and government?
The topic of collaboration as a research subject has been well-studied in most fields, and some metadata about collaboratories is well-documented (e.g., h-index, bibliometrics, award amounts, recipient demographics, etc.); however, metrics and reference data that can be used to accurately assess the functioning of the collaboratory unit have yet to be established and modeled. Indeed, there is not a single published dataset or evidence-based best practice to inform the design and/or operation of the research collaboratory. In short, the collaboratory as a subject of scientific research remains unexamined. This systemic lack of performance metrics results in persistent shortcomings that pervade every level of decision-making—from program selection by graduate school applicants to strategic planning within the Office of the President.
Our interdisciplinary project will explore two questions: 1) How can we scientifically assess collaborative research practices and apply those findings to a continuous improvement process? and 2) How does our framework improve the practice of research more generally? To better understand the state of collaborative practices, including gaps and opportunities, we will employ a mixed-methods approach to 1) gather survey data from cross-functional research team members; 2) map the landscape of historic and current collaboratory practices through interview data and literature review; and 3) conduct reflexive assessments to establish baseline data for an evidence-based collaboratory model. We will test our reflexive methodology on the inaugural cohort of the SDS faculty, staff, students, and professionals. Our goal is to create an evidence-based model for collaboratory research that is robust enough to produce metrics across disciplines and valances.
Intellectual Merit
Intellectual Merit
Our study introduces a transformative approach to research practice by developing specific tools, identifying proven best practices, and providing metrics that accelerate discovery through highly effective open science collaboration. Our research 1) makes the first significant contribution to the literature on evidence-based models in collaboratory research; 2) builds on the handful of qualitative studies and recommendations for fostering a collaborative culture in convergence research; 3) responds to the call from stakeholders to produce evidence-based approaches that advance the practice of research; 4) advances the body of scholarship that addresses general collaboration issues; and 5) develops novel tools, methods, and metrics for evidence-based collaborative research.
Broader Impacts
Broader Impacts
Our study is designed to advance the needs and goals of 1) underrepresented individuals and institutions in the research ecosystem by developing a new framework for inclusive participation within the collaboratory, as well as an evidence-based rubric for assembling collaboratory teams; 2) STEM policymakers and research sponsors through recommendations that detail the modes of support that are required to optimize convergence research outcomes; 3) STEM researchers across sectors by providing methods and metrics to improve convergence research practices and outcomes; 4) the larger STEM community, including research administration professionals, research clients, research sponsors, students, professional societies, and oversight bodies by establishing ethical and responsible research; and 5) the global data science community by illustrating the value of the collaboratory model.
Expected Outcomes
Expected Outcomes
Regardless of scientific findings, the research process will deliver significant contributions to interdisciplinarity at UVA as we organize existing information and resources, explore probing questions that elicit new insights into institutional capacity. We will deliver tooling and training, and we will be able to provide clarity on a few persistent challenges, such as:
Finding potential collaborators across campus, and initiating interdisciplinary project work
Resolving funding & allocation conflicts across departments
Managing facilities & resources for multi-department collaborations
Building public-private partnerships with resource-limited interdisciplinary capacity
Identifying opportunities & preparing proposals that translate across disciplines
Raising awareness of campus resources
Facilitating interdisciplinary work within the faculty community
Effective gatekeeping, oversight, and guidance for research in a collaboratory setting
Workshop facilitation fosters UVA's collaboratory culture by responding to the community of practice (internalization) with new collaboratory infrastructure (institutionalization).
Courtesy Melson (2022)Policies, Protocols & Commitments
Policies, Protocols & Commitments
Our reflexive research project models OS practices and our methods are led by FAIR and TRUST principles. We follow the guidance outlined in the Beijing Declaration on Research Data and our ethical and responsible research commitments are aligned with JEDI+A and CARE Principles for Indigenous Data Governance. Our protocols are registered with OSF, ARTiFACTS, and GitHub. The point of contact for policies and protocols is Merc Fox. Confidential concerns may be directed to the Advisory Board and submitted anonymously.
Our data, process, and findings are curated to be Findable, Accessible, Interoperable, and Re-usable whenever possible. FAIR artifacts generally require time for processing, review, and posting to the project website; however, we are obligated to share our project's knowledge, tools, and resources with the research community to the extent possible and reasonable.Findable material is Accessible on the website. Our methods model Interoperable good practices in research, and our products are delivered on Open Access (OA) platforms for Re-use.The CARE Principles for Indigenous Data Governance assert human-in-the-loop intentionality in sociotechnical systems by centering the role of the individual or community within the research enterprise.Collective Benefit: Our project will extract knowledge from the UVA community of practice (COP), and then return it in the form of collaboratory frameworks that meet the stated needs and interests of the COP. Authority to Control: We will develop metrics that enable the researchers to determine how their work, their effort, their curiosity, and their outputs should be defined and valued.Responsibility: We apply rigorous OA approaches and active, ongoing engagement with campus stakeholders to share our resources and learnings, and to provide complementary support that promotes the interest and well-being of the COP.Ethics: In addition to applying JEDI+A, FAIR, TRUST, and data sharing best practices, we treat research ethics as a management tool that promotes well-being across the COP, and prevents fraud and misuse, which is frequently unintentional--particularly when multiple disciplinary customs come together in the collaboratory.TRUST Principles provide guidance for digital knowledge stewardship, and reinforce other best practices in data science, information science, AI systems, and sociotechnical research. We support Transparency, Responsibility, User focus, Sustainability, and Technology through the curation of our project journal.
OS
Open Science (OS) practices range from sharing published findings after an embargo period to engaged research on a fully open platform, with the former providing more protection to intellectual property and quality control, and the latter offering more access and input. Our project delivers transparency whenever possible, and asserts quality control through registered protocols, which prevent changes to tools, methods, and commitments.FAIR
Our data, process, and findings are curated to be Findable, Accessible, Interoperable, and Re-usable whenever possible. FAIR artifacts generally require time for processing, review, and posting to the project website; however, we are obligated to share our project's knowledge, tools, and resources with the research community to the extent possible and reasonable.Findable material is Accessible on the website. Our methods model Interoperable good practices in research, and our products are delivered on Open Access (OA) platforms for Re-use.The CARE Principles for Indigenous Data Governance assert human-in-the-loop intentionality in sociotechnical systems by centering the role of the individual or community within the research enterprise.Collective Benefit: Our project will extract knowledge from the UVA community of practice (COP), and then return it in the form of collaboratory frameworks that meet the stated needs and interests of the COP. Authority to Control: We will develop metrics that enable the researchers to determine how their work, their effort, their curiosity, and their outputs should be defined and valued.Responsibility: We apply rigorous OA approaches and active, ongoing engagement with campus stakeholders to share our resources and learnings, and to provide complementary support that promotes the interest and well-being of the COP.Ethics: In addition to applying JEDI+A, FAIR, TRUST, and data sharing best practices, we treat research ethics as a management tool that promotes well-being across the COP, and prevents fraud and misuse, which is frequently unintentional--particularly when multiple disciplinary customs come together in the collaboratory.TRUST Principles provide guidance for digital knowledge stewardship, and reinforce other best practices in data science, information science, AI systems, and sociotechnical research. We support Transparency, Responsibility, User focus, Sustainability, and Technology through the curation of our project journal.FAIR
JEDI+A
Justice: We express our commitments to Justice, Equality, Diversity, and Accessibility by developing and promoting the collaboratory as a specialized research infrastructure that embraces partnerships, ideas, and methods that differ and even diverge from the status quo. Equity: We observe equity by delivering opportunity, knowledge, trust, and authority to our staff and Collabolier partners based not on the individual's professional credential or status, but on their direct contribution to our work, as well as the potential downstream impact that work might have on the individual and/or community.Inclusion: We adopt Articles 2-4 of the SDS Inclusive Excellence Plan (Articles 1 and 5 extend beyond the scope and capacity of our project to enable). Article 2: Continuously promote and strengthen an inclusive community of trust, a culture of integrity, mutual respect, excellence, collaboration, and innovation. Article 3: Enable faculty, staff, and students to work across traditional boundaries and prepare servant-leaders to shed new light on enduring and profound questions in our diverse community and globally connected world. Article 4: Be a community that consistently lives its values and ensures that our systems enable our students, faculty, and staff to do their best work.DMP
The project's data management plan can be accessed at GitHub and by request.IRB
Our project was determined Exempt from IRB oversight due to sponsor requirements and low risk to research participants.Project Team
Project Team
Research Assistants
Research Assistants
Rui Chen
Rui Chen is currently pursuing a B.A. in Statistics at the University of Virginia, specializing in Finance and Business. With internships ranging from a Research Assistant at UVA's School of Data Science, credit analyst at Bank of East Aisa, to a Financial Consultant collaborating with Ernst & Young, Chen has demonstrated exceptional skills in data management, financial analysis, and strategic communication. Leveraging technical proficiency in R, Tableau, and Robotic Process Automation.
Ymanee Monestine
Ymanee is undergraduate student at the University of Virginia, pursuing a Bachelor’s degree in Cognitive Science with a concentration in Psychology. Her professional goal is to become a clinical psychologist in the field of acute trauma, with a particular focus in religion and its effects, both positive and negative. She hopes to pursue a research and clinical career in psychology featuring independent and collaborative cognitive science and data science research projects.
Page updated
Report abuse