Examining the Communication, Organization and Development Patterns of FOSS Projects
Andrew Swerdlow
University of Victoria
Dept. Computer Science
Victoria, BC, Canada
swerdlow@uvic.ca
ABSTRACT
This paper examines the common trends in Free Open Source Software (FOSS) development. We will provide an enumeration of practices that are common to successful FOSS projects. The intent of examining FOSS projects strategies for success is to help other organizations deal with the challenges similar to those that FOSS projects have over come.
INDEX OF TERMS
Distributed software engineering, FOSS, global software development, open source.
I. INTRODUCTION
The open software movement has become a major player in the software development industry, creating products that are highly reliable and competitive with their commercial counterparts. Although there has been some skepticism about the viability of FOSS, the reality of the situations is that open source development is building up momentum and has proven to be a good means of developing highly robust complex software systems. Some of the major success stories of the open source movement are Apache web server, Mozillia web browser and Linux operating system to name a few. Commercial organizations around the world are experimenting with ways to capitalize on the FOSS phenomenon. Some of the initiatives include offering cash bounties for software. As an example of how seriously industry is taking the FOSS movement, software vendor Computer Associates (CA) offered a one million dollar reward for the development of a data base converter [1]. They later paid out cash awards to several groups that submitted solutions. CA is just one of many companies that believe that FOSS development will give them a commercial advantage in the market place. Industry has been supporting FOSS by providing FOSS groups with infrastructure resources as well as skilled paid programmers. An example is the relationship between AOL-Time Warner and Mozillia. AOL-Time Warner is a commercial organization that develops web technologies for profit including Netscape. In 1998 AOL-Time Warner released their source code for Netscape communicator into general public. The source code was then subsequently used as the foundation for the Mozillia web browser, a FOSS initiative. The Mozillia team is made up of non-paid volunteers as well as paid employees from Netscape [2]. It seems that some FOSS projects have been able to overcome some of the problems related to communication, organization and coordination in distributed settings. The goal of this paper is to examine the strategies in 7 different FOSS projects, and enumerate which practices they have in common. By identifying these practices, it is our hope that some of the strategies could be examined for their potential usefulness in other projects that face comparable constraints as the FOSS projects we examined.
This paper will be laid out in the flowing manner. The rest of this section will identify our research question. Section 2 will provide a survey of the relevant literature as well as examine seven specific open source projects: Apache, Mozilla, Postgres, Eclipse, Debian, Helix and Openoffice. Section 3 will focus on discussing the data collected on 7 FOSS projects. Section 4 will be a discussion on the results of our research. We offer our conclusions in section 5.
B. Research questions
The goal of this paper is to identify common practices in successful FOSS projects. In specific we are interested in how successful projects communicate, deal with geographical distribution, organize themselves and their development processes. We will base our study on the following research questions:
Q1. What communication practices are common to successful FOSS projects?
Some FOSS projects can be very complex systems that have many developers working together on the same projects. An example of this is Debian Linux. The source code for Debian 3.1 had over 213 Million lines of code in the project. They also have over 800 registered developers One of the practices we wanted to examine in FOSS was how they were able to communicate despite many of the classic problems in distributed software development mentioned in [3], such as budgetary concerns and time zone restrictions.
Q2. What practices do FOSS projects use to mitigate the negative effects of working in highly geographically distribute teams?
Studies of commercial distributed software development have shown that separation in space and time can negatively impact the productivity of a given project [3]. We wanted to understand how FOSS projects overcome the limitations of working in a highly distributed work environment.
Q3. What organizational strategies are common to FOSS projects?
One of the defining elements of FOSS projects is how they are organized. This question is asked so that we can find trends in the way that FOSS projects organize themselves. Once we have enumerated the common practices it is our hope to determine if these practices are the positive contributors to the projects success.
Q4. What practices do FOSS projects use to elicited and refined requirements?
For most large scale commercial projects to be successful, requirements are the foundation of the development process. Case studies indicate that proper requirements processes are beneficial to distributed software development projects, as demonstrated in [4] for example. It is common in industry to follow the waterfall pattern of software development. There are six different phases of the lifecycle that occur in the following order: Requirements, Design, Implementation, Validation, Release and Maintenance. The formal requirements process usually involves actively eliciting requirements from stakeholders followed by validating them before implementation. Projects then develop their system on those requirements. In FOSS project requirements seem to follow a different process. They do not pear to follow the waterfall approach. We would like to observe if there is a distinct development pattern common to FOSS projects.
II. Methodology and Data Sources
A. Literature Review
When we consider the research question we have proposed, it is important to take into account what motivates developers to join FOSS projects. Over the last decade there have been many noteworthy studies on FOSS projects trying to understand the motivations of FOSS developers [5,6]. In general the motivations of developers that volunteer in FOSS projects can be characterized into two groups, intrinsic and extrinsic [5]. Intrinsic motivations relate to deriving pleasure form the act of participating in the projects. The extrinsic motivations relate to receiving some indirect reward for their work, such as employment considerations. We mention this because it is important to understand that FOSS communities are diverse organizations that face many communication and organizational challenges. Some examples of challenges are minimal budgets for face-to-face meetings, maintaining volunteer status [7], and preserving quality assurance. It is how FOSS projects overcome these challenges that provide us with opportunities for research.
The organizational structure of FOSS projects have been characterized into two basic types described in [1,8,9]. Commonly smaller projects have a core group that does much of the work and decision making. Some of the major contributors in the core groups might be funded by commercial organizations. There is also usually a much larger group of volunteers that help develop the projects by providing patches and testing for bugs.
The second type of project Herbsleb and Mockus describe in [2] are larger projects. They hypothesize that larger projects usually separate the work into smaller groups, “creating in effect, several related OSS projects”. This breaks the projects down in to modules, which have individual ownership demonstrated in [2,10]. In the smaller situations there seems to be more awareness of the project as a whole that is the core developers are aware of many different components of the project and they commonly work of different parts of the project [2]. For extremely large projects such as the Debian linux project described in [10] it would be extremely difficult to have knowledge of a large percentage of the project due to the vast size and diversity of the code base.
Many FOSS projects have to deal with extreme geographic dispersion, complex tasks, and minimal funding for face-to-face meetings. They also have to do this with a limited budget for new tools and equipment. There has been some research on providing cognitive support for awareness in FOSS projects suggested by [11,12]. Although these tools seem to provide some useful functionality to FOSS projects it has been suggested that FOSS projects already provide sufficient collaborative support through the tools they already use. The authors of [13] suggest that awareness is indeed maintained in FOSS projects and that the tools that they use provide a high level of support. In [13] they state that awareness information is implicit in FOSS projects due to their open nature and the openness of the tools they use. FOSS projects use many tools that encourage a large group to be aware of dissensions and actions that are taken. Some of the examples give in the paper are email lists and IRC channels, conversations are broadcasted to many community members not just the intended participants in the conversation. This allows others to become aware of the actions of others. So although FOSS projects do not have any explicit mechanism for providing awareness they seem to opt for general purpose tools that implicitly provide awareness information. In this paper we will examine FOSS projects that are indeed successful and examine how they achieve their success using the limited recourses they have.
B. Project Websites
This section of the paper will describe the mythology we followed for collecting our data. We will also present some of the data we gathered.
We followed a simple process for gathering data that is easily recreatable for researchers that wish to use the technique for analyzing other FOSS projects. For the purposes of collecting data for this paper we visited 7 different Open Source Software project websites: Apache, Mozilla, PostgresSQL, Eclipse, Debian, Helix and Openoffice. We selected the 7 projects in a random fashion; the only factor we used in the selection was the success of the projects. We acknowledge that other less successful projects might also have followed the same processes with less success; however, for this study we have not examined unsuccessful projects. We also note that many of the projects we examined have commercial roots see figure 1. This is important to consider because a commercial project might structure its code base to support a specific model of development. This model may or may not be supportable in FOSS environments. We preformed exhaustive reviews of the websites, and collected data pertaining to:
•Communication Model and Tools,
•Organizational Structure,
•Development Processes,
•Number of Developers and
•Age of the Project.
The data was entered into tables indexed by project. Other information sources such as literature reviews described in the previous section and email correspondence was incorporated in the tables for information that could not be found on the project websites.
Figure 1: Table describing the funding status of the projects.
C. Email Correspondents
Email was sent out to the mailing lists maintained by the members of each project. The email mailing list contact information was found on the projects web sites. Five of the seven projects contacted responded with email. The quickest response was within 12 hours of emailing, and the longest was 2 weeks. The emails were addressed to the general project contact emails listed on the projects website. The emails contained 5 questions about the specific projects. The questions asked about the following items:
•What is the number of developers?
•How to contribute?
•What is the organization structure?
•What is the requirements process?
•What makes your project successful?
The results of the email correspondence were put in to tables indexed by project and questions.
III. Project Comparison
Once we had collected our data we then went on to analyze the collected information, with the intent to enumerate trends in the data patterns. We first started off by examining the communication patterns of the FOSS projects.
A. Communication Patterns
In this section we will try to answer Q1 and Q2, by examining each of the 7 FOSS projects communication models. At this time we would also like to note that FOSS projects are commonly an example of Global Software Development (GSD) projects, where stakeholders from many different countries work together on one project. In commercial GSD projects inadequate communication frequently leads to delay, exceeded budgets and project failures as noted in [14]. Much research has been focused on creating tool support for improved communications in GSD situations. However there has been no specific tool that has met all the needs of the intended users. Many of the tools that have been proposed for commercial GSD projects have a high level of complexity some are discussed in [15].
An examination of the FOSS project websites revealed that the projects seemed to use a combination of unsophisticated communication tools. The tools were usually free general purpose asynchronous text based tools. This seemed completely opposite of tools designed for GSD. All the projects we examined used a similar tool set:
•Configuration Management (CVS or SVN)
•Mailing Lists
•Bug tracking Database
•Project website
•Chat room
All of the tools are simple text based web technologies; they are free and accessible from many different operating system platforms. Most of the tools are asynchronous.
Figure 2: Table describing the distributed communication tools functionality.
As you can see in Figure 2, the common tools of FOSS projects support a distributed asynchronous multi-point communication model. Where multi-point refers to the intended audience of the communication. When examining these different communication tools we wanted to characterize their combined collaborative support. So we have decided to use a Workstyle model described by Wu in [16] to visualize the tools collaborative potential. We modified Wu’s model to illustrate tools support working styles in six dimensional space. The collaborative functionality from the combination of tools can be plotted in the space. This will indicate the set of work styles that can be supported. This model will allow us to visualize the different possible working styles provided by the FOSS tool sets.
Figure 3: Work style model for FOSS communication tools, based on [16].
When reviewing the tool set of successful FOSS projects it would seem at first that the tool sets are limited and could only provide basic support for working in a distributed environment. However, when looking at a combination of all the tools that for the FOSS workspace, it would seem that they provide a rich range of working styles.
B. Successful Organizational Practices
The way FOSS projects organize themselves is also of interest, because it appears to de dramatically different than commercial projects. In this paper we will summarize the results of our observations of projects documentation and email correspondence. Our intent is to create an enumeration of the organizational practices of FOSS projects. The final goal will be to isolate the specific practices that positively impact the development of large scale distributed software development projects. Of the seven projects we examined we found that the organization size ranged between 100-900 people participating in the development of the projects. On closer examination we found that there is a smaller subset of the developers that will do a majority of the work, and the others were occasional contributors. This supports conclusions in [2,8,9]. The participants seem to be organized into a community structure rather that a corporate hierarchy. Some of properties of FOSS communities are mentioned by O’Mahony and Ferro in [10]. The communities are groups of distributed individuals who voluntarily contribute work on a specific project. The work is usually focused on a common interest or need. There are usually democratic processes with in the community about making decisions. Strangely enough our data indicated that less than half of the projects we surveyed had elected leaders. Manny of the projects had their leads appointed.
The project websites for all the FOSS projects provided information about the conduct of those in the communities. Some of the conduct rules pertained to membership, requirements, and development procedures. Almost all of the projects would allow new users to submit code in the form of a patch to the developer in charge of the area of code. Once a new developer has proved that they can submit quality code they are usually given more rights and access to the source code. Every respondent in our email conversations indicated that the community structure of their organization played an important role in the development of their projects. The following is a statement provided by one of the members of one of the projects we examined:
“A community process is more democratic in nature. This leads to a more transparent setup where future innovations .... can be quickly and efficiently developed”
The communities are generally a flat structure with some leadership roles. The majority of projects that we observed there was some project leaderships in the form of project managers (PM) and committees. The PM roles appeared to be more like project administrator that provided guidance and direction to the projects, they also aid with conflict resolution. This is somewhat different than the traditional view of PM’s in commercial projects. Although PM’s in commercial projects commonly provide guidance and conflict resolution as well, they also do much more. They take on a supervisory role and are in charge of insuring productivity of their employees and managing their activities. Leadership was also provided in the form of committees. These committees once again seem to provide community administration rather than managerial support in a commercial sense. Here is another quote form another contributor from one of the surveyed projects:
“Each project has its own lead that approves changes to the code-base. Beyond that, we have a flat structure where everyone is a volunteer and can make individual contributions.”
It also appeared that developer’s specific roles were less defined than in commercial projects. Although roles are not always clearly defined in FOSS projects, that is not to say that there are no roles at all. One significant distinction between developers in FOSS projects are those that are paid and those that are not. It would seem that almost all FOSS projects that we examined have some paid developers. It should be noted that there are significantly fewer paid developers then those that are unpaid. Lakhani and Wolf in [5] indicate that the paid members of the communities spend 51% more time working on project related activities than non-paid volunteers.
Perhaps one of the most interesting facts discovered in our research was the large amount of commercial involvement in FOSS projects. In a majority of the cases the FOSS projects are supported from commercial organizations, some examples are: Mozillia and AOL-Time Warner, Open Office and Sun Microsystems, Helix and Real Networks, etc. These companies commonly provide support in the form of hardware infrastructure and personnel. An example is AOL gave Mozilla a two million dollar donation to launch the project, Sun provides paid employees to develop Openoffice etc. It is important to remember that a corporation’s main goal is to make a profit for its share holders. This means that the corporate motives are not purely altruistic. Some of the common motivational factors for corporations’ to support FOSS work are discussed in [17].
Although some of the FOSS projects we surveyed had fairly strong commercial ties it was apparent that their organizational philosophies were different than most commercial projects. One of the common attitudes we noticed was the idea of transparency. Most projects tried to make all their process transparent to the community. That is when decisions are made the community is informed of the rationales of the decisions, this allows community members to contribute in decision making. This transparency also aids in enforcing community memory.
All the above factors aid in defining FOSS projects from traditional commercial projects. It would seem that some of the common organizational strategies employed by FOSS projects are: strong sense of community, open decision making processes, volunteers, and some commercial support.
C. Development Processes
The development process for FOSS projects is also interesting to review. As we have stated it is our intention in this paper to find processes that are common to most successful FOSS projects and examine if they are factors that directly contribute to the success of the projects. It should be noted that each project we surveyed, we observed that they all have some what different development processes, however there are some observations that are worth mentioning. In general most projects follow an iterative development approach. We will provide a general characterization of the development process based on documents we found on the projects websites and email correspondence. In most cases the process starts with a proposal from a member of the community or an industry partner. The proposal is reviewed by some members of the community; the number of members is usually dependant on the scope of the proposal. Large projects that impact many components of the project will commonly be reviewed and validated by the larger community. Smaller projects that have little impact on other project components will usually be reviewed by smaller groups. Once the proposal is validated then the proposal will be implemented. At this point there is the potential for some iteration of the implementation and the validation process; this is usually to insure that the components that are added are of a high quality. Once the implementation is complete then the component is added to the main build and distributed in the next release. Once the build has been distributed to the larger community, the software will be used and tested by many people. It is common that this large group of users will help in the review process by finding any issues with the components.
Figure 3: Typical Life cycle for Open Source Projects.
We also had some interesting observations about how the FOSS projects develop and elicit their requirements. The medium for proposing and managing requirements is commonly email mailing lists and issue tracking systems. There also seems to be some common trends with the way that FOSS projects define which proposed requirements will be implemented. Two of the main factors that motivate the acceptance of requirements are: if the requirement satisfies the community interests, and it contributes to the organizations mission. The requirements are likely to be implemented if they provide a service to the community or if they perpetuate the ideologies of the community. In both cases it is common for the community to decide which requirements are accepted.
In General it was interesting to note that six of the seven projects followed an iterative pattern of development, where development iterates between implementation and review. It was also significant to observe that none of the seven projects we surveyed followed a classic waterfall approach to software development.
IV. Hypotheses
A. Questions
In this paper we examine seven different FOSS projects, where we framed our observations in the context of research questions answering inquiries on:
• Communication practices
• Dealing with geographically distribute teams
• Organizational strategies
• Developing requirements
We were able to identify trends in the data that lead us to hypothesize some answers to our research questions. It should be noted that we only examined a small subset of the amount of successful FOSS projects, so to validate our hypothesis they should be tested on a larger sample of FOSS projects and perform deeper studies on those projects. We will leave that for future work.
B. Hypothesis
It is our intent that the hypothesis’s we pose here stimulate future research on the successful practices of FOSS projects. By endeavoring to validate our hypotheses we can gain a greater understanding of FOSS projects and how they overcome problems related to communication, organization and coordination.
Hypothesis 1. There is a common set of tools that FOSS projects use, they are free, general purpose, text based tools that function in an asynchronous manor.
It was apparent by examining the online project documentation that all the projects used similar communication tools to maintain contact. Each project website provided a list of the recommended communication tools. The fact that the tools are free general purpose tools that are easily accessible to anyone fits the FOSS philosophy of equality and openness. There are many complex tools that provide collaborative support for distributed software development. However, all the FOSS projects examined chose a similar tool set. All the tools are basic, easy to use, tools that are highly robust and reliable. It seems that the basic tools fulfill the needs of the developers in a way the more complex tools do not.
Hypothesis 2. The combinations of tools defined in H1 used in FOSS projects are sufficient to help distributed project members mitigate the negative effects of separation in time and space, therefore effectively and efficiently collaborating on large complex projects.
We base this claim on the fact that combinations of tools used in FOSS provides developers with a large combination of working styles, and seems to provide a great deal of awareness to community members. We also basis this claim on the accomplishments of the projects we examined. It is easy to claim that by the Apache, Mozilla, Postgres, Eclipse, Debian, Helix and Openoffice are all successful projects that have a high level of complexity. It could also be reasonable to assume that the tools they used to work in their highly distributed environment were a factor in their success. However, it should be noted that some of the projects early development by commercial institutions might have been done in collocated settings.
Hypothesis 3. In most cases FOSS projects are organized in communities with predominantly flat structures, where decision making in done in an open and transparent manor.
The reasoning behind this hypothesis was determined mostly through email correspondence with members of the different communities, and review of community policies. It is likely that the reason leadership roles are less authoritarian in FOSS projects than commercial projects is due to the volunteer nature of FOSS projects.
Hypothesis 4. Most successful FOSS projects receive support from commercial organizations.
We base this hypothesis on the fact that almost every FOSS project we examined has some form of commercial support. We also noticed that there seems to be an emerging trend for paying developers prizes for developing solutions as mentioned in [1].
Hypothesis 5. Most FOSS projects follow an iterative development process, where requirements are accepted if they serve a community need or philosophy.
This is founded on reviewing the process of adding features to the FOSS projects and buy communicating with the developers of the projects.
V. Conclusion
This paper has enumerated some of the practices that have made FOSS projects successful. When identifying the successful trend we also wanted to consider if any of these practices could be useful in commercial situations. However we did not hypothesize on which practices would indeed be valuable to commercial projects due to the lack of data on commercial projects, instead we offer the results to the community so that they could review and consider implementing some of the itemized success factors. We will offer one comment to industry members that are considering implementing these practices. It would likely be easy for commercial projects to adopt some of these practices and could potentially yield very positive results.
We would like to point out that it is likely that not all the practices of FOSS projects can be implemented in commercial situations, but it is useful to understand the reasons why some practice would be appropriate and some would not. That said we believe that many commercial organization could learn form examining the practices of FOSS especially those in distributed software development situations. In specific the community approach to software development would seem to be a powerful technique of developing complex software systems. It would be advisable to continue to examine there practices in hopes to better understand how they achieve their success. One of the key factors that will likely help commercial projects is how FOSS projects explicitly try to maintain a community memory to insure continuity and awareness; this helps the organization maintain a set of knowledge and provides continuity when developers leave the community. In contrast commercial projects use documentation to maintain knowledge within the organization. Perhaps commercial projects would benefit from FOOS practices of maintaining community memory.
REFERENCES
1. Dunn, D., “CA Offers Cash For Database Converters”. InformationWeek, Aug. 3, 2004 Available: http://informationweek.com/shared/-printableArticle.jhtml?articleID=26805678
2. Mockus, A., Fielding, R., Herbsleb, J., “Two case studies of open source software development: Apache and Mozilla”. ACM Transactions on Software Engineering and Methodology 11, (3), 1–38, 2002.
3. Herbsleb, J., Moitra D. “Guest Editor’s Introduction: Global Software Development” in IEEE Software, vol 8 (2), pp. 16-20, 2001.
4. Prickladnicki, R., Audy, J., Evaristo, R,. “Requirements Management in Global Software Development: Preliminary Findings from a Case Study in a SW-CMM context”. Proceedings of International Workshop on Global Software Development 2003, 6p, 2003.
5. Lakhani, K., Wolf, R,. “Why Hackers Do What They Do: Understanding Motivation Effort in Free/Open Source Software Projects”. MIT Sloan School of Management: 28, 2003.
6. Herrman, S., Hertel, G., Niedner, S., “Motivation of software developers in open source projects: an Internet-based survey of contributors to the Linux kernel”. Research Policy (special issue on open source software development), 2003.
7. Michlmayr, M., “Managing Volunteer Activity in Free Software Projects”. Proceedings of the 2004 USENIX Annual Technical Conference, Freenix Track. 93–102, 2004.
8. Mockus, A., Fielding, R., Herbsleb, J., “A Case Study of Open Source Software Development: The Apache Server” Proceedings of ICSE2000, 2000
9. German, D. “The GNOME project: a case study of open source, global software development”. Journal of Software Process: Improvement and Practice, pages 201–215, Aug. 2004.
10.O'Mahony, S., Fabrizio F., "Hacking Alone? The Effects of Online and Offline Participation on Open Source Community Leadership." Organization Science (under review).
11.Ankolekar, A., Herbsleb, J., Sycara, K.. “Addressing Challenges to Open Source Collaboration With the Semantic Web”. Proceedings of Taking Stock of the Bazaar: The 3rd Workshop on Open Source Software Engineering, the 25th International Conference on Software Engineering (ICSE), Portland OR, USA, pp 9-13. May 3-10 2003
12.Cubranic, D., Murphy, G,. “Hipikat Recommending pertinent software development artifacts”. In ICSE 2003 [13], pages 408–418, 2003.
13.Gutwin, C., Penner, R., Schneider, K. “Group Awareness in Distributed Software Development”. In Proc. of 2004 ACM Conference on Computer-Supported Cooperative Work. ACM Press, 72-81, 2004,.
14.Herbsleb, J.D., Mockus, A., Finholt, T.A., Grinter, R.E. “An Empirical Study of Global Software Development: Distance and Speed. In International Conference on Software Engineering”. Toronto, Canada, May 15-18, pp. 81-90, 2001.
15.Storey, M., Cubranic, D., German, D., “On the use of visualization to support awareness of human activities in software development: a survey and a framework”. SOFTVIS 193-2029, 2005.
16.Wu. J, "Tools for Collaborative Software Design", In Queen's University School of Computing Technical Report List (part W), 2003.
17. John Koenig, Seven open source business strategies for competitive advantage, May 14, 2004. Available:
http://management.itmanagersjournal.com