Experiments in SE: Promises & Perils

Dr. Natalia Juristo is full professor of software engineering with the Computing School at the Technical University of Madrid (UPM) since 1997 and holds a FiDiPro (Finland Distinguish Professor) research grant since 2013.

She was the Director of the UPM MSc in Software Engineering from 1992 to 2002 and the coordinator of the Erasmus Mundus European Master on SE (whith the participation of the University of Bolzano, the University of Kaiserslautern and the University of Blekinge) from 2007 to 2012.

Natalia will be General Chair for ICSE 2021 that will be held in Madrid. She has served in several Program Committees ICSE, RE, REFSQ, ESEM, ISESE and others. She has been Program Chair for EASE13, ISESE04 and SEKE97 and General Chair for ESEM07, SNPD02 and SEKE01.

She has been member of several Editorial Boards, including Transactions on SE, Journal of Empirial Software Engineering and Software magazine among others. Natalia has been Guest Editor of special issues in several journals, including Journal of Empirial Software Engineering, IEEE Software, Journal of Software and Systems, Data and Knowledge Engineering and the International Journal of Software Engineering and Knowledge Engineering.


This talk starts brushing up some basics on experiments: which are their hallmarks and strengths, and what makes them different from other types of empirical studies.

Then I will go over some of the main challenges that experimentation is facing today in SE:

  • Methodological shortcomings. Several of the experiments being conducted (and published) today suffer weaknesses that need to be addressed in order to get reliable findings. Experimentation is a quite novel approach for the big majority of SE researchers so it should not be a surprise that our experiments are a bit weak still. But rise awareness on this is critical to improve.
  • Lack of replications and families of experiments. Still most of the experiments run are just one shot. However isolated experiments are not the mean to generate mature findings. Replicate the studies in order to increase results reliability and grouping them in families to allow the reveal of moderator variables is the way to go if we really want to make SE an experimental discipline
  • Industry experiments. Laboratory experiments are just the beginning, field experiments (clinical trials in medicine) increase findings external validity. Unfortunately, we advance slow towards such end. Software industry lacks several of the motivations that medical doctors have to conduct clinical experiments. Is it feasible to follow such experimental model?

Multimodel Software Process Improvement


Ana Regina Cavalcanti da Rocha is full professor on Software Engineering in the Systems Engineering and Computer Science Program at COPPE – Federal University of Rio de Janeiro-Brazil. She has a MSc and a PhD on Computer Science, both from the Pontifícia Universidade Católica do Rio de Janeiro. Her teaching experience and research has been mainly focused on the following topics: software quality, software process, development environments and knowledge management. She was recognized as an outstanding contributor to the QUATIC conference series, and in particular to the QUATIC'2010 edition.

She was the technical leader of the team responsible for the definition of MR-MPS-Software (The Brazilian Reference Model for Software Process Improvement). Now she is involved in the definition of a new model that aims at the evaluation of software products.


Software organizations need to implement several different models and standards to improve their processes due to their specific characteristics or contractual requirements. No model or standard offers a complete solution for software process improvement. Which model should an organization select? How to implement more than one model or standard together? This talk will discuss these problems and will present an approach to process improvement based on several models and the organization´s most critical problems. Finally we will present a new model developed for software product evaluation that integrates four dimensions: organizational, software engineering, service and product quality characteristics and is based on multi model process improvement.

OutSystems Quality Made In Portugal


Lúcio Ferrão is co-founder and Principal Software Engineer at OutSystems since 2001. He holds a Software Engineer degree from Instituto Superior Técnico (1992-1997)

He loves to code, reduce complexity, supervise MSc thesis, and bridge the gap between industry and academia.

Lúcio is deeply involved in a long running research partnership between OutSystems and FCT NOVA focused on security concepts, process modeling, clearer application interface definitions, and language design.


With the goal of producing application software with minimal coding, including a wider range of people that do not have formal programming experience, Low-code development platforms are being presented in the industry as pragmatical approaches with graphical user interfaces to produce intirely operational applications or extend functionalities. In certain domains, such as database development, business processes, and web applications, lead to accelerated delivery, with a considerable reduction of the cost to produce software.

OutSystems is a Portuguese company that recently took the role of leadership for the Low Code development platforms in a very competitive landscape. In this talk you learn the history of how a small company managed complexity, quality and inovation in order to fuel growth and recognition. You will also meet the team organization challenges, product dimentions, inovation processes, and quality processes that are used internally to create an environment that drives excelency.