Chapter 6 Release Engineering Analytics
6.1 Motivation
Release engineering is a software engineering discipline concerned with the development, implementation, and improvement of processes to deploy high-quality software reliably and predictably [63]. The changes made by the developers of a software system should eventually be integrated and deployed such that end users may benefit from them. In recent years, software industry has adopted techniques to build infrastructure and pipeline which automate the process of releasing software to an increasingly large degree. These modern release engineering practices have resulted in adoption of various other practices such as releasing new versions of a software system in significantly shorter cycles.
Most of the developments in release engineering are industry-driven, with less research to support the developments. Efforts to close this gap would be relevant both for practitioners as well as researchers [4]. On the one hand, claims and rationales are presented by the industry to justify practices in release engineering, but these are often not empirically validated. For this reason, research should aim to build an understanding of the actual effects of release engineering practices on the software development process. On the other hand, software engineering researchers need to be aware of modern release engineering practices in order to account for them in their analyses. Otherwise, their lack of familiarity with these practices will likely result in biases in their study results.
This systematic literature review aims to provide an overview of the software analytics research that has been conducted so far on modern release engineering. Its main purpose is to identify the apparent gap between research and practice, in order to guide further research efforts.
6.1.1 Research Questions
Contrary to what is regularly the case, advances in release engineering practices are driven by industry, instead of scientific research. Building on this idea, our questions are constructed to identify in which ways existing modern release engineering practices should still be studied in software analytics research. Our review thus aims to answer the following questions.
RQ1: How is modern release engineering done in practice?
This question aims to identify the so-called “state of the practice” in release engineering. We will summarize practices that have been adopted in modern release engineering. In addition, we will identify the tools utilized to implement modern release engineering.
RQ2: What aspects of modern release engineering have been studied in software analytics research so far?
In order to answer this question, we investigate the practices that previous empirical studies have focused on. In doing so, we identify the associated costs and benefits that have been found, and the analysis methods used.
RQ3: What aspects of modern release engineering make for relevant study objects in future software analytics research?
By answering this question, we aim to identify the gaps between practice and research in release engineering. This way, our intent is not only to guide but also to motivate future research.
6.2 Research Protocol
We follow the procedure to conduct systematic reviews as proposed by Kitchenham [104]. We set up strategies for searching, selecting and quality-assessing studies. Subsequently, we extracted data from the selected studies and synthesized the answers to our research questions.
All the papers found were stored in a custom-built web-based tool for conducting literature reviews. The source code of this tool is published in a GitHub repository.16 The tool was hosted on a virtual private server such that all retrieved publications were stored centrally, accessible to all reviewers.
In order to save space in this chapter of the book, we have omitted the full research protocol from this chapter. The interested can find our research protocol in detail in Section 7.5.
6.3 Answers
In this section, we answer each research question presented in Section 7.1.1.
6.3.1 RQ1: Modern Release Engineering Practices
How is modern release engineering done in practice?
Adams et al. [4] and Karvonen et al. [94] have described release engineering practices that are currently in use in the industry. Adams et al. [4] focused on modern release engineering, while Karvonen et al. [94] investigated agile release engineering, which is a subset of all modern release engineering. Both studies agree that modern release engineering consists of the following components:
Rapid Releases (RR).
In contrast with traditional release cycles, software systems following Rapid Release cycles regularly push new releases to users in a regular schedule. For example, FireFox releases a new version every six weeks.
DevOps.
According to Dyck et al. [63]: “DevOps is an organizational approach that stresses empathy and cross-functional collaboration within and between teams (especially development and IT operations) in software development organizations, in order to operate resilient systems and accelerate delivery of changes.”
Continuous Integration (CI).
To quote Adams et al. [4]: “[CI] refers to the activity of continuously polling the [version control system] for new commits or merges, checking these revisions out on dedicated build machines, compiling them and running an initial set of tests to check for regressions.”
Continuous Deployment or Continuous Delivery.
When the CI tests pass, the code can be automatically deployed to the production environment. The difference between these terms is that continuous delivery does not require that changes are automatically deployed, but continuous deployment always automatically deploys changes. However, the change might not yet be released, because it can be hidden using feature toggles [108].
Besides these four components, Adams et al. [4] also identified three other concepts that are used in modern release engineering, specifically to release software as often as possible and thus enable continuous deployment or delivery:
Branching and Merging.
There are several possible strategies of collaboration. Branching and merging is one of them and is most widely used. Typically, merging must be done as often as possible in order to avoid conflicts thereby enabling release of software as often as possible.
Build System.
With the build configuration stored inside the project, every developer (or automated tool) only needs to issue a single command in order to build the project, instead of manually having to configure the build process every time.
Building a software must be done in a consistent way, such that each build produces the same result.
Infrastructure-as-Code.
In the same alley of “storing configuration”, infrastructure-as-code means that the server (or virtual machine) on which a software product is running can also be automatically configured with code, instead of having to configure each server manually.
Besides these seven components that are more technical, Poo-Caamaño [154] has identified that there are also social aspects to modern release engineering. Specifically, most large software projects have a dedicated Release Team that will decide on the release strategies and communicate them to others.
6.3.2 RQ2: Studied Parts of Release Engineering
This section aims to answer to the question: What aspects of modern release engineering have been studied in software analytics research so far?
Over the years, the software industry has come up with innovative approaches to deliver new features and fixes in a more efficient and faster manner. This has resulted in case studies being done to assess the associated risk and cost factors, and what benefits certain strategies can give.
Khomh et al. [97] have looked into the effects of switching from traditional to rapid release cycles in the case of Mozilla Firefox. The paper has concluded that users do not experience significantly more post-release bugs, bugs are fixed faster, but that users experience bugs earlier in the software execution. Mantyla et al. [125] have also considered data from Mozilla Firefox and has examined the impact of release engineering on testing efforts. Observations of the paper conclude that the rapid release cycle performs more test executions per day, but these tests focus on a smaller subset of the test case corpus and that testing happens closer to release and is more continuous. Da Costa et al. [48] has further zoomed into the integration of addressed issues and have considered data from Mozilla Firefox, as well as data from ArgoUML and Eclipse. The paper found that addressed issues are usually delayed in a rapid release cycle and are often excluded from releases. Similar conclusions based on Mozilla Firefox were made by Da Costa et al. [49], who found that minor-traditional releases tend to have less integration delay than major/minor-rapid releases.
Castelluccio et al. [37] examined the practice of patch uplifting in the release management at Mozilla Firefox where patches that fix critical issues, or implement high-value features are often promoted directly from the development channel to a stabilization channel. The paper evaluated the characteristics of patch uplift decisions and interviewed three Mozilla release managers. The paper concluded that the majority of patch uplift decisions are made due to a wrong functionality or crash. The specificity and code author of patches that are requested to be uplifted are also a major factor for release managers.
In response to case studies being done on many prominent open source software projects, Teixeira [177] has described OpenStack’s shift to a liberal six-month release cycle. As this is an ongoing study, the results given by the paper are preliminary and only observe the process. OpenStack’s release process can be considered as a hybrid of feature-based and time-based releases. OpenStack encourages regular releases but also attempts to include new features at each regular release.
Rather than focussing on topics such as issue and delays, Poo-Caamaño [154] focusses on the communication in release engineering in the cases of GNOME and OpenStack. Through analyzing over 2.5 years of communication, the paper has made a number of observations. The paper found that developers tend to communicate through blogs, bug trackers, conferences, and hackfests. Another finding is that a release team is set to define requirements, quality standards, and coordination through (direct) communication. Although only the mailing lists of the projects were studied, defined challenges include keeping everyone informed and engaged, monitoring changes and setting priorities in cross-project coordination.
Laukkanen et al. [108] have described what effects modern release engineering have on software with different organizational contexts. This study specifically focusses on continuous deployment practices. The paper has found that high internal quality standards combined with the large distributed organizational context of large corporations slowed the verification process down and therefore had a negative impact on release capability. However, in small corporations, the lack of internal verification measures due to a lack of resources was mitigated by code review, disciplined CI and external verification by customers in customer environments. More about the factors that can play a role is addressed by Rodríguez et al. [163], where an overview of contributing factors in continuous deployment are defined and categorized based on literature between 2001 and 2014.
As rapid release cycles and continuous deployment are topics that are new and emerging, not enough research has been done to generalize any conclusions that are made in the case studies discussed in this section. This is why all the empirical studies in this survey have one major sidenote in common: more case studies are needed. Open challenges such as these will be discussed in the next section.
6.3.3 RQ3: Future Research
This section aims to answer the question: What aspects of modern release engineering make for relevant study objects in future software analytics research?
6.3.3.1 General Suggestions
The body of literature that we analyzed for this survey mostly comprised case studies that employed quantitative analysis methods. From these studies, interesting conclusions have been drawn about the effects of release engineering practices on software development processes in specific contexts. However, the generalizability of the findings in these case studies is very limited. Therefore, in general many studies suggest that future research efforts focus on performing additional case studies, both to verify existing findings and to study new relationships and new contexts [4, 37, 44, 94, 97, 108, 177]. It also seems worthwhile to triangulate findings by complementing data analyses with other quantitative (e.g. a survey) or qualitative (e.g. an interview) methods [94]. Finally, additional literature reviews will allow researchers to keep an overview of the most recent developments and findings in the area of release engineering [108, 163].
Apart from verifying results, it might be worthwhile to leverage them by constructing analysis tools for practitioners. For example, Castelluccio et al. [37] suggest exploring possibilities to leverage their research by building classifiers capable of automatically assessing the risk associated with patch uplift candidates and recommend patches that can be uplifted safely. Also, companies seem to be struggling with the adoption of continuous delivery and deployment, so a checklist for analyzing readiness for these practices might be developed [94].
The review by Karvonen et al. [94] makes a number of general suggestions for future research. In particular, there should be more attention to comprehensively reporting how practices are implemented and in which context they are embedded, instead of just stating that they are used. Also, the viewpoints of different stakeholders other than developers can be taken into account. For example, the customer perceptions regarding the adoption of a certain practice can be investigated.
6.3.3.2 Directions for Specific Practices
Rapid Releases
As established, rapid releases are a prevalent topic in current research on modern release engineering. However, it will be useful to verify these results given the fact that this research mainly involves case studies (most of which are only concerned with Mozilla Firefox due to the availability of data). To this end, there are opportunities to further investigate the effects of switching to rapid releases on:
- code integration [37, 48, 49, 172],
- testing efforts [125],
- software quality [97, 98],
- (library) adoption [69]
- time pressure and work patterns [44].
DevOps
When it comes to DevOps, future research is needed to refine its definition such that it is uniform and valid for many situations [63]. According to Karvonen et al. [94], it seems that the goals in DevOps are congruent with those in release engineering, and future research on this topic is therefore highly relevant in order to study modern release engineering.
Continuous Delivery / Deployment
Research on continuous deployment seems to be still in its infancy, therefore Rodríguez et al. [163] have suggested a significant number of different concrete opportunities for future research. In general, they conclude that the topic needs an increase in the number and rigor of empirical studies, and thus it presents opportunities for software analytics research. In a systematic literature review, Laukkanen et al. [107] identified 40 problems, 28 causal relationships and 29 solutions related to the adoption of continuous delivery. These problems and solutions can be studied further to deepen the understanding of the nature of these problems and how to apply their solutions. Some of the problems that are more of a human or organizational nature might be involved with a broader spectrum of changes, so it should also be investigated to what extent these problems are specific to continuous delivery.
Continuous Integration / Build system
One of the main issues that seems to be obstructing organizations in adopting modern release engineering practices is build design [107]. In a case study by Laukkanen et al. [108], it was found that a complex automated build and integration system led to a more undisciplined one, which in turn slowed down the verification and release processes. Therefore, future research might investigate how developers can make their builds more maintainable and of higher quality, how anti-patterns in the design of the build can be refactored, and how continuous integration can be made faster and more energy efficient [4].
6.4 Conclusion
In this literature survey, we have provided an answer to the following three research questions:
RQ1: How is modern release engineering done in practice?
We found that there are six important technical aspects to modern release engineering: Rapid Releases, DevOps, Continuous Integration, Continuous Deployment, Branching and Merging, Build Configuration and Infrastructure-as-code. The most important social aspect of modern release engineering is communication.
RQ2: What aspects of modern release engineering have been studied in software analytics research so far?
At this point in time, case studies have mainly focussed on the resulting factors of switching from a traditional release cycle to a rapid release cycle, and what effects this has in various organizational contexts. As all included studies suggest, more empirical studies are needed to be able to make general conclusions in the novel field of release engineering.
RQ3: What aspects of modern release engineering make for relevant study objects in future software analytics research?
In general, more empirical research is required to validate and generalize the results of many previous case studies within the field of release engineering. In addition to performing case studies involving quantitative analyses, it may be beneficial to triangulate results using various research methods. Also, future research should more comprehensively describe how practices are implemented, and consider different stakeholders. For each practice, future research is suggested on the one hand to further investigate their effects on the development process, and on the other hand to investigate problems involved with their adoption.
References
[63] Dyck, A. et al. 2015. Towards definitions for release engineering and devops. Release engineering (releng), 2015 ieee/acm 3rd international workshop on (2015), 3–3.
[4] Adams, B. and McIntosh, S. 2016. Modern release engineering in a nutshell–why researchers should care. Software analysis, evolution, and reengineering (saner), 2016 ieee 23rd international conference on (2016), 78–90.
[104] Kitchenham, B. 2004. Procedures for performing systematic reviews. Keele, UK, Keele University. 33, 2004 (2004), 1–26.
[94] Karvonen, T. et al. 2017. Systematic literature review on the impacts of agile release engineering practices. Information and Software Technology. 86, (2017), 87–100.
[108] Laukkanen, E. et al. 2018. Comparison of release engineering practices in a large mature company and a startup. Empirical Software Engineering. (2018), 1–43.
[154] Poo-Caamaño, G. 2016. Release management in free and open source software ecosystems.
[97] Khomh, F. et al. 2015. Understanding the impact of rapid releases on software quality. Empirical Software Engineering. 20, 2 (2015), 336–373.
[125] Mäntylä, M.V. et al. 2015. On rapid releases and software testing: A case study and a semi-systematic literature review. Empirical Software Engineering. 20, 5 (2015), 1384–1425.
[48] Costa, D.A. da et al. 2014. An empirical study of delays in the integration of addressed issues. 2014 ieee international conference on software maintenance and evolution (2014), 281–290.
[49] Costa, D.A. da et al. 2016. The impact of switching to a rapid release cycle on the integration delay of addressed issues - an empirical study of the mozilla firefox project. 2016 ieee/acm 13th working conference on mining software repositories (msr) (2016), 374–385.
[37] Castelluccio, M. et al. 2017. Is it safe to uplift this patch? An empirical study on mozilla firefox. Proceedings - 2017 IEEE International Conference on Software Maintenance and Evolution, ICSME 2017 (2017), 411–421.
[177] Teixeira, J. 2017. Release early, release often and release on time. an empirical case study of release management. Open source systems: Towards robust practices (Cham, 2017), 167–181.
[163] Rodríguez, P. et al. 2017. Continuous deployment of software intensive products and services: A systematic mapping study. Journal of Systems and Software. 123, (2017), 263–291.
[44] Claes, M. et al. 2017. Abnormal working hours: Effect of rapid releases and implications to work content. IEEE International Working Conference on Mining Software Repositories (2017), 243–247.
[172] Souza, R. et al. 2015. Rapid releases and patch backouts: A software analytics approach. IEEE Software. 32, 2 (2015), 89–96.
[98] Khomh, F. et al. 2012. Do faster releases improve software quality?: An empirical case study of mozilla firefox. Proceedings of the 9th ieee working conference on mining software repositories (Piscataway, NJ, USA, 2012), 179–188.
[69] Fujibayashi, D. et al. 2017. Does the release cycle of a library project influence when it is adopted by a client project? SANER 2017 - 24th IEEE International Conference on Software Analysis, Evolution, and Reengineering (2017), 569–570.
[107] Laukkanen, E. et al. 2017. Problems, causes and solutions when adopting continuous delivery—A systematic literature review. Information and Software Technology. 82, (2017), 55–79.