Chapter 8 Runtime and Performance Analytics
In this chapter, we discuss the field of performance and runtime analytics. Using Kitchenham’s method [104], we have narrowed down the scope of this survey. For inspiration, we first explored five recent papers on runtime and performance analytics published at top conferences and journals. These five papers were selected because the papers discuss the “software” side of performance and runtime analytics, which is consistent with the scope of this book. We have chosen to focus on the field of performance Vs. energy consumption. This choice was made since studies on energy consumption refer to a contemporary and thriving research area of the runtime and performance analytics field.
8.1 Introduction
Energy consumption is an important factor in the day-to-day usage of software. Especially in the field of software development for mobile devices, considering energy consumption determines the battery-life and limits the usage time of the device. Additionally, battery life is seen as a very important attribute for most smartphone owners. 92% of potential smartphone buyers consider battery life as a significant factor in their selection criteria [142]. From the same research, it is also found that 66% of smartphone owners would pay more for a device with longer battery life, given that, on average, 63% of the users are unsatisfied with their current devices’ battery life. Although the importance of energy efficient software is clear, programmers lack the knowledge on the best practices in order to reduce software energy consumption. Also, the current educational material is not focused on this field either [145]. In an analysis of questions posted on StackOverflow.com, on the topic of energy efficiency, performed by Pinto et. al. [150], it was found that although programmers had questions related to energy efficiency, they rarely received appropriate advice. With over 2 billion daily smartphone and tablet users worldwide [58] it is evident that this lack of knowledge needs to be addressed. To help programmers close this knowledge gap, hardware-based tools have been introduced. These tools can determine the energy profile of an application with high precision. However, for these measurements, costly hardware components need to be acquired [58]. Software-based solutions solve the problem of requiring expensive hardware and are easier in use, but also less precise.
In this chapter, we represent the current state of energy efficiency in software development for apps. For this, we set up the following three research questions:
- RQ1: What is the state of the art of energy efficiency in software development for mobile apps?
- RQ2: What is the state of practice of energy efficiency in software development for mobile apps?
- RQ3: What future work needs to be done in the field of energy efficiency in software development for mobile apps?
In the following paragraph, the methodology of the study we have followed will be presented. After that, each research question will be answered. In the last paragraph of this chapter the conclusion on energy efficiency in software development for apps will be made.
8.2 Methodology
In order to answer our research questions, we have retrieved over 30 papers from the selected field. These papers have been found by searching on Google Scholar with a set of filters. First off, only papers from the following journals and conferences have been selected:
- ACM Transactions on Software Engineering Methodology (TOSEM),
- Empirical Software Engineering (EMSE),
- IEEE Transactions on Software Engineering (TSE),
- Information and Software Technology (IST),
- Journal of Systems and Software (JSS),
- ACM Computing Surveys (CSUR),
- International Conference on Software Engineering (ICSE),
- Foundations of Software Engineering (SIGSOFT FSE),
- International Conference on Automated Software Engineering (ASE),
- Working Conference on Mining Software Repositories (MSR)
- Symposium on Operating Systems Design and Implementation (OSDI)
Furthermore, during the selection of the papers, we gave priority to papers that are published not too long ago; preferably after 2012. But if a paper has been cited many times and its findings are interesting, it is also considered in our survey.
Given these two criteria the following search queries have been made:
- “Android AND Energy Efficiency AND Software”
- “Energy AND Android AND Runtime AND Analysis AND performance AND Software”
From the selected papers some backward and forward references have also been selected for us to get the complete picture. All papers are also checked for relevance to the topic.
To answer RQ1, a more in-depth investigation into available tools and guidelines for the energy efficiency is performed. For RQ2, the focus lies on what tools, guidelines and best practices are currently being used in mobile software development. Finally, RQ3 is answered by finding papers used for both RQ1 and RQ2.
8.3 RQ1: State of the Art
Many researchers found that there is an increasing demand for energy efficient software. In their research, they often propose state of the art approaches which assist developers. These approaches can be separated into two main categories: tools and guidelines. This section will contain an overview of all state of the art tools and guidelines that satisfy our search criteria. This overview is used to answer RQ1.
8.3.1 Tools
The table below shows a quick overview of the tools used to answer RQ1.
| Reference | Tool | Description |
|---|---|---|
| [52] | Leafactor | Analyze code smells |
| [116] | NavyDroid | Locate energy inefficiencies |
| [100] | Static analysis tool | Identify graphical energy bugs |
| [165] | APOA | Compare energy consumption of apps |
| [13] | Energy Patch | Detect, validate and repair energy leaks |
| [57] | PETrA | Measure energy consumption of apps |
| [147] | jStanley | Detect and improve energy bugs in Java |
Leafactor
Using static code analysis and automatic refactoring, Leafactor is able to apply Android-specific optimizations of energy efficiency. The possible optimizations are indicated by a leaf icon, and the fixed priority is provided by the official Android lint documentation. This priority reflects the severity of the energy performance, from 1 to 10 with 10 being the most severe energy consumption. Leafactor is able to detect five known energy-related optimizations.
NavyDroid
NavyDroid is a tool created on top of the Java Pathfinder (JPF). Being constructed as a strengthened DFA (deterministic finite automaton), it can accurately simulate the paused state, the killed state, and related state transitions of an activity. NavyDroid can detect complex patterns of wake lock misuses (for example multiple lock acquisitions).
Static analysis tool
The proposed static analysis is a novel static optimization technique for eliminating drawing commands to produce energy-efficient apps. The technique is exploiting the insight that static analysis is able to predict future behavior of the app. With the examples of loop invariant texture analysis, packing, and identical frames detection it indicates total energy savings up to 44% of the total energy consumption of the device.
APOA
A recommendation system which can be used in any marketplace for helping users and developers to compare apps in terms of performance. As an input, APOA uses a set of metrics and rating of apps in Comma Separated Value (CSV) format, as well as optimization metrics (considering the context of usage). The result is a Pareto optimal front, from which the user selects the most preferred solution.
Energy Patch
A framework that can systematically detect and fix energy bugs in mobile apps, in a scalable fashion. It uses a combination of static and dynamic analysis techniques to detect, validate, and repair energy bugs in Android apps. This enables EnergyPatch to quickly narrow down to the potential program paths along which energy bugs may occur.
PETrA
A novel software-based tool for measuring the energy consumption of Android apps. PETrA (Power Estimation Tool for Android) measures the energy consumption of Android apps by relying on the tools and APIs provided with the publicly available Project Volta. This means that all smartphones equipped with Android 5.0 or higher are compatible with that. The tool provides similar performance to hardware-based solutions.
jStanley
jStanley is an Eclipse plugin that helps developers to detect energy bugs. jStanley identifies in programs the use of Java collections, check if they are energy efficient and have good runtime performance and make recommendations for the usage of functions, which are more efficient as alternatives. Not only energy leaks are highlighted by the tool, but also better alternatives are provided. The tool is driven by a set of CSV files that contains energy consumption and runtime values of a wide variety of methods. These values are harvested on a specific device of the researchers. With the help of these values, the possible performance boost is calculated.
Most of the state of the art tools can be classified as performing either measurement of energy consumption or code analysis. Although plenty of such tools have been proposed, with low citation numbers, none of them seem to have had a big influence on research. There is, however, the possibility, given the growing demand for energy efficiency, that the amount and quality of tools will increase in the near future.
8.3.2 Guidelines
Another aspect of the state of the art is the currently proposed guidelines for energy awareness and improvements. To answer RQ1, seven papers have been selected that present guidelines. The research by Cruz et al. [51] shows that, interestingly enough, the best practices provided by Google falls short in addressing energy consumption. However, they come up with guidelines that do give improvements. Their guideline shows that correct usage of Android methods such as iewHolder, DrawAllocation, WakeLock, ObsoleteLayoutParam, and Recycle improved energy efficiency. From the research by McIntosh et al.[129] it is noted that machine learning is now also implemented by excited developers into their apps, having an impact on the mobile device’s battery life. They have combined empirical measurements of different machine learning algorithm implementations with complexity theory to provide the guideline that some implementations of algorithms, such as J48, MLP, and SMO, generally perform better than others. However, they also recommend that for optimal results developers must consider their own specific application since many factors, e.g. dataset size, can influence the performance. Finally, Li et al. [113] investigated commonly recommended energy-saving practices, to see if they are valid. The topics mentioned in this paper include the following aspects: optimization of the sending of HTTP packets, efficient memory usage, sufficient array lengths, valid static invocations, and correct field access. This work gives guidance to mobile app developers on bundling network packets up to a certain size and using certain coding practices for reading array length information, adequately accessing class fields, and performing invocations all led to reduced energy consumption.
8.4 RQ2: State of Practice
Now that we have an overview of the state of the art regarding energy consumption on Android-driven mobile devices, we investigate what is the current state of practice. In this section, we try to find an answer to RQ2 stated in the introduction, by going through the papers we have retrieved and using the methodology described in Section 2. First, we study how different software engineering approaches affect energy consumption and if developers have any knowledge or training regarding energy saving programming techniques. Furthermore, we try to see if there are any tools being used, which help programmers to decrease energy consumption. To make this more specific, we divide RQ2 into sub-research questions:
- SQ1: What are the current development approaches used in practice and how does that affect energy consumption?
- SQ2: What do programmers currently know and do about energy saving app development?
8.4.1 Answering SQ1
The programming language used by developers when implementing new mobile apps is one of the things that is taken into consideration for better energy usage and performance. There have been a number of studies into the effects of using a certain programming language for building software applications. Java is still the most used language for Android applications. According to Oliveira et al. [142], Java is actually not a good choice when limiting energy consumption is one of your priorities. One of the contributions of Oliveira’s et al. paper is a study where hybrid implementations of applications are compared with the original, pure Java implementations, in terms of energy consumption and performance. The paper suggests that hybrid implementations might be a good solution, but the problem is that the study was done on just four Android applications. The paper did use the Rosetta Code Repository, but this does not represent commercial software development.
Another comparable recent research paper investigated the energy-delay product (EDP), which is defined as “a weighted function of the energy consumption and run-time performance product” [72]:
\(E * T^w\)
where \(E\) is the total energy used to complete a task, \(T\) the time and \(w\) is a value that represents how important energy savings are with regard to time. The advantage of using this function is that performance is not neglected. The results in the paper show that the programming language should be chosen depending on the programming task. From these results, it is clear that Java is not performing well when comparing it to other languages, as it can be seen in the table below. However, it should be noted that the paper does not test this on actual Android applications, but rather on the Rosetta Code Repository.
![Programming Languages Average Weighted EDP Ranking, retrieved from [72]](figures/EDPranking.jpeg)
Programming Languages Average Weighted EDP Ranking, retrieved from [72]
The above two papers are examples of research studies that suggest that the current state of practice, i.e. developing applications in Java only, is not in line with the current state of the art on energy efficiency. In spite of these research papers, developers are still working mostly with Java. One possible explanation is that Java is simply compatible with almost every system, because of the Java Virtual Machine (JVM). We suggest that further research should investigate why developers are sticking to Java. It seems that the availability of software libraries, the needed learning time, the available bug-fixing tools, and the current expertise of developers might have a significant impact on their choices.
8.4.2 Answering SQ2
In the previous sections, we have seen examples of the state of the art, as well as software development methods that could lead to improvements in energy consumption. However, this knowledge only holds value when the community can get developers willing to use these approaches. There have been a number of studies that look into what is actually happening.
One example is the paper by Moura et al. [138]. In this paper, a study is conducted by looking at a large number of energy-aware commits in GitHub. This analysis has yielded a list of approaches that are being used by developers in practice. These include frequency and voltage scaling, usage of power efficient libraries, and more. The study notes that the vast majority of the commits focus on the lower levels of the software stack. Furthermore, only 16.2% of the commits were related to the use of more efficient libraries or data structures. There is also a number of software qualities that have been shown to be popular when considering energy consumption. These include correctness, responsiveness, and performance. Another point that researches have addressed is whether software developers were certain of their energy-saving commits. The paper suggests that there are definitely cases where developers are not certain about their energy saving changes. This might be caused by the fact that there are a few user-friendly tools available for aiding developers in making energy-aware decisions. One example of such a tool that attempts to assist the developer in making energy friendly decisions is jStanley [147]. However, both this paper itself and our own replication study, indicate that current attempts on efficient energy consumption are not anywhere close to ideal. Most of the studies are limited to the static analysis of the source code and with the usability of the presented tools to be poor. This prevents such techniques of being used in practice. Additionally, there is the problem of how to actually measure the energy being consumed. A number of papers explain how hard this task actually is. Moura et al. [138] also alluded to the fact that developers that do use third-party energy-management tools often lack trust in the accuracy of the tools that they are using. The measurement of energy-consumption is not actually as straightforward as one may think. In a number of papers, it is explained that hardware-based solutions are accurate enough, but require expensive components. This is a serious issue, especially in the field of mobile software development. On the other hand, software-based solutions are cheaper but less accurate. Now there is work being done on creating more accurate software solutions [58]. Early studies show promising results in combining the best of both worlds in order to obtain improved results.
8.5 RQ3: Future research
In this section, we will take a look at the suggested future research directions that have been mentioned in the papers used in this survey. We attempt to extract some general trends from the future research suggestions that are included. A similar opinion among almost all of the papers discussed in this section is that there is an agreement on the fact that the research field of energy consumption is still not very mature. The conclusions of the majority of papers mention that more research is required. However, in this survey, we found that there is a consensus regarding the belief that it is hard for researchers to make strong claims, mainly due to the doubts about both the measurement methods and the limitations of the available research methodologies on the field.
One of the fields of research we have looked at, in the previous sections, was the one referred to the relationship between programming languages and energy consumption. Multiple papers suggest that the programming languages used for implementation have an influence on the energy consumption and performance [72, 142]. These papers, however, do not look into the influence of specific features of the programming languages on energy consumption and performance.
We came across a number of tools which can be used to aid developers to manage the energy consumption of applications. However, to our knowledge, there are no studies reporting on the actual use of such tools. For future research, we suggest the investigation of the number of developers that actually use such tools. Furthermore, it is important for researchers to investigate which features should be included in tools for optimal use. For example, we replicated the paper that introduces jStanley, a tool which can be used for energy and performance optimization [147]. We faced the issue that this tool cannot implement the suggested improvements in an easy and efficient way. In particular, we analyzed the AssertJ open source project from GitHub, and we got more than 300 energy saving suggestions. A user has to click on every suggestion to actually implement the recommended improvement. This is tedious work. Similarly, a lot of the papers that mention tools that could be used in practice are currently being tested with benchmarks. However, it is always the question whether these benchmarks are representative enough of the real world to hold actual merit. Additionally, the outcomes of the papers introducing these tools are not comparable. Testing all these tools on the same projects would give better insights into what tools are actually useful for the developers to use them in terms of achieving as much energy efficiency as possible.
Even though to our knowledge, there have not been any studies yet referring to whether the tools described in the current research studies are actually been used in practice, there have been some studies into energy-saving practices on GitHub. For instance, studies taking into account commit messages [14, 138] have shed some light into the state of practice. These studies looked into the commit messages containing keywords related to energy consumption and classified the corresponding code. The future research directions that are indicated by such papers is that it will be important to verify that the energy-saving commits actually have an impact on the overall performance of the software. That is both in terms of energy consumption, as well as metrics for runtime performance and usability. Similarly, future work could be done on verifying that the energy-saving techniques are consistent across different software platforms.
Finally, it is to be noted that nearly all the investigated research studies are based on open-source projects from GitHub repositories. Research on proprietary and closed-source software could possibly lead to different results and would, therefore, be interesting to be considered.
8.6 Conclusion
Having answered the research questions of this chapter in the sections above, we can conclude that energy awareness with regard to developing applications for mobile devices, and more specific Android, needs more attention. A lot of research is done in the field, resulting in guidelines and tools that help developers. But these guidelines are very generic and also the given energy awareness information is project-specific. The tools presented by recent research papers hold claims to improvements, but for example, the validated tool jStanley is not as easy to use as it is claimed in the original paper. More research into the claims made in the papers describing these tools is needed. In practice, it is observed that the main language for creating Android apps (Java) is not the most energy efficient option. Furthermore, developers often do not give enough priority to energy-saving options and when they do they are often unsure about the effects of their changes.
References
[104] Kitchenham, B. 2004. Procedures for performing systematic reviews. Keele, UK, Keele University. 33, 2004 (2004), 1–26.
[142] Oliveira, W. et al. 2017. A study on the energy consumption of android app development approaches. Mining software repositories (msr), 2017 ieee/acm 14th international conference on (2017), 42–52.
[145] Pang, C. et al. 2016. What do programmers know about software energy consumption? IEEE Software. 33, 3 (2016), 83–89.
[150] Pinto, G. et al. 2014. Mining questions about software energy consumption. Proceedings of the 11th working conference on mining software repositories (2014), 22–31.
[58] Di Nucci, D. et al. 2017. Software-based energy profiling of android apps: Simple, efficient and reliable? Software analysis, evolution and reengineering (saner), 2017 ieee 24th international conference on (2017), 103–114.
[52] Cruz, L. and Abreu, R. 2018. Using automatic refactoring to improve energy efficiency of android apps. arXiv preprint arXiv:1803.05889. (2018).
[116] Liu, Y. et al. 2017. NavyDroid: Detecting energy inefficiency problems for smartphone applications. Proceedings of the 9th asia-pacific symposium on internetware (2017), 8.
[100] Kim, C.H.P. et al. 2016. Static program analysis for identifying energy bugs in graphics-intensive mobile apps. Modeling, analysis and simulation of computer and telecommunication systems (mascots), 2016 ieee 24th international symposium on (2016), 115–124.
[165] Saborido, R. et al. 2018. An app performance optimization advisor for mobile device app marketplaces. Sustainable Computing: Informatics and Systems. (2018).
[13] Banerjee, A. et al. 2018. Energypatch: Repairing resource leaks to improve energy-efficiency of android apps. IEEE Transactions on Software Engineering. 44, 5 (2018), 470–490.
[57] Di Nucci, D. et al. 2017. Petra: A software-based tool for estimating the energy profile of android applications. Proceedings of the 39th international conference on software engineering companion (2017), 3–6.
[147] Pereira, R. et al. 2018. JStanley: Placing a green thumb on java collections. Proceedings of the 33rd acm/ieee international conference on automated software engineering (2018), 856–859.
[51] Cruz, L. and Abreu, R. 2017. Performance-based guidelines for energy efficient mobile applications. Mobile software engineering and systems (mobilesoft), 2017 ieee/acm 4th international conference on (2017), 46–57.
[129] McIntosh, A. et al. 2018. What can android mobile app developers do about the energy consumption of machine learning? Empirical Software Engineering. (2018), 1–40.
[113] Li, D. and Halfond, W.G. 2014. An investigation into energy-saving programming practices for android smartphone app development. Proceedings of the 3rd international workshop on green and sustainable software (2014), 46–53.
[72] Georgiou, S. et al. 2018. What are your programming language’s energy-delay implications? Proceedings of the 15th international conference on mining software repositories (2018), 303–313.
[138] Moura, I. et al. 2015. Mining energy-aware commits. Proceedings of the 12th working conference on mining software repositories (2015), 56–67.
[14] Bao, L. et al. 2016. How android app developers manage power consumption?: An empirical study by mining power management commits. Proceedings of the 13th international conference on mining software repositories (2016), 37–48.