Towards green scientific data compression through high-level I/O interfaces

Alforov, Y., Novikova, A., Kuhn, M., Kunkel, J. and Ludwig, T. (2018) Towards green scientific data compression through high-level I/O interfaces. In: SBAC-PAD 2018, 2018-12-31, Lyon, France, pp. 209-216.

Abstract/Summary

Every HPC system today has to cope with a deluge of data generated by scientific applications, simulations or large- scale experiments. The upscaling of supercomputer systems and infrastructures, generally results in a dramatic increase of their energy consumption. In this paper, we argue that techniques like data compression can lead to significant gains in terms of power efficiency by reducing both network and storage requirements. To that end, we propose a novel methodology for achieving on-the-fly intelligent determination of energy efficient data reduction for a given data set by leveraging state-of-the-art compression algorithms and meta data at application-level I/O. We motivate our work by analyzing the energy and storage saving needs of real-life scientific HPC applications, and review the various compression techniques that can be applied. We find that the resulting data reduction can decrease the data volume transferred and stored by as much as 80% in some cases, consequently leading to significant savings in storage and networking costs.