Thibault Falque
580 California St., Suite 400
San Francisco, CA, 94104
Metrics: Towards a Unified Library for Experimenting Solvers
2020, HAL (Le Centre pour la Communication Scientifique Directe)
Sign up for access to the world's latest research
Abstract
When developing a SAT solver, one of the most important parts is to perform experiments so as to evaluate its performance. Most of the time, this process remains the same, so that everybody collects almost the same statistics about the solver execution. However, how many scripts are there to retrieve experimental data and draw scatter or cactus plots? Probably as many as researchers in the domain. Based on this observation, this paper introduces Metrics, a Python library, aiming to unify and make easier the analysis of solver experiments. The ambition of Metrics is to provide a complete toolchain from the execution of the solver to the analysis of its performance. In particular, this library simplifies the retrieval of experimental data from many different inputs (including the solver's output), and provides a nice interface for drawing commonly used plots, computing statistics about the execution of the solver, and effortlessly organizing them (e.g., in Jupyter notebooks). In the end, the main purpose of Metrics is to favor the sharing and reproducibility of experimental results and their analysis.
References (9)
- Gilles Audemard and Laurent Simon. On the glucose SAT solver. Int. J. Artif. Intell. Tools, 27(1):1840001:1-1840001:25, 2018.
- Daniel Le Berre and Anne Parrain. The sat4j library, release 2.
- J. Satisf. Boolean Model. Comput., 7(2-3):59-6, 2010.
- Juliana Freire, Norbert Fuhr, and Andreas Rauber. Reproducibility of Data-Oriented Experiments in e-Science (Dagstuhl Seminar 16041). Dagstuhl Reports, 6(1):108-159, 2016.
- Thomas Herndon, Michael Ash, and Robert Pollin. Does high public debt consistently stifle eco- nomic growth? a critique of reinhart and rogoff. Cambridge Journal of Economics, 38(2):257-279, 2014.
- Yang-Min Kim, Jean-Baptiste Poline, and Guillaume Dumas. Experimenting with reproducibility: a case study of robustness in bioinformatics. GigaScience, 7(7):giy077, July 2018.
- Dirk Pilat and Yukiko Fukasaku. Oecd principles and guidelines for access to research data from public funding. Data Science Journal, 6:4-11, 06 2007.
- Carmen Reinhart and Kenneth Rogoff. Growth in a time of debt. American Economic Review, 100:573-78, 05 2010.
- Olivier Roussel. Controlling a Solver Execution: the runsolver Tool. Journal on Satisfiability, Boolean Modeling and Computation, 7:139-144, 2011.