Module Test.Benchmark

A DSL for benchmark descriptions embedded into Curry

Author: Michael Hanus

Version: February 2019

Summary of exported operations:

benchmark :: IO a -> Benchmark a   
A benchmark is basically an I/O action to compute the benchmark results.
mapBench :: (a -> b) -> Benchmark a -> Benchmark b   
Maps benchmark results according to a given mapping (first argument).
pairBench :: Benchmark a -> Benchmark b -> Benchmark (a,b)   
Combines two benchmarks to a single benchmark where the results are paired.
withPrepare :: Benchmark a -> IO () -> Benchmark a   
Adds some initial preparation action to a benchmark, e.g., to generate benchmark data.
withCleanup :: Benchmark a -> IO () -> Benchmark a   
Adds some final cleanup action to a benchmark.
prepareBenchmarkCleanup :: IO () -> IO a -> IO () -> Benchmark a   
A benchmark with some preparation and some final cleanup.
(*>) :: MultiRunnable a => Int -> Benchmark a -> Benchmark a   
Iterates a benchmark multiple times and computes the average result.
iterateBench :: ([a] -> b) -> Int -> Benchmark a -> Benchmark b   
Iterates a benchmark multiple times and computes the average according to a given average function (first argument).
diffBench :: (a -> a -> a) -> Benchmark a -> Benchmark a -> Benchmark a   
Computes the difference between two benchmarks according to a given difference operation (first argument).
(.-.) :: Benchmark Float -> Benchmark Float -> Benchmark Float   
Computes the numeric difference between two Float-valued benchmarks.
runOn :: (a -> Benchmark b) -> [a] -> Benchmark [(a,b)]   
Runs a parameterized benchmark on a list of input data.
runUntilOn :: (a -> Benchmark b) -> (b -> Bool) -> [a] -> Benchmark [(a,b)]   
Runs a Maybe benchmark on an (infinite) input list of values until a benchmark delivers Nothing.
runUntilNothingOn :: Eq a => (b -> Benchmark (Maybe a)) -> [b] -> Benchmark [(b,a)]   
Run a Maybe benchmark on an (infinite) input list of values until a benchmark delivers Nothing.
execBench :: Benchmark a -> IO a   
Executes a benchmark and returns the benchmark results.
benchTimeNF :: IO a -> Benchmark Float   
Benchmark the time (in seconds) to compute the normal form of an expression.
benchCommandOutput :: String -> Benchmark (Int,String,String)   
This operation constructs a benchmark that simply returns the output of a shell command.
cmdResultAverage :: [CmdResult] -> CmdResult   
The average of a list of command benchmark results.
exitStatus :: CmdResult -> Int   
The exit status of the command benchmark result.
elapsedTime :: CmdResult -> Float   
The elapsed time (in seconds) of the command benchmark result.
cpuTime :: CmdResult -> Float   
The cpu time (in seconds) of the command benchmark result.
systemTime :: CmdResult -> Float   
The system time (in seconds) of the command benchmark result.
maxResidentMemory :: CmdResult -> Int   
The maximum resident size (in Kilobytes) of the command benchmark result.
benchCommand :: String -> Benchmark CmdResult   
Benchmark the execution of a shell command.
elapsedTime4Command :: String -> Benchmark Float   
Benchmark the elapsed time (in seconds) to execute a shell command.
cpuTime4Command :: String -> Benchmark Float   
Benchmark the cpu time (in seconds) to execute a shell command.
benchCommandWithLimit :: String -> Float -> Benchmark (Maybe CmdResult)   
Benchmark the execution of a shell command where a maximum time limit for the execution (in seconds) is given.
getHostName :: IO String   
Retrieve the host name.
getOS :: IO String   
Retrieve the operating system name (e.g., "Linux").
getSystemDescription :: IO String   
Retrieve the operating system description (e.g., "Ubuntu 12.04.3 LTS").
getSystemID :: IO String   
Retrieve the operating system id (e.g., "Ubuntu").
getSystemRelease :: IO String   
Retrieve the operating system release (e.g., "12.04").
getCoreNumber :: IO String   
Retrieve the number of cores.
getCPUModel :: IO String   
Retrieve the model of the CPU (e.g., "Intel(R) Core(TM) i5 CPU...").

Exported datatypes:


Benchmark

Representation of benchmarks. A benchmark consists of some preparation, e.g., to generate benchmark data, some final cleanup, and the benchmark itself. If a benchmark is executed several times, the preparation and cleanup work is done only once.

Constructors:


CmdResult

The result type for benchmarks timing the executing a shell command. Currently, a result contains the command, exit status, elapsed time, cpu time, system time (in seconds), and the maximum resident size (in Kilobytes).

Constructors:


Exported operations:

benchmark :: IO a -> Benchmark a   

A benchmark is basically an I/O action to compute the benchmark results.

mapBench :: (a -> b) -> Benchmark a -> Benchmark b   

Maps benchmark results according to a given mapping (first argument).

pairBench :: Benchmark a -> Benchmark b -> Benchmark (a,b)   

Combines two benchmarks to a single benchmark where the results are paired.

withPrepare :: Benchmark a -> IO () -> Benchmark a   

Adds some initial preparation action to a benchmark, e.g., to generate benchmark data. If the benchmark already contains some preparation, this new preparation is executed first.

withCleanup :: Benchmark a -> IO () -> Benchmark a   

Adds some final cleanup action to a benchmark. If the benchmark already contains some cleanup action, this new cleanup is executed last.

prepareBenchmarkCleanup :: IO () -> IO a -> IO () -> Benchmark a   

A benchmark with some preparation and some final cleanup. In this case, the preparation and cleanup work tightly belongs to the benchmark, i.e., it is repeated with every iteration of the benchmark.

(*>) :: MultiRunnable a => Int -> Benchmark a -> Benchmark a   

Iterates a benchmark multiple times and computes the average result. The preparation and cleanup actions of the benchmark are only executed once, i.e., they are not iterated. The number of executions (first argument) must be postive.

iterateBench :: ([a] -> b) -> Int -> Benchmark a -> Benchmark b   

Iterates a benchmark multiple times and computes the average according to a given average function (first argument). The preparation and cleanup actions of the benchmark are only executed once, i.e., they are not iterated.

diffBench :: (a -> a -> a) -> Benchmark a -> Benchmark a -> Benchmark a   

Computes the difference between two benchmarks according to a given difference operation (first argument). This could be useful to evaluate some kernel of a computation where the ressources to prepare the benchmark data are measured by a separate benchmark and subtracted with this operation.

(.-.) :: Benchmark Float -> Benchmark Float -> Benchmark Float   

Computes the numeric difference between two Float-valued benchmarks. This could be useful to evaluate some kernel of a computation where the resources to prepare the benchmark data are measured by a separate benchmark and subtracted with this operation.

runOn :: (a -> Benchmark b) -> [a] -> Benchmark [(a,b)]   

Runs a parameterized benchmark on a list of input data. The result is a benchmark returning a list of pairs consisting of the input data and the benchmark result for this input data.

Example call:
(runOn bench benchdata)
Parameters:
  • bench : the benchmark parameterized by the input data
  • benchdata : the list of input data for the benchmarks
Returns:
Benchmark with the list of input data and benchmark results pairs

runUntilOn :: (a -> Benchmark b) -> (b -> Bool) -> [a] -> Benchmark [(a,b)]   

Runs a Maybe benchmark on an (infinite) input list of values until a benchmark delivers Nothing.

Example call:
(runUntilOn bench benchdata)
Parameters:
  • bench : the Maybe benchmark parameterized by the input data
  • benchdata : the list of input data for the benchmarks
Returns:
Benchmark with the list of input data and benchmark results pairs

runUntilNothingOn :: Eq a => (b -> Benchmark (Maybe a)) -> [b] -> Benchmark [(b,a)]   

Run a Maybe benchmark on an (infinite) input list of values until a benchmark delivers Nothing.

Example call:
(runUntilNothingOn bench benchdata)
Parameters:
  • bench : the Maybe benchmark parameterized by the input data
  • benchdata : the list of input data for the benchmarks
Returns:
Benchmark with the list of input data and benchmark results pairs

execBench :: Benchmark a -> IO a   

Executes a benchmark and returns the benchmark results.

benchTimeNF :: IO a -> Benchmark Float   

Benchmark the time (in seconds) to compute the normal form of an expression. The expression is created by an I/O action (first parameter). This avoids the sharing of the normalization process between multiple runs of the benchmark and provides more flexibility, e.g., to read benchmark input data from global variables.

benchCommandOutput :: String -> Benchmark (Int,String,String)   

This operation constructs a benchmark that simply returns the output of a shell command. To be more precise, the constructed benchmark contains as a result the exit status and the standard and error output string produced by the execution of the given shell command (provided as the argument).

cmdResultAverage :: [CmdResult] -> CmdResult   

The average of a list of command benchmark results. The exit status average is zero of all are zero.

exitStatus :: CmdResult -> Int   

The exit status of the command benchmark result.

Further infos:
  • solution complete, i.e., able to compute all solutions

elapsedTime :: CmdResult -> Float   

The elapsed time (in seconds) of the command benchmark result. If the exit status is non-zero, an error is raised.

cpuTime :: CmdResult -> Float   

The cpu time (in seconds) of the command benchmark result. If the exit status is non-zero, an error is raised.

systemTime :: CmdResult -> Float   

The system time (in seconds) of the command benchmark result. If the exit status is non-zero, an error is raised.

maxResidentMemory :: CmdResult -> Int   

The maximum resident size (in Kilobytes) of the command benchmark result. If the exit status is non-zero, an error is raised.

benchCommand :: String -> Benchmark CmdResult   

Benchmark the execution of a shell command. Returns benchmark results containing the exit status, elapsed time, cpu time, system time, and the maximum resident size (in Kilobytes).

elapsedTime4Command :: String -> Benchmark Float   

Benchmark the elapsed time (in seconds) to execute a shell command.

cpuTime4Command :: String -> Benchmark Float   

Benchmark the cpu time (in seconds) to execute a shell command.

benchCommandWithLimit :: String -> Float -> Benchmark (Maybe CmdResult)   

Benchmark the execution of a shell command where a maximum time limit for the execution (in seconds) is given. Returns Nothing, if the time limit is reached or the command terminated with a non-zero exit code, or Just the benchmark results.

getHostName :: IO String   

Retrieve the host name.

getOS :: IO String   

Retrieve the operating system name (e.g., "Linux").

getSystemDescription :: IO String   

Retrieve the operating system description (e.g., "Ubuntu 12.04.3 LTS").

getSystemID :: IO String   

Retrieve the operating system id (e.g., "Ubuntu").

getSystemRelease :: IO String   

Retrieve the operating system release (e.g., "12.04").

getCoreNumber :: IO String   

Retrieve the number of cores. Implemented by counting the processor entries in /proc/cpuinfo

getCPUModel :: IO String   

Retrieve the model of the CPU (e.g., "Intel(R) Core(TM) i5 CPU..."). Implemented by look up the model name in /proc/cpuinfo. The copyright and trademark abbreviations are omitted.