Module Test.Benchmark

A DSL for benchmark descriptions embedded into Curry

Author: Michael Hanus

Version: May 2021

Summary of exported operations:

benchmark :: IO a -> Benchmark a  Deterministic 
A benchmark is basically an I/O action to compute the benchmark results.
mapBench :: (a -> b) -> Benchmark a -> Benchmark b  Deterministic 
Maps benchmark results according to a given mapping (first argument).
pairBench :: Benchmark a -> Benchmark b -> Benchmark (a,b)  Deterministic 
Combines two benchmarks to a single benchmark where the results are paired.
withPrepare :: Benchmark a -> IO () -> Benchmark a  Deterministic 
Adds some initial preparation action to a benchmark, e.g., to generate benchmark data.
withCleanup :: Benchmark a -> IO () -> Benchmark a  Deterministic 
Adds some final cleanup action to a benchmark.
prepareBenchmarkCleanup :: IO () -> IO a -> IO () -> Benchmark a  Deterministic 
A benchmark with some preparation and some final cleanup.
(**>) :: MultiRunnable a => Int -> Benchmark a -> Benchmark a  Deterministic 
Iterates a benchmark multiple times and computes the average result.
iterateBench :: ([a] -> b) -> Int -> Benchmark a -> Benchmark b  Deterministic 
Iterates a benchmark multiple times and computes the average according to a given average function (first argument).
diffBench :: (a -> a -> a) -> Benchmark a -> Benchmark a -> Benchmark a  Deterministic 
Computes the difference between two benchmarks according to a given difference operation (first argument).
(.-.) :: Benchmark Float -> Benchmark Float -> Benchmark Float  Deterministic 
Computes the numeric difference between two Float-valued benchmarks.
runOn :: (a -> Benchmark b) -> [a] -> Benchmark [(a,b)]  Deterministic 
Runs a parameterized benchmark on a list of input data.
runUntilOn :: (a -> Benchmark b) -> (b -> Bool) -> [a] -> Benchmark [(a,b)]  Deterministic 
Runs a Maybe benchmark on an (infinite) input list of values until a benchmark delivers Nothing.
runUntilNothingOn :: Eq a => (b -> Benchmark (Maybe a)) -> [b] -> Benchmark [(b,a)]  Deterministic 
Run a Maybe benchmark on an (infinite) input list of values until a benchmark delivers Nothing.
execBench :: Benchmark a -> IO a  Deterministic 
Executes a benchmark and returns the benchmark results.
benchTimeNF :: IO a -> Benchmark Float  Deterministic 
Benchmark the time (in seconds) to compute the normal form of an expression.
benchCommandOutput :: String -> Benchmark (Int,String,String)  Non-deterministic 
This operation constructs a benchmark that simply returns the output of a shell command.
cmdResultAverage :: [CmdResult] -> CmdResult  Deterministic 
The average of a list of command benchmark results.
exitStatus :: CmdResult -> Int  Deterministic 
The exit status of the command benchmark result.
elapsedTime :: CmdResult -> Float  Deterministic 
The elapsed time (in seconds) of the command benchmark result.
cpuTime :: CmdResult -> Float  Deterministic 
The cpu time (in seconds) of the command benchmark result.
systemTime :: CmdResult -> Float  Deterministic 
The system time (in seconds) of the command benchmark result.
maxResidentMemory :: CmdResult -> Int  Deterministic 
The maximum resident size (in Kilobytes) of the command benchmark result.
benchCommand :: String -> Benchmark CmdResult  Non-deterministic 
Benchmark the execution of a shell command.
elapsedTime4Command :: String -> Benchmark Float  Non-deterministic 
Benchmark the elapsed time (in seconds) to execute a shell command.
cpuTime4Command :: String -> Benchmark Float  Non-deterministic 
Benchmark the cpu time (in seconds) to execute a shell command.
benchCommandWithLimit :: String -> Float -> Benchmark (Maybe CmdResult)  Non-deterministic 
Benchmark the execution of a shell command where a maximum time limit for the execution (in seconds) is given.
getHostName :: IO String  Deterministic 
Retrieve the host name.
getOS :: IO String  Deterministic 
Retrieve the operating system name (e.g., "Linux").
getSystemDescription :: IO String  Deterministic 
Retrieve the operating system description (e.g., "Ubuntu 12.04.3 LTS").
getSystemID :: IO String  Deterministic 
Retrieve the operating system id (e.g., "Ubuntu").
getSystemRelease :: IO String  Deterministic 
Retrieve the operating system release (e.g., "12.04").
getCoreNumber :: IO String  Deterministic 
Retrieve the number of cores.
getCPUModel :: IO String  Deterministic 
Retrieve the model of the CPU (e.g., "Intel(R) Core(TM) i5 CPU...").

Exported datatypes:


Benchmark

Representation of benchmarks. A benchmark consists of some preparation, e.g., to generate benchmark data, some final cleanup, and the benchmark itself. If a benchmark is executed several times, the preparation and cleanup work is done only once.

Constructors:


CmdResult

The result type for benchmarks timing the executing a shell command. Currently, a result contains the command, exit status, elapsed time, cpu time, system time (in seconds), and the maximum resident size (in Kilobytes).

Constructors:


Exported operations:

benchmark :: IO a -> Benchmark a  Deterministic 

A benchmark is basically an I/O action to compute the benchmark results.

mapBench :: (a -> b) -> Benchmark a -> Benchmark b  Deterministic 

Maps benchmark results according to a given mapping (first argument).

pairBench :: Benchmark a -> Benchmark b -> Benchmark (a,b)  Deterministic 

Combines two benchmarks to a single benchmark where the results are paired.

withPrepare :: Benchmark a -> IO () -> Benchmark a  Deterministic 

Adds some initial preparation action to a benchmark, e.g., to generate benchmark data. If the benchmark already contains some preparation, this new preparation is executed first.

withCleanup :: Benchmark a -> IO () -> Benchmark a  Deterministic 

Adds some final cleanup action to a benchmark. If the benchmark already contains some cleanup action, this new cleanup is executed last.

prepareBenchmarkCleanup :: IO () -> IO a -> IO () -> Benchmark a  Deterministic 

A benchmark with some preparation and some final cleanup. In this case, the preparation and cleanup work tightly belongs to the benchmark, i.e., it is repeated with every iteration of the benchmark.

(**>) :: MultiRunnable a => Int -> Benchmark a -> Benchmark a  Deterministic 

Iterates a benchmark multiple times and computes the average result. The preparation and cleanup actions of the benchmark are only executed once, i.e., they are not iterated. The number of executions (first argument) must be postive.

iterateBench :: ([a] -> b) -> Int -> Benchmark a -> Benchmark b  Deterministic 

Iterates a benchmark multiple times and computes the average according to a given average function (first argument). The preparation and cleanup actions of the benchmark are only executed once, i.e., they are not iterated.

diffBench :: (a -> a -> a) -> Benchmark a -> Benchmark a -> Benchmark a  Deterministic 

Computes the difference between two benchmarks according to a given difference operation (first argument). This could be useful to evaluate some kernel of a computation where the ressources to prepare the benchmark data are measured by a separate benchmark and subtracted with this operation.

(.-.) :: Benchmark Float -> Benchmark Float -> Benchmark Float  Deterministic 

Computes the numeric difference between two Float-valued benchmarks. This could be useful to evaluate some kernel of a computation where the resources to prepare the benchmark data are measured by a separate benchmark and subtracted with this operation.

runOn :: (a -> Benchmark b) -> [a] -> Benchmark [(a,b)]  Deterministic 

Runs a parameterized benchmark on a list of input data. The result is a benchmark returning a list of pairs consisting of the input data and the benchmark result for this input data.

Example call:
(runOn bench benchdata)
Parameters:
  • bench : the benchmark parameterized by the input data
  • benchdata : the list of input data for the benchmarks
Returns:
Benchmark with the list of input data and benchmark results pairs

runUntilOn :: (a -> Benchmark b) -> (b -> Bool) -> [a] -> Benchmark [(a,b)]  Deterministic 

Runs a Maybe benchmark on an (infinite) input list of values until a benchmark delivers Nothing.

Example call:
(runUntilOn bench benchdata)
Parameters:
  • bench : the Maybe benchmark parameterized by the input data
  • benchdata : the list of input data for the benchmarks
Returns:
Benchmark with the list of input data and benchmark results pairs

runUntilNothingOn :: Eq a => (b -> Benchmark (Maybe a)) -> [b] -> Benchmark [(b,a)]  Deterministic 

Run a Maybe benchmark on an (infinite) input list of values until a benchmark delivers Nothing.

Example call:
(runUntilNothingOn bench benchdata)
Parameters:
  • bench : the Maybe benchmark parameterized by the input data
  • benchdata : the list of input data for the benchmarks
Returns:
Benchmark with the list of input data and benchmark results pairs

execBench :: Benchmark a -> IO a  Deterministic 

Executes a benchmark and returns the benchmark results.

benchTimeNF :: IO a -> Benchmark Float  Deterministic 

Benchmark the time (in seconds) to compute the normal form of an expression. The expression is created by an I/O action (first parameter). This avoids the sharing of the normalization process between multiple runs of the benchmark and provides more flexibility, e.g., to read benchmark input data from global variables.

benchCommandOutput :: String -> Benchmark (Int,String,String)  Non-deterministic 

This operation constructs a benchmark that simply returns the output of a shell command. To be more precise, the constructed benchmark contains as a result the exit status and the standard and error output string produced by the execution of the given shell command (provided as the argument).

cmdResultAverage :: [CmdResult] -> CmdResult  Deterministic 

The average of a list of command benchmark results. The exit status average is zero of all are zero.

exitStatus :: CmdResult -> Int  Deterministic 

The exit status of the command benchmark result.

Further infos:
  • solution complete, i.e., able to compute all solutions

elapsedTime :: CmdResult -> Float  Deterministic 

The elapsed time (in seconds) of the command benchmark result. If the exit status is non-zero, an error is raised.

cpuTime :: CmdResult -> Float  Deterministic 

The cpu time (in seconds) of the command benchmark result. If the exit status is non-zero, an error is raised.

systemTime :: CmdResult -> Float  Deterministic 

The system time (in seconds) of the command benchmark result. If the exit status is non-zero, an error is raised.

maxResidentMemory :: CmdResult -> Int  Deterministic 

The maximum resident size (in Kilobytes) of the command benchmark result. If the exit status is non-zero, an error is raised.

benchCommand :: String -> Benchmark CmdResult  Non-deterministic 

Benchmark the execution of a shell command. Returns benchmark results containing the exit status, elapsed time, cpu time, system time, and the maximum resident size (in Kilobytes).

elapsedTime4Command :: String -> Benchmark Float  Non-deterministic 

Benchmark the elapsed time (in seconds) to execute a shell command.

cpuTime4Command :: String -> Benchmark Float  Non-deterministic 

Benchmark the cpu time (in seconds) to execute a shell command.

benchCommandWithLimit :: String -> Float -> Benchmark (Maybe CmdResult)  Non-deterministic 

Benchmark the execution of a shell command where a maximum time limit for the execution (in seconds) is given. Returns Nothing, if the time limit is reached or the command terminated with a non-zero exit code, or Just the benchmark results.

getHostName :: IO String  Deterministic 

Retrieve the host name.

getOS :: IO String  Deterministic 

Retrieve the operating system name (e.g., "Linux").

getSystemDescription :: IO String  Deterministic 

Retrieve the operating system description (e.g., "Ubuntu 12.04.3 LTS").

getSystemID :: IO String  Deterministic 

Retrieve the operating system id (e.g., "Ubuntu").

getSystemRelease :: IO String  Deterministic 

Retrieve the operating system release (e.g., "12.04").

getCoreNumber :: IO String  Deterministic 

Retrieve the number of cores. Implemented by counting the processor entries in /proc/cpuinfo

getCPUModel :: IO String  Deterministic 

Retrieve the model of the CPU (e.g., "Intel(R) Core(TM) i5 CPU..."). Implemented by look up the model name in /proc/cpuinfo. The copyright and trademark abbreviations are omitted.