quaterion.eval.evaluator module

class Evaluator(metrics: Union[BaseMetric, Dict[str, BaseMetric]], sampler: BaseSampler)

Bases: object

Calculate metrics on the whole datasets

Calculates metric on the whole dataset or on sampled part of it. Evaluation might be time and memory consuming operation.

Parameters:

• metrics – dictionary of metrics instances for calculation

• sampler – sampler selects embeddings and labels to perform partial evaluation

evaluate(dataset: Union[Sized, Iterable, Dataset], model: SimilarityModel) → Dict[str, Tensor]

Compute metrics on a dataset

Parameters:

• dataset – Sized object, like list, tuple, torch.utils.data.Dataset, etc. to compute metrics

• model – SimilarityModel instance to perform objects encoding

Returns:

Dict[str, torch.Tensor] - dict of computed metrics