Metrics#

`build_metric`	Build an evaluation metric.
`SerializableResult`	Final evaluation result that can be serialized to a JSON or yaml file.
`ReInferenceMetric`	Re-inference based metric.
`ReTrainingMetric`	Re-training based metric.
`AttributionMethod`	Protocol of attribution (also known as explanation) methods.

Building Functions#

build_metric#

saliency_metrics.metrics.build_metric(cfg, default_args=None)[source]#

Build an evaluation metric.

Parameters

cfg (Dict) – Config dictionary.
default_args (Optional[Dict]) – Other default arguments.

Return type

Union[ReInferenceMetric, ReTrainingMetric]

Returns

A evaluation metric.

General Protocols and Base Classes#

Serializable Result#

protocol saliency_metrics.metrics.SerializableResult[source]#

Final evaluation result that can be serialized to a JSON or yaml file.

This protocol is runtime checkable.

Classes that implement this protocol must have the following methods / attributes:

add_single_result(single_result)[source]#

Add single evaluation result.

Parameters: single_result (Any) – The result on a single sample or the result under a single re-training setting.
Return type: None
Returns: None

abstract dump(file_path)[source]#

Save the result to a JSON or yaml file.

Parameters: file_path (str) – Output file path.
Return type: None
Returns: None

Re-inference Based Metric#

protocol saliency_metrics.metrics.ReInferenceMetric[source]#

Re-inference based metric.

A Metric implementing this protocol performs per-sample evaluation at inference time. Specifically, it first perturbs the input image according to the saliency map and then measure the degradation of the model’s prediction.

This protocol is runtime checkable.

Classes that implement this protocol must have the following methods / attributes:

abstract evaluate(img, smap, target, **kwargs)[source]#

Perform evaluation on a single sample.

Parameters

img (Union[Tensor, ndarray]) – Input image.
smap (Union[Tensor, ndarray]) – Saliency map.
target (Union[Tensor, int]) – Ground-truth target.
kwargs (Any) – Other optional arguments for example the image path, original image size, and so on.

Return type

Dict

Returns

Evaluation result on a single sample.

get_result()[source]#

Get the cached result.

Return type: SerializableResult
Returns: The final evaluation result for the whole dataset.

abstract update(single_result)[source]#

Given the evaluation result on a single sample, update the cached result (for the whole dataset).

Parameters: single_result (Dict) – Evaluation result on a single sample.
Return type: None
Returns: None

Re-training Based Metric#

protocol saliency_metrics.metrics.ReTrainingMetric[source]#

Re-training based metric.

A metric implementing this protocol re-trains a model on a perturbed dataset and evaluate the performance degradation.

This protocol is runtime checkable.

Classes that implement this protocol must have the following methods / attributes:

abstract evaluate(cfg, dist_args=None)[source]#

Perform re-training evaluation on the whole dataset.

Parameters

cfg (Config) – Config. It specifies the hyper-parameters of e.g., dataset, model, optimizer, lr-scheduler, max epochs etc.
dist_args (Optional[Dict]) – DDP training hyper-parameters e.g. nproc_per_node, backend etc. See also: Parallel.

Return type

None

Returns

None

get_result()[source]#

Get the cached result.

Return type: SerializableResult
Returns: The final evaluation result for the whole dataset.

Attribution Method#

protocol saliency_metrics.metrics.AttributionMethod[source]#

Protocol of attribution (also known as explanation) methods.

This protocol is mainly used in the Sanity Check metric, where an attribution method must implement this protocol.

This protocol is runtime checkable.

Classes that implement this protocol must have the following methods / attributes:

abstract attribute(img, target, as_ndarray=True, interpolate=True, interpolate_args=None, **kwargs)[source]#

Attribute and produce the saliency map.

Note

The method performs attribution on single image, i.e., the local explanation.

Parameters

img (Tensor) – Input image with shape (1, num_channels, height, width).
target (Union[int, Sequence[int], Tensor]) – Ground-truth target.
as_ndarray (bool) – If True, then convert the saliency map into a ndarray.
interpolate (bool) – if True, then resize and interpolate the saliency map to the image’s spatial size.
interpolate_args (Optional[Dict]) – Other arguments for interpolation. See also saliency_metrics.utils.resize_img().
**kwargs – Other keyword arguments for attribution.

Return type

Union[Tensor, ndarray, Tuple[Union[Tensor, ndarray]]]

Returns

A saliency map or a tuple of saliency maps.