medkit.text.metrics.classification

Classes

TextClassificationEvaluator

An evaluator for attributes of TextDocuments.

Module Contents

class medkit.text.metrics.classification.TextClassificationEvaluator(attr_label: str)

An evaluator for attributes of TextDocuments.

attr_label

_extract_attr_values(docs: list[medkit.core.text.TextDocument]) → list[str | int | bool]

Prepare docs attrs to compute the metric.

Parameters:

docslist of TextDocument

List of documents with attributes

Returns:

list of str or int or bool

List with the representation of the attribute by document.

compute_classification_report(true_docs: list[medkit.core.text.TextDocument], predicted_docs: list[medkit.core.text.TextDocument], metrics_by_attr_value: bool = True, average: typing_extensions.Literal[macro, weighted] = 'macro') → dict[str, float | int]

Compute classification metrics of document attributes giving annotated documents.

This method uses sklearn.metrics.classification_report to compute precision, recall and F1-score for value of the attribute.

Warning

The set of true and predicted documents must be sorted to calculate the metric

Parameters:

true_docslist of TextDocument

Text documents containing attributes of reference

predicted_docs: list of TextDocument

Text documents containing predicted attributes

metrics_by_attr_value: bool, default=True

Whether return metrics by attribute value. If False, only global metrics are returned

averagestr, default=”macro”

Type of average to be performed in metrics. - macro, unweighted mean (default) - weighted, weighted average by support (number of true instances by attr value)

Returns:

dict of str to float or int

A dictionary with the computed metrics

compute_cohen_kappa(docs_annotator_1: list[medkit.core.text.TextDocument], docs_annotator_2: list[medkit.core.text.TextDocument]) → dict[str, float | int]

Compute the cohen’s kappa score, an inter-rated agreement score between two annotators.

This method uses ‘sklearn’ as backend to compute the level of agreement.

Warning

The set of documents must be sorted to calculate the metric

Parameters:

docs_annotator_1list of TextDocument

Text documents containing attributes annotated by the first annotator

docs_annotator_2list of TextDocument

Text documents to compare, these documents contain attributes annotated by the other annotator

Returns:

dict of str to float or int

A dictionary with cohen’s kappa score and support (number of annotated docs). The value is a number between -1 and 1, where 1 indicates perfect agreement; zero or lower indicates chance agreement.

compute_krippendorff_alpha(docs_annotators: list[list[medkit.core.text.TextDocument]]) → dict[str, float | int]

Compute the Krippendorff alpha score, an inter-rated agreement score between multiple annotators.

Warning

Documents must be sorted to calculate the metric.

Note

See medkit.text.metrics.irr_utils.krippendorff_alpha for more information about the score

Parameters:

docs_annotatorslist of list of TextDocument

A list of list of Text documents containing attributes. The size of the list is the number of annotators to compare.

Returns:

dict of str to float or int

A dictionary with the krippendorff alpha score, number of annotators and support (number of documents). A value of 1 indicates perfect reliability between annotators; zero or lower indicates absence of reliability.