Skip to content

BatchMetric metric

Bases: Metric

Batched metric base class (the metric counterpart of BatchReward).

A regular Metric is updated one sample at a time by the trainer. A BatchMetric instead receives the entire batch at once, so it can compute quantities that only make sense across a group of samples, e.g. pass@k over the k samples drawn for a single problem.

The trainer detects BatchMetric instances and routes the whole (y_true, y_pred) batch to them (mirroring how BatchReward is fed via has_batch_rewards / compute_batch); non-batch metrics in the same compile(metrics=[...]) list keep their per-sample updates.

To be implemented by subclasses:

  • update_state(y_true, y_pred): y_true and y_pred are lists of length batch_size (one batch). Accumulate state from the group.
  • result(): return the current scalar (or dict) value.
Note

With the "whole batch = one problem's k samples" convention, set batch_size = k so each batch handed to the metric is the k samples of a single problem.

Example:

program.compile(
    metrics=[
        synalinks.metrics.BatchMetric(),
    ],
)
Source code in synalinks/src/metrics/batch_metric.py 
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
@synalinks_export(["synalinks.BatchMetric", "synalinks.metrics.BatchMetric"])
class BatchMetric(Metric):
    """Batched metric base class (the metric counterpart of `BatchReward`).

    A regular `Metric` is updated one sample at a time by the trainer. A
    `BatchMetric` instead receives the **entire batch at once**, so it can
    compute quantities that only make sense across a group of samples, e.g.
    ``pass@k`` over the ``k`` samples drawn for a single problem.

    The trainer detects `BatchMetric` instances and routes the whole
    ``(y_true, y_pred)`` batch to them (mirroring how `BatchReward` is fed via
    `has_batch_rewards` / `compute_batch`); non-batch metrics in the same
    `compile(metrics=[...])` list keep their per-sample updates.

    To be implemented by subclasses:

    * ``update_state(y_true, y_pred)``: ``y_true`` and ``y_pred`` are lists of
      length ``batch_size`` (one batch). Accumulate state from the group.
    * ``result()``: return the current scalar (or dict) value.

    Note:
        With the "whole batch = one problem's k samples" convention, set
        ``batch_size = k`` so each batch handed to the metric is the ``k``
        samples of a single problem.

    Example:

    ```python
    program.compile(
        metrics=[
            synalinks.metrics.BatchMetric(),
        ],
    )
    ```
    """

    async def update_state(self, y_true, y_pred):
        raise NotImplementedError

    def result(self):
        raise NotImplementedError

    def _obj_type(self):
        return "BatchMetric"