Bases: LMOperationalMetric
Average time-to-last-token (TTLT) in seconds over streamed LM calls.
Measured per streamed call as the wall-clock from the provider request
start to the final non-empty chunk, then averaged over the streamed calls
in this run -- i.e. the mean end-to-end duration of a streamed response.
Always greater than or equal to AvgTimeToFirstToken; the gap between the
two is the generation span. Only streaming=True calls contribute, so this
reads 0.0 on runs that never stream.
Example:
program.compile(
metrics=[
synalinks.metrics.AvgTimeToLastToken(),
],
)
Source code in synalinks/src/metrics/lm_metrics.py
| @synalinks_export("synalinks.metrics.AvgTimeToLastToken")
class AvgTimeToLastToken(LMOperationalMetric):
"""Average time-to-last-token (TTLT) in seconds over streamed LM calls.
Measured per streamed call as the wall-clock from the provider request
start to the final non-empty chunk, then averaged over the streamed calls
in this run -- i.e. the mean end-to-end duration of a streamed response.
Always greater than or equal to `AvgTimeToFirstToken`; the gap between the
two is the generation span. Only `streaming=True` calls contribute, so this
reads 0.0 on runs that never stream.
Example:
```python
program.compile(
metrics=[
synalinks.metrics.AvgTimeToLastToken(),
],
)
```
"""
def __init__(self, name="avg_time_to_last_token"):
super().__init__(name=name)
def result(self):
calls = self._delta("streaming_calls")
if calls <= 0:
return 0.0
return self._delta("streaming_ttlt_s") / calls
|