Bases: LMOperationalMetric
Fraction of prompt tokens served from cache: cached / prompt_tokens.
A high value here is one of the biggest cost levers; aim for 0.7+
on a production workload with stable system prompts.
Example:
program.compile(
metrics=[
synalinks.metrics.CacheHitRate(),
],
)
Source code in synalinks/src/metrics/lm_metrics.py
| @synalinks_export("synalinks.metrics.CacheHitRate")
class CacheHitRate(LMOperationalMetric):
"""Fraction of prompt tokens served from cache: cached / prompt_tokens.
A high value here is one of the biggest cost levers; aim for 0.7+
on a production workload with stable system prompts.
Example:
```python
program.compile(
metrics=[
synalinks.metrics.CacheHitRate(),
],
)
```
"""
def __init__(self, name="cache_hit_rate"):
super().__init__(name=name)
def result(self):
prompt = self._delta("prompt_tokens")
if prompt <= 0:
return 0.0
return self._delta("cached_tokens") / prompt
|