Score Data Models
Metric-shaped data models: discretized scales the LM can pick from.
FineScore
Bases: float, Enum
A discretized confidence score on a 21-level scale from 0.0 to 1.0.
FineScore is the finer-grained sibling of Score: it steps by 0.05
instead of 0.1, giving the language model twice the resolution when it
picks a confidence level. Every Score value is also a FineScore
value (with the same name), with one extra level slotted between each
pair. Like Score, it is both a float and an Enum, so the JSON
schema constrains the model to one of the fixed values while Python
code can use the value in arithmetic directly.
Reach for FineScore when the 11 levels of Score are too coarse
(e.g. ranking many candidates that would otherwise tie), and stay with
Score when you want the model to commit to broader buckets.
The labels:
| Name | Value |
|---|---|
VERY_BAD |
0.00 |
BAD |
0.05 |
POOR |
0.10 |
VERY_LOW |
0.15 |
BELOW_AVERAGE |
0.20 |
LOW |
0.25 |
LOW_AVERAGE |
0.30 |
SLIGHTLY_LOW |
0.35 |
MEDIUM_LOW |
0.40 |
SLIGHTLY_BELOW_MEDIUM |
0.45 |
MEDIUM |
0.50 |
SLIGHTLY_ABOVE_MEDIUM |
0.55 |
MEDIUM_HIGH |
0.60 |
SLIGHTLY_HIGH |
0.65 |
ABOVE_AVERAGE |
0.70 |
HIGH |
0.75 |
HIGH_AVERAGE |
0.80 |
VERY_HIGH |
0.85 |
GOOD |
0.90 |
EXCELLENT |
0.95 |
VERY_GOOD |
1.00 |
Example:
import synalinks
class Relevance(synalinks.DataModel):
relevance: synalinks.FineScore = synalinks.Field(
description="How relevant the document is to the query",
)
# FineScore values are real floats, usable in arithmetic.
assert synalinks.FineScore.HIGH == 0.75
assert synalinks.FineScore.GOOD == synalinks.Score.GOOD
Source code in synalinks/src/backend/pydantic/metrics.py
79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 | |
Rating
Bases: int, Enum
A discretized rating on a 5-level integer scale from 1 to 5.
Rating is the integer, Likert-style counterpart of Score: instead
of a float between 0.0 and 1.0, the language model picks one of five
whole numbers, 1 (worst) to 5 (best). Because Rating is both an int
and an Enum, the JSON schema constrains the model to exactly those
five values while Python code can use the value in arithmetic
directly (e.g. averaging several ratings, or dividing by 5 to get a
score between 0.2 and 1.0).
Reach for Rating when you want the familiar "rate this from 1 to 5"
framing (star ratings, Likert surveys, rubric grades), and for Score
or FineScore when you want a normalized float.
The labels are the spelled-out numbers ONE (1) through FIVE (5).
Example:
import synalinks
class Review(synalinks.DataModel):
rating: synalinks.Rating = synalinks.Field(
description="Overall quality of the answer, from 1 (worst) to 5 (best)",
)
# Rating values are real ints, usable in arithmetic.
assert synalinks.Rating.FOUR == 4
normalized = synalinks.Rating.FOUR / 5 # 0.8
Source code in synalinks/src/backend/pydantic/metrics.py
Rating10
Bases: int, Enum
A discretized rating on a 10-level integer scale from 1 to 10.
Rating10 is the 10-point variant of Rating: the language model
picks one whole number from 1 (worst) to 10 (best). Like Rating, it
is both an int and an Enum, so the JSON schema constrains the model
to exactly those ten values while Python code can use the value in
arithmetic directly (e.g. divide by 10 to get a score between 0.1 and
1.0).
Reach for Rating10 when the 5 levels of Rating are too coarse but
you still want the "rate this out of 10" framing; use Rating20 for
an even finer integer scale, and Score / FineScore when you want a
normalized float instead.
The labels are the spelled-out numbers ONE (1) through TEN (10).
Example:
import synalinks
class Review(synalinks.DataModel):
rating: synalinks.Rating10 = synalinks.Field(
description="Overall quality of the answer, from 1 (worst) to 10 (best)",
)
# Rating10 values are real ints, usable in arithmetic.
assert synalinks.Rating10.SEVEN == 7
normalized = synalinks.Rating10.SEVEN / 10 # 0.7
Source code in synalinks/src/backend/pydantic/metrics.py
Rating20
Bases: int, Enum
A discretized rating on a 20-level integer scale from 1 to 20.
Rating20 is the 20-point variant of Rating: the language model
picks one whole number from 1 (worst) to 20 (best). Like Rating, it
is both an int and an Enum, so the JSON schema constrains the model
to exactly those twenty values while Python code can use the value in
arithmetic directly (e.g. divide by 20 to get a score between 0.05 and
1.0).
Reach for Rating20 when you need fine integer resolution (e.g. a
"grade out of 20" rubric, or ranking many candidates that would tie on
a coarser scale); use Rating or Rating10 for broader buckets, and
Score / FineScore when you want a normalized float instead.
The labels are the spelled-out numbers ONE (1) through TWENTY (20).
Example:
import synalinks
class Grade(synalinks.DataModel):
grade: synalinks.Rating20 = synalinks.Field(
description="Grade of the essay, from 1 (worst) to 20 (best)",
)
# Rating20 values are real ints, usable in arithmetic.
assert synalinks.Rating20.FIFTEEN == 15
normalized = synalinks.Rating20.FIFTEEN / 20 # 0.75
Source code in synalinks/src/backend/pydantic/metrics.py
Score
Bases: float, Enum
A discretized confidence score on an 11-level scale from 0.0 to 1.0.
Use Score as the type of a DataModel field when you want the
language model to pick a confidence level from a fixed set of named
values rather than to emit an arbitrary float. Because Score is
both a float and an Enum, the JSON schema constrains the model
to one of the eleven labels, while downstream Python code can use
the value in arithmetic directly (e.g. 0.95 is Score.GOOD + 0.05).
The labels:
| Name | Value |
|---|---|
VERY_BAD |
0.0 |
POOR |
0.1 |
BELOW_AVERAGE |
0.2 |
LOW_AVERAGE |
0.3 |
MEDIUM_LOW |
0.4 |
MEDIUM |
0.5 |
MEDIUM_HIGH |
0.6 |
ABOVE_AVERAGE |
0.7 |
HIGH_AVERAGE |
0.8 |
GOOD |
0.9 |
VERY_GOOD |
1.0 |
Example:
import synalinks
class Sentiment(synalinks.DataModel):
joy: synalinks.Score = synalinks.Field(
description="How strongly the text expresses joy",
)
anger: synalinks.Score = synalinks.Field(
description="How strongly the text expresses anger",
)
# Score values are real floats, usable in arithmetic.
blended = synalinks.Score.GOOD + 0.05 # approx 0.95
See synalinks/src/metrics/f_score_metrics.py and
examples/19_multi_objective_lm_selection.py for end-to-end usage
inside metrics and multi-label classification.
Source code in synalinks/src/backend/pydantic/metrics.py
get_score_type(identifier)
Resolve a discretized score scale from a class or a string name.
A "score type" is any Enum whose members are also int or float
(Score, FineScore, Rating, Rating10, Rating20, or your own).
Modules such as SelfCritique and LMAsJudge take a score_type
argument and use this helper so that both the class itself and its
name (as stored in a serialized config) are accepted.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
identifier
|
type | str | None
|
The score type class, its name
( |
required |
Returns:
| Type | Description |
|---|---|
type
|
The resolved score type class. |
Source code in synalinks/src/backend/pydantic/metrics.py
normalize_score(value, score_type)
Normalize a raw score to a float between 0.0 and 1.0.
The lowest member of score_type maps to 0.0 and the highest to
1.0, linearly in between; e.g. Rating.THREE (3 on a 1..5 scale)
maps to 0.5, Rating10.SEVEN to 0.667, and any Score /
FineScore value is returned unchanged since it already spans 0..1.
The result is clamped to [0.0, 1.0] so a value slightly outside the
scale (e.g. from a lenient provider) cannot produce an invalid reward.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
value
|
int | float | Enum
|
The raw score, as emitted by the
language model (a plain number or a member of |
required |
score_type
|
type | str
|
The scale the value was picked from
(see |
required |
Returns:
| Type | Description |
|---|---|
float
|
The normalized score between 0.0 and 1.0. |
Source code in synalinks/src/backend/pydantic/metrics.py
score_type_bounds(score_type)
Return (minimum, maximum) member values of a score type.
score_type_description(score_type)
Return a short, LM-facing description of the scale of a score type.
Source code in synalinks/src/backend/pydantic/metrics.py
score_type_json_type(score_type)
Return the JSON schema type ("integer" or "number") of a score type.