Metrics
A Metric is a function that is used to judge the performance of your program.
Metric functions are similar to reward functions, except that the results from evaluating a metric are not used when training the program. Note that you may use any reward function as a metric.
Metrics Overview
- Base Metric class
- Metric wrappers and reduction metrics
- Accuracy metrics
- Precision and Recall metrics
- FScore metrics
- Regression metrics
- Language Model operational metrics
- Embedding Model operational metrics
- Program operational metrics
All built-in metrics
Base metrics
Batch metrics
Reduction and wrapper metrics
Accuracy metrics
Precision and recall metrics
F-score metrics
BinaryF1ScoreBinaryFBetaScoreCategoricalF1ScoreCategoricalFBetaScoreF1ScoreFBetaScoreListF1Score(alias ofCategoricalF1Score)ListFBetaScore(alias ofCategoricalFBetaScore)
Regression metrics
Agent sampling metrics
Language model operational metrics
AvgCacheCreationTokensPerCallAvgCachedTokensPerCallAvgCostPerCallAvgInputTokensPerCallAvgLatencyAvgOptimizerCacheCreationTokensPerCallAvgOptimizerCachedTokensPerCallAvgOptimizerCostPerCallAvgOptimizerInputTokensPerCallAvgOptimizerLatencyAvgOptimizerOutputTokensPerCallAvgOptimizerReasoningTokensPerCallAvgOptimizerTotalTokensPerCallAvgOutputTokensPerCallAvgReasoningTokensPerCallAvgRewardCacheCreationTokensPerCallAvgRewardCachedTokensPerCallAvgRewardCostPerCallAvgRewardInputTokensPerCallAvgRewardLatencyAvgRewardOutputTokensPerCallAvgRewardReasoningTokensPerCallAvgRewardTotalTokensPerCallAvgTimeToFirstTokenAvgTimeToLastTokenAvgTotalTokensPerCallAvgTrajectoryTimeToFirstTokenCacheCreationTokensCachedTokensCacheHitRateCostErrorRateFailedCallsFallbackActivationsInputTokensLMOperationalMetricLMOptimizersOperationalMetricLMRewardsOperationalMetricOptimizerCacheCreationTokensOptimizerCachedTokensOptimizerCacheHitRateOptimizerCostOptimizerErrorRateOptimizerFailedCallsOptimizerFallbackActivationsOptimizerInputTokensOptimizerOutputTokensOptimizerReasoningTokensOptimizerReasoningTokenShareOptimizerThroughputOptimizerTokensPerSecondOptimizerTotalTokensOutputTokensReasoningTokensReasoningTokenShareRewardCacheCreationTokensRewardCachedTokensRewardCacheHitRateRewardCostRewardErrorRateRewardFailedCallsRewardFallbackActivationsRewardInputTokensRewardOutputTokensRewardReasoningTokensRewardReasoningTokenShareRewardThroughputRewardTokensPerSecondRewardTotalTokensThroughputTokensPerSecondTotalTokens
Embedding model operational metrics
AvgEmbeddingCachedTokensPerCallAvgEmbeddingCostPerCallAvgEmbeddingLatencyAvgEmbeddingTokensPerCallAvgEmbeddingVectorsPerCallAvgOptimizerEmbeddingCachedTokensPerCallAvgOptimizerEmbeddingCostPerCallAvgOptimizerEmbeddingLatencyAvgOptimizerEmbeddingTokensPerCallAvgOptimizerEmbeddingVectorsPerCallAvgRewardEmbeddingCachedTokensPerCallAvgRewardEmbeddingCostPerCallAvgRewardEmbeddingLatencyAvgRewardEmbeddingTokensPerCallAvgRewardEmbeddingVectorsPerCallEmbeddingCachedTokensEmbeddingCacheHitRateEmbeddingCostEmbeddingErrorRateEmbeddingFailedCallsEmbeddingFallbackActivationsEmbeddingModelOperationalMetricEmbeddingModelOptimizersOperationalMetricEmbeddingModelRewardsOperationalMetricEmbeddingThroughputEmbeddingTokensEmbeddingTokensPerSecondEmbeddingVectorsEmbeddingVectorsPerSecondOptimizerEmbeddingCachedTokensOptimizerEmbeddingCacheHitRateOptimizerEmbeddingCostOptimizerEmbeddingErrorRateOptimizerEmbeddingFailedCallsOptimizerEmbeddingFallbackActivationsOptimizerEmbeddingThroughputOptimizerEmbeddingTokensOptimizerEmbeddingTokensPerSecondOptimizerEmbeddingVectorsOptimizerEmbeddingVectorsPerSecondRewardEmbeddingCachedTokensRewardEmbeddingCacheHitRateRewardEmbeddingCostRewardEmbeddingErrorRateRewardEmbeddingFailedCallsRewardEmbeddingFallbackActivationsRewardEmbeddingThroughputRewardEmbeddingTokensRewardEmbeddingTokensPerSecondRewardEmbeddingVectorsRewardEmbeddingVectorsPerSecond