Skip to content

MMLU

get_input_data_model()

Returns MMLU input data model for pipeline configurations.

Returns:

Type Description
DataModel

The MMLU input data model
Source code in synalinks/src/datasets/built_in/mmlu.py 
42
43
44
45
46
47
48
49
50
@synalinks_export("synalinks.datasets.mmlu.get_input_data_model")
def get_input_data_model():
    """
    Returns MMLU input data model for pipeline configurations.

    Returns:
        (DataModel): The MMLU input data model
    """
    return MMLUQuestion

get_output_data_model()

Returns MMLU output data model for pipeline configurations.

Returns:

Type Description
DataModel

The MMLU output data model
Source code in synalinks/src/datasets/built_in/mmlu.py 
53
54
55
56
57
58
59
60
61
@synalinks_export("synalinks.datasets.mmlu.get_output_data_model")
def get_output_data_model():
    """
    Returns MMLU output data model for pipeline configurations.

    Returns:
        (DataModel): The MMLU output data model
    """
    return MMLUAnswer

iterable_dataset(repeat=1, batch_size=1, limit=None, split='validation')

Streaming dataset for RL-style training.

Default split is "validation" (1.5k rows) — the conventional training pool for MMLU. Pass split="auxiliary_train" for the larger 99k auxiliary corpus, or split="test" for evaluation.

Returns:

Type Description
HuggingFaceDataset

A streaming, iterable dataset.
Source code in synalinks/src/datasets/built_in/mmlu.py 
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
@synalinks_export("synalinks.datasets.mmlu.iterable_dataset")
def iterable_dataset(repeat=1, batch_size=1, limit=None, split="validation"):
    """
    Streaming dataset for RL-style training.

    Default split is ``"validation"`` (1.5k rows) — the conventional
    training pool for MMLU. Pass ``split="auxiliary_train"`` for the
    larger 99k auxiliary corpus, or ``split="test"`` for evaluation.

    Returns:
        (HuggingFaceDataset): A streaming, iterable dataset.
    """
    return HuggingFaceDataset(
        path="cais/mmlu",
        name="all",
        split=split,
        streaming=True,
        input_data_model=MMLUQuestion,
        input_template=_INPUT_TEMPLATE,
        output_data_model=MMLUAnswer,
        output_template=_OUTPUT_TEMPLATE,
        batch_size=batch_size,
        limit=limit,
        repeat=repeat,
    )

load_data()

Load and format data from HuggingFace.

MMLU is an evaluation-only benchmark; the conventional split here is the validation set (1.5k examples, useful for few-shot prompt tuning) as train and the test set (14k examples) as test.

Example:

(x_train, y_train), (x_test, y_test) = synalinks.datasets.mmlu.load_data()

Returns:

Type Description
tuple

The train and test data ready for training
Source code in synalinks/src/datasets/built_in/mmlu.py 
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
@synalinks_export("synalinks.datasets.mmlu.load_data")
def load_data():
    """
    Load and format data from HuggingFace.

    MMLU is an evaluation-only benchmark; the conventional split here is
    the ``validation`` set (1.5k examples, useful for few-shot prompt
    tuning) as ``train`` and the ``test`` set (14k examples) as ``test``.

    Example:

    ```python
    (x_train, y_train), (x_test, y_test) = synalinks.datasets.mmlu.load_data()
    ```

    Returns:
        (tuple): The train and test data ready for training
    """
    x_train, y_train = _split_dataset("validation")
    x_test, y_test = _split_dataset("test")
    return (x_train, y_train), (x_test, y_test)