Skip to content

Multimodal Data Models

Multimodal content types.

A modality is just a content part of a Message: the OpenAI/litellm chat-completion content is either a string or a list of parts, and an image or an audio clip is one of those parts. So multimodal input is plain chat-completion: Image and Audio are special types you drop straight into a ChatMessage's content list, mixed with the text that usually goes with them:

synalinks.ChatMessage(
    role="user",
    content=[
        "What is in this picture? Answer in one sentence.",
        synalinks.Image(url="https://example.com/cat.png"),
    ],
)

ChatMessage normalizes each element to its wire part (a plain str becomes a text part, an Image/Audio becomes its own part), so the message stays a strict chat-completion message and flows through the normal generator path.

Two ways a source becomes a payload

A url or path is just a reference; at some point the actual bytes have to be read and inlined as a base64 data: URI so every provider, including local ones that cannot fetch a URL themselves, receives the same self-contained payload. There are two moments that can happen, for two different use cases:

  1. At construction, when you build an Image(url=...) / Audio(path=...) in Python. The bytes are fetched/read immediately, so an unreachable source fails loudly right where it is written and the object is reusable without re-fetching. This is the right default for hand-written, interactive use.

  2. Per batch, at inference, for content built from raw JSON rather than the Python constructor: most importantly Dataset rows, which go through ChatMessages.model_validate_json(...). There the image_url/input_audio parts stay as lightweight references (a url/path), so a dataset of a million images never inlines a million payloads into memory. Resolution is deferred to resolve_content_media, which the language model calls on the messages it is about to send; i.e. only the current batch's media is ever resolved at once, and it is freed as soon as the request goes out.

A payload already inline (base64 data, or a data: URI) is left untouched by both paths, so the per-batch resolver is a no-op on construction-resolved content.

Adding a modality (video, documents, ...) is a new DataModel here with a to_content_part() method; nothing else in the stack needs to change.

Audio

Bases: DataModel

An audio content part for a chat message.

Provide a source (a url, a local file path, or raw base64 data) and a container format (e.g. "wav" or "mp3"). The chat-completion input_audio part carries inline base64 only, so a url/path is fetched and inlined at construction.

Source code in synalinks/src/backend/pydantic/media.py 
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
@synalinks_export(
    [
        "synalinks.backend.Audio",
        "synalinks.Audio",
    ]
)
class Audio(DataModel):
    """An audio content part for a chat message.

    Provide a source (a `url`, a local file `path`, or raw base64 `data`) and
    a container `format` (e.g. ``"wav"`` or ``"mp3"``). The chat-completion
    `input_audio` part carries inline base64 only, so a `url`/`path` is fetched
    and inlined at construction.
    """

    url: Optional[str] = Field(
        default=None,
        description="An http(s) URL pointing to the audio; fetched at construction.",
    )
    path: Optional[str] = Field(
        default=None,
        description="A local file path to the audio; read at construction.",
    )
    data: Optional[str] = Field(
        default=None,
        description="Raw base64-encoded audio bytes.",
    )
    format: Optional[str] = Field(
        default=None,
        description="The audio container format, e.g. `wav` or `mp3`.",
    )

    @model_validator(mode="after")
    def _inline_source(self):
        """Fetch/read a `url`/`path` source and inline it as base64 `data`.

        Runs once, at construction. A payload already given as base64 `data` is
        left untouched.
        """
        if self.data is not None:
            return self
        read = _read_source(self.url, self.path)
        if read is not None:
            raw, _ = read
            self.data = base64.b64encode(raw).decode("ascii")
        return self

    def to_content_part(self):
        """Render this audio as an OpenAI/litellm `input_audio` content part."""
        return {
            "type": "input_audio",
            "input_audio": {"data": self.data, "format": self.format},
        }

to_content_part()

Render this audio as an OpenAI/litellm input_audio content part.

Source code in synalinks/src/backend/pydantic/media.py 
215
216
217
218
219
220
def to_content_part(self):
    """Render this audio as an OpenAI/litellm `input_audio` content part."""
    return {
        "type": "input_audio",
        "input_audio": {"data": self.data, "format": self.format},
    }

Image

Bases: DataModel

An image content part for a chat message.

Provide exactly one source: a url (http(s):// or a data: URI), a local file path, or raw base64 data (with its mime_type). Drop it into a ChatMessage's content list next to the accompanying text. A url/path is resolved to the actual payload at construction (see the module docstring).

Source code in synalinks/src/backend/pydantic/media.py 
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
@synalinks_export(
    [
        "synalinks.backend.Image",
        "synalinks.Image",
    ]
)
class Image(DataModel):
    """An image content part for a chat message.

    Provide exactly one source: a `url` (``http(s)://`` or a ``data:`` URI), a
    local file `path`, or raw base64 `data` (with its `mime_type`). Drop it
    into a `ChatMessage`'s content list next to the accompanying text. A
    `url`/`path` is resolved to the actual payload at construction (see the
    module docstring).
    """

    url: Optional[str] = Field(
        default=None,
        description="An http(s) URL or a `data:` URI pointing to the image.",
    )
    path: Optional[str] = Field(
        default=None,
        description="A local file path to the image; read at construction.",
    )
    data: Optional[str] = Field(
        default=None,
        description="Raw base64-encoded image bytes (used with `mime_type`).",
    )
    mime_type: Optional[str] = Field(
        default=None,
        description="The image MIME type, e.g. `image/png` (used with `data`).",
    )

    @model_validator(mode="after")
    def _inline_source(self):
        """Fetch/read a `url`/`path` source and inline it as base64 `data`.

        Runs once, at construction. A source already inline (base64 `data` or
        a ``data:`` URI in `url`) is left untouched.
        """
        if self.data is not None:
            return self
        read = _read_source(self.url, self.path)
        if read is not None:
            raw, mime = read
            self.data = base64.b64encode(raw).decode("ascii")
            if self.mime_type is None:
                self.mime_type = mime
        return self

    def to_content_part(self):
        """Render this image as an OpenAI/litellm `image_url` content part.

        A resolved image emits an inline base64 ``data:`` URI; a bare ``data:``
        URI `url` is passed through as-is.
        """
        if self.data:
            mime = self.mime_type or "image/png"
            return {
                "type": "image_url",
                "image_url": {"url": f"data:{mime};base64,{self.data}"},
            }
        return {"type": "image_url", "image_url": {"url": self.url}}

to_content_part()

Render this image as an OpenAI/litellm image_url content part.

A resolved image emits an inline base64 data: URI; a bare data: URI url is passed through as-is.

Source code in synalinks/src/backend/pydantic/media.py 
153
154
155
156
157
158
159
160
161
162
163
164
165
def to_content_part(self):
    """Render this image as an OpenAI/litellm `image_url` content part.

    A resolved image emits an inline base64 ``data:`` URI; a bare ``data:``
    URI `url` is passed through as-is.
    """
    if self.data:
        mime = self.mime_type or "image/png"
        return {
            "type": "image_url",
            "image_url": {"url": f"data:{mime};base64,{self.data}"},
        }
    return {"type": "image_url", "image_url": {"url": self.url}}

normalize_content(content)

Normalize a ChatMessage.content value to its chat-completion shape.

A plain string or a non-list value is returned unchanged; a list has each element mapped through _normalize_content_element.

Source code in synalinks/src/backend/pydantic/media.py 
238
239
240
241
242
243
244
245
246
def normalize_content(content):
    """Normalize a `ChatMessage.content` value to its chat-completion shape.

    A plain string or a non-list value is returned unchanged; a list has each
    element mapped through `_normalize_content_element`.
    """
    if isinstance(content, list):
        return [_normalize_content_element(e) for e in content]
    return content

place_tool_result_media(messages, native, vision, audio)

Put the images and audio of tool results where the provider takes them.

messages is a list of chat-completion wire dicts. A tool message whose content lists image_url / input_audio parts (a tool that returned media) keeps its images when the model takes images in a tool result (native: Claude, Gemini 3) and reads images at all (vision). Every other part is taken out of the tool result, which becomes plain text: media the model takes moves to a user message right after the run of tool messages (the only role that takes it on the OpenAI-style APIs, and the only one litellm maps audio from), media it does not take is dropped, with a note in the tool result either way.

Parameters:

Name Type Description Default
messages list

The chat-completion wire messages.

 required
native bool

Whether the model takes images in tool results.

 required
vision bool

Whether the model accepts images at all.

 required
audio bool

Whether the model accepts audio at all.

 required

Returns:

Type Description
list

The adapted messages; messages itself is left untouched.
Source code in synalinks/src/backend/pydantic/media.py 
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
def place_tool_result_media(messages, native, vision, audio):
    """Put the images and audio of tool results where the provider takes them.

    `messages` is a list of chat-completion wire dicts. A tool message whose
    content lists `image_url` / `input_audio` parts (a tool that returned
    media) keeps its images when the model takes images in a tool result
    (`native`: Claude, Gemini 3) and reads images at all (`vision`). Every
    other part is taken out of the tool result, which
    becomes plain text: media the model takes moves to a `user` message
    right after the run of tool messages (the only role that takes it on
    the OpenAI-style APIs, and the only one litellm maps audio from), media
    it does not take is dropped, with a note in the tool result either way.

    Args:
        messages (list): The chat-completion wire messages.
        native (bool): Whether the model takes images in tool results.
        vision (bool): Whether the model accepts images at all.
        audio (bool): Whether the model accepts audio at all.

    Returns:
        (list): The adapted messages; `messages` itself is left untouched.
    """
    adapted = []
    moved = []
    for message in messages:
        if moved and message.get("role") != "tool":
            adapted.append({"role": "user", "content": moved})
            moved = []
        content = message.get("content")
        if message.get("role") != "tool" or not isinstance(content, list):
            adapted.append(message)
            continue
        images = [part for part in content if part.get("type") == "image_url"]
        clips = [part for part in content if part.get("type") == "input_audio"]
        kept = images if native and vision else []
        if not clips and len(kept) == len(images):
            adapted.append(message)
            continue
        notes = []
        for parts, kind, accepted in (
            ([] if kept else images, "image", vision),
            (clips, "audio clip", audio),
        ):
            if not parts:
                continue
            if accepted:
                if not moved:
                    moved.append(
                        {
                            "type": "text",
                            "text": "The media returned by the tool calls above:",
                        }
                    )
                moved.extend(parts)
                notes.append(f"[{len(parts)} {kind}(s) attached in the next message]")
            else:
                notes.append(
                    f"[{len(parts)} {kind}(s) omitted: this model does not accept them]"
                )
        text = "".join(part.get("text", "") for part in content)
        text = "\n".join([text, *notes])
        adapted.append(
            {
                **message,
                "content": [{"type": "text", "text": text}, *kept] if kept else text,
            }
        )
    if moved:
        adapted.append({"role": "user", "content": moved})
    return adapted

resolve_content_media(messages) async

Resolve every deferred media reference in messages to an inline payload.

messages is a list of chat-completion wire dicts (one batch's worth). Any image_url part pointing at an http(s)/file:// source, and any input_audio part carrying a deferred url/path/marker source, is fetched/read and inlined as base64. Parts that are already inline (a data: URI or base64 data, e.g. content built from a constructed Image/Audio) are left untouched, so construction-resolved messages and text-only conversations pay nothing.

Source code in synalinks/src/backend/pydantic/media.py 
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
async def resolve_content_media(messages):
    """Resolve every deferred media reference in `messages` to an inline payload.

    `messages` is a list of chat-completion wire dicts (one batch's worth). Any
    `image_url` part pointing at an ``http(s)``/``file://`` source, and any
    `input_audio` part carrying a deferred `url`/`path`/marker source, is
    fetched/read and inlined as base64. Parts that are already inline (a
    ``data:`` URI or base64 `data`, e.g. content built from a constructed
    `Image`/`Audio`) are left untouched, so construction-resolved messages and
    text-only conversations pay nothing.
    """
    for message in messages:
        content = message.get("content")
        if not isinstance(content, list):
            continue
        for part in content:
            if not isinstance(part, dict):
                continue
            if part.get("type") == "image_url":
                await _resolve_image_part(part)
            elif part.get("type") == "input_audio":
                await _resolve_audio_part(part)
    return messages