Skip to content

Knowledge Extraction and Storage

Open In Colab

Knowledge Extraction and Storage

Synalinks provides a powerful knowledge base system for extracting, storing, and retrieving structured knowledge. This example demonstrates extracting structured information from invoices and documents, storing them, and querying them later.

graph LR
    subgraph Extraction
        A[Document] --> B[Generator]
        B --> C[Structured Data]
    end
    subgraph Storage
        C --> D[UpdateKnowledge]
        D --> E[(KnowledgeBase)]
    end
    subgraph Retrieval
        F[Query] --> G[RetrieveKnowledge]
        E --> G
        G --> H[Results]
    end

Creating a Knowledge Base

The KnowledgeBase uses DuckDB as the underlying storage engine, providing full-text search and optional vector similarity search:

# Define your data model
class Invoice(synalinks.DataModel):
    invoice_number: str = synalinks.Field(description="Invoice number")
    vendor: str = synalinks.Field(description="Vendor name")
    total: float = synalinks.Field(description="Total amount")
    description: str = synalinks.Field(description="Description of items")

# Create a knowledge base
knowledge_base = synalinks.KnowledgeBase(
    uri="duckdb://./invoices.db",
    data_models=[Invoice],
    embedding_model=embedding_model,  # Optional, for similarity search
)

Extracting Information with Generator

Use a Generator to extract structured information from unstructured text:

inputs = synalinks.Input(data_model=DocumentText)
extracted = await synalinks.Generator(
    data_model=Invoice,
    language_model=language_model,
)(inputs)

Storing Data with UpdateKnowledge

The UpdateKnowledge module stores data models in the knowledge base:

stored = await synalinks.UpdateKnowledge(
    knowledge_base=knowledge_base,
)(extracted)

Retrieving Data with RetrieveKnowledge

The RetrieveKnowledge module uses hybrid search to find relevant records:

results = await synalinks.RetrieveKnowledge(
    knowledge_base=knowledge_base,
    language_model=language_model,
    search_type="hybrid",
    k=5,
)(query)

Key Takeaways

  • KnowledgeBase: Unified interface for storing and searching structured data using DuckDB with full-text and vector search capabilities.
  • UpdateKnowledge: Module for inserting/upserting data models into the knowledge base using the first field as primary key.
  • RetrieveKnowledge: Module for intelligent retrieval using LM-generated search queries with hybrid search (full-text + vector).
  • Structured Extraction: Use Generators to extract typed data from unstructured text like invoices, receipts, or documents.

Program Visualizations

Invoice Extraction Pipeline

invoice_extraction

Business Q&A System

business_qa

API References

Answer

Bases: DataModel

An answer based on retrieved information.

Source code in examples/12_knowledge_extraction_and_storage.py 
179
180
181
182
183
184
class Answer(synalinks.DataModel):
    """An answer based on retrieved information."""

    answer: str = synalinks.Field(
        description="The answer to the user's question based on retrieved data",
    )

Customer

Bases: DataModel

Extracted customer information.

Source code in examples/12_knowledge_extraction_and_storage.py 
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
class Customer(synalinks.DataModel):
    """Extracted customer information."""

    customer_id: str = synalinks.Field(
        description="Unique customer identifier",
    )
    name: str = synalinks.Field(
        description="Customer name (person or company)",
    )
    email: str = synalinks.Field(
        description="Customer email address",
    )
    description: str = synalinks.Field(
        description="Additional notes about the customer",
    )

DocumentText

Bases: DataModel

Raw document text to extract information from.

Source code in examples/12_knowledge_extraction_and_storage.py 
123
124
125
126
127
128
class DocumentText(synalinks.DataModel):
    """Raw document text to extract information from."""

    text: str = synalinks.Field(
        description="The raw text content of the document",
    )

Invoice

Bases: DataModel

Extracted invoice information.

Source code in examples/12_knowledge_extraction_and_storage.py 
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
class Invoice(synalinks.DataModel):
    """Extracted invoice information."""

    invoice_number: str = synalinks.Field(
        description="The unique invoice number or ID",
    )
    vendor: str = synalinks.Field(
        description="The name of the vendor or supplier",
    )
    date: str = synalinks.Field(
        description="The invoice date (YYYY-MM-DD format)",
    )
    total_amount: float = synalinks.Field(
        description="The total amount due",
    )
    currency: str = synalinks.Field(
        description="The currency (e.g., USD, EUR)",
    )
    description: str = synalinks.Field(
        description="A brief description of the invoice items or services",
    )

Query

Bases: DataModel

A user query for searching the knowledge base.

Source code in examples/12_knowledge_extraction_and_storage.py 
171
172
173
174
175
176
class Query(synalinks.DataModel):
    """A user query for searching the knowledge base."""

    query: str = synalinks.Field(
        description="The search query or question",
    )

Source

--8 < --"examples/12_knowledge_extraction_and_storage.py:110"

Run log

The log below is the unedited combined output of running the example above with local models (ollama).

Full run log: examples/12_knowledge_extraction_and_storage.log 
Example 1: Creating a Knowledge Base
==================================================
Knowledge base created at: ./examples/business_data.db
Tables: []

Example 2: Extracting and Storing Invoices
==================================================

Extracting invoices...
  - INV-2024-001: TechSupply Co. - 307.84 USD
  - INV-2024-002: Cloud Services Inc. - 149.0 EUR
  - INV-2024-003: Office Furniture Ltd. - 1127.0 USD

Example 3: Extracting and Storing Customers
==================================================

Extracting customers...
  - CUST-001: Acme Corporation (john.doe@acme.com)
  - CUST-002: Jane Smith (jane.smith@startup.io)

Example 4: Searching the Knowledge Base
==================================================

Search for 'cloud' in invoices:
  Found: {'invoice_number': 'INV-2024-002', 'vendor': 'Cloud Services Inc.', 'date': '2024-01-20', 'total_amount': 149.0, 'currency': 'EUR', 'description': 'Monthly subscription for cloud hosting services\n        - Basic Plan (January 2024)\n        - Storage: 500GB\n        - Bandwidth: Unlimited', 'embedding': None, 'score': 1.0}

Search for 'office equipment purchase':
  Found: {'invoice_number': 'INV-2024-003', 'vendor': 'Office Furniture Ltd.', 'date': '2024-01-02', 'total_amount': 1127.0, 'currency': 'USD', 'description': "{ \n    'Standing Desk - Adjustable Height': 599.0,\n    'Ergonomic Chair - Premium': 449.0,\n    'Desk Lamp - LED': 79.0\n}", 'embedding': None, 'score': 0.032266458495966696}
  Found: {'invoice_number': 'INV-2024-001', 'vendor': 'TechSupply Co.', 'date': '2024-01-15', 'total_amount': 307.84, 'currency': 'USD', 'description': '10x USB-C Cables @ $12.99 each\n- 5x Wireless Mouse @ $29.99 each', 'embedding': None, 'score': 0.01639344262295082}
  Found: {'invoice_number': 'INV-2024-002', 'vendor': 'Cloud Services Inc.', 'date': '2024-01-20', 'total_amount': 149.0, 'currency': 'EUR', 'description': 'Monthly subscription for cloud hosting services\n        - Basic Plan (January 2024)\n        - Storage: 500GB\n        - Bandwidth: Unlimited', 'embedding': None, 'score': 0.016129032258064516}

Example 5: Q&A System with RetrieveKnowledge
==================================================

Asking questions:

Q: What is the total amount of the invoice from TechSupply?
A: The total amount of the invoice from TechSupply is $307.84.

Q: Which invoice is for cloud services?
A: The invoice for cloud services is INV-2024-002.

Q: What is Jane Smith's email?
A: Jane Smith's email is jane.smith@startup.io.

Q: How much was the standing desk invoice?
A: The standing desk invoice was $1127.0 (USD). The invoice number is INV-2024-003.


Example 6: Listing All Stored Records
==================================================

Customer (2 records):
  - CUST-001: Acme Corporation
  - CUST-002: Jane Smith

Invoice (3 records):
  - INV-2024-001: TechSupply Co. - 307.84 USD
  - INV-2024-002: Cloud Services Inc. - 149.0 EUR
  - INV-2024-003: Office Furniture Ltd. - 1127.0 USD

Done!