Why AI Models Consist of Multiple Tensors

This document explains why a .safetensors file representing an AI model is composed of many distinct tensors rather than one giant tensor.

---

The Core Reason: Neural Networks Are Graphs of Operations

A neural network isn't a single mathematical operation—it's a directed graph of many distinct operations, each with its own learned parameters.

graph LR
    subgraph "Simple Neural Network"
        I[Input] --> L1["Layer 1<br/>weights + bias"]
        L1 --> A1[Activation]
        A1 --> L2["Layer 2<br/>weights + bias"]
        L2 --> A2[Activation]
        A2 --> L3["Layer 3<br/>weights + bias"]
        L3 --> O[Output]
    end

    style L1 fill:#3498db,color:#fff
    style L2 fill:#3498db,color:#fff
    style L3 fill:#3498db,color:#fff

Each blue box represents separate tensors with different shapes and purposes.

---

Reason 1: Different Shapes for Different Operations

Each layer performs a specific mathematical operation requiring a specific tensor shape.

graph TB
    subgraph "Transformer Block - Different Tensor Shapes"
        EMB["Embedding<br/>[vocab_size × hidden_dim]<br/>50257 × 768"]

        QKV["Q, K, V Projections<br/>[hidden × hidden] each<br/>768 × 768"]

        ATT_OUT["Attention Output<br/>[hidden × hidden]<br/>768 × 768"]

        FFN1["FFN Layer 1<br/>[hidden × 4×hidden]<br/>768 × 3072"]

        FFN2["FFN Layer 2<br/>[4×hidden × hidden]<br/>3072 × 768"]

        LN["Layer Norm<br/>[hidden]<br/>768"]
    end

    EMB --> QKV --> ATT_OUT --> FFN1 --> FFN2
    LN -.-> |"Applied at multiple points"| QKV

    style EMB fill:#e74c3c,color:#fff
    style QKV fill:#3498db,color:#fff
    style FFN1 fill:#27ae60,color:#fff
    style FFN2 fill:#27ae60,color:#fff
    style LN fill:#9b59b6,color:#fff

You cannot combine a [50257 × 768] tensor with a [768 × 768] tensor into one—they have incompatible shapes and serve different purposes.

---

Reason 2: Weights vs. Biases

Even within a single layer, weights and biases are separate tensors:

graph LR
    subgraph "Single Linear Layer: y = Wx + b"
        X["Input x<br/>[batch, 768]"]
        W["Weight W<br/>[768, 768]<br/>2D tensor"]
        B["Bias b<br/>[768]<br/>1D tensor"]
        Y["Output y<br/>[batch, 768]"]

        X --> |"matrix multiply"| MUL["×"]
        W --> MUL
        MUL --> ADD["+"]
        B --> ADD
        ADD --> Y
    end

    style W fill:#3498db,color:#fff
    style B fill:#e74c3c,color:#fff

The weight is a 2D matrix, the bias is a 1D vector. Different shapes = different tensors.

---

Reason 3: Modularity and Flexibility

Separate tensors enable:

Capability	How Separate Tensors Help
Fine-tuning	Freeze some layers, train others
Transfer learning	Replace just the final layer
LoRA adapters	Add small tensors alongside existing ones
Pruning	Remove individual layers or heads
Quantization	Apply different precision to different layers
Inspection	Analyze specific layer behaviors

graph TB
    subgraph "Fine-Tuning Example"
        direction TB
        FROZEN["Frozen Layers<br/>(loaded, not trained)"]
        TRAINABLE["New Classification Head<br/>(randomly initialized, trained)"]

        FROZEN --> TRAINABLE
    end

    style FROZEN fill:#95a5a6,color:#fff
    style TRAINABLE fill:#27ae60,color:#fff

If everything were one tensor, you couldn't selectively freeze or modify parts.

---

Reason 4: Memory Efficiency During Computation

During inference, not all tensors are needed simultaneously:

sequenceDiagram
    participant RAM as System RAM
    participant GPU as GPU Memory
    participant Compute as Computation

    Note over RAM: All tensors stored

    RAM->>GPU: Load Layer 1 weights
    GPU->>Compute: Process Layer 1
    GPU-->>RAM: Optionally offload

    RAM->>GPU: Load Layer 2 weights
    GPU->>Compute: Process Layer 2
    GPU-->>RAM: Optionally offload

    Note over GPU: Only active layer<br/>needs GPU memory

Separate tensors allow layer-by-layer streaming for models larger than GPU memory.

---

Reason 5: A Real Model's Tensor Inventory

A typical transformer model contains these tensor categories:

graph TB
    subgraph "GPT-2 Style Model Tensors"
        subgraph "Embeddings"
            WTE["wte.weight<br/>Token embeddings<br/>[50257, 768]"]
            WPE["wpe.weight<br/>Position embeddings<br/>[1024, 768]"]
        end

        subgraph "Per-Layer Tensors (×12 layers)"
            LN1["ln_1.weight, ln_1.bias<br/>[768] each"]
            ATT["attn.c_attn.weight<br/>[768, 2304]"]
            ATTB["attn.c_attn.bias<br/>[2304]"]
            PROJ["attn.c_proj.weight<br/>[768, 768]"]
            LN2["ln_2.weight, ln_2.bias<br/>[768] each"]
            MLP1["mlp.c_fc.weight<br/>[768, 3072]"]
            MLP2["mlp.c_proj.weight<br/>[3072, 768]"]
        end

        subgraph "Final"
            LNF["ln_f.weight, ln_f.bias<br/>[768] each"]
        end
    end

    style WTE fill:#e74c3c,color:#fff
    style WPE fill:#e74c3c,color:#fff
    style ATT fill:#3498db,color:#fff
    style MLP1 fill:#27ae60,color:#fff
    style MLP2 fill:#27ae60,color:#fff

GPT-2 Small has 148 separate tensors. GPT-2 XL has 292 tensors. LLaMA 70B has thousands.

---

Why Not Flatten Everything Into One Tensor?

You could technically concatenate all parameters into a single 1D tensor:

graph LR
    subgraph "Hypothetical Single Tensor"
        FLAT["[param_0, param_1, param_2, ..., param_N]<br/>One giant 1D array"]
    end

    subgraph "Problems"
        P1["❌ Lose shape information"]
        P2["❌ Can't do matrix operations directly"]
        P3["❌ Must reshape constantly"]
        P4["❌ Can't load partially"]
        P5["❌ Can't apply different dtypes"]
    end

    FLAT --> P1
    FLAT --> P2
    FLAT --> P3
    FLAT --> P4
    FLAT --> P5

    style FLAT fill:#e74c3c,color:#fff

You'd need a separate metadata structure to track where each layer's parameters begin and end, what shape to reshape them into, etc.—essentially recreating what safetensors already does, but less efficiently.

---

The Mathematical Perspective

A neural network computes a function by composing many smaller functions:

output = f_n( f_{n-1}( ... f_2( f_1( input ) ) ... ) )

Each function f_i typically has the form:

f_i(x) = activation( W_i · x + b_i )

Where: - W_i is a weight matrix (2D tensor) - b_i is a bias vector (1D tensor) - Each layer has its own W and b

This mathematical structure inherently requires multiple separate parameter tensors.

---

Analogy: A Car vs. Its Parts

Think of asking "why does a car have multiple parts instead of one?"

graph TB
    subgraph "A Car's Components"
        ENGINE["Engine<br/>(converts fuel to motion)"]
        TRANS["Transmission<br/>(transfers power)"]
        WHEELS["Wheels<br/>(contact ground)"]
        STEERING["Steering<br/>(direction control)"]
        BRAKES["Brakes<br/>(stopping)"]
    end

    ENGINE --> TRANS --> WHEELS
    STEERING --> WHEELS
    BRAKES --> WHEELS

    style ENGINE fill:#e74c3c,color:#fff
    style TRANS fill:#3498db,color:#fff
    style WHEELS fill:#27ae60,color:#fff

Each component: - Has a different shape (you can't combine an engine and a wheel) - Serves a different function - Can be replaced independently - Must be connected in a specific way

Neural network tensors are the same—each serves a specific purpose in the computation.

---

Summary

Reason	Explanation
Mathematical necessity	Different operations require different tensor shapes
Architectural structure	Networks are graphs of distinct operations
Weights vs. biases	Even one layer has multiple parameter tensors
Modularity	Enables fine-tuning, pruning, quantization
Memory management	Allows streaming and partial loading
Framework design	PyTorch/TensorFlow organize models as named parameter collections

---

Conclusion

The multiple-tensor design directly mirrors the mathematical structure of neural networks. Each tensor represents a learnable parameter matrix or vector for a specific operation in the computation graph.

A single-tensor design would: 1. Lose the natural correspondence between tensors and operations 2. Require constant reshaping during computation 3. Prevent partial loading and fine-tuning 4. Make the model file format more complex, not simpler

The safetensors format (and all model formats) use multiple named tensors because that's what neural networks fundamentally are: collections of distinct, shaped parameter arrays connected in a computation graph.