deerflow-code/offline-backend-20260512/backend/docs/summarization.md

Conversation Summarization

DeerFlow includes automatic conversation summarization to handle long conversations that approach model token limits. When enabled, the system automatically condenses older messages while preserving recent context.

Overview

The summarization feature uses DeerFlow's DeerFlowSummarizationMiddleware (a subclass of LangChain's SummarizationMiddleware) to monitor conversation history and trigger summarization based on configurable thresholds. When activated, it:

1. Monitors message token counts in real-time
2. Triggers summarization when thresholds are met
3. Keeps recent messages intact while summarizing older exchanges
4. Maintains AI/Tool message pairs together for context continuity
5. Substitutes the summary for the older messages in the model request only

Non-destructive (display-preserving) compression

Important: DeerFlow compresses transiently, not destructively.

LangChain's stock SummarizationMiddleware runs in before_model and persists a RemoveMessage(REMOVE_ALL_MESSAGES) to the checkpoint — which permanently deletes the original Q&A from both the model context and the conversation the frontend renders. DeerFlow overrides this:

• before_model / abefore_model are no-ops (the destructive path is disabled).
• Compression runs in wrap_model_call / awrap_model_call: for a single model call, the message list is replaced with [summary, *recent] via request.override(messages=...). Nothing is written back to state.
• The full original messages stay in the checkpoint, so the frontend keeps showing the user's real conversation. The summary message is named summary and flagged hide_from_ui, and because it only ever lives in the transient request it never reaches the frontend at all.
• The generated summary is cached per-thread so we don't pay an extra LLM call on every model invocation. The cached summary is reused (no LLM call) until enough new messages accumulate to push the compressed context back over the trigger threshold (a sticky boundary mirroring the stock middleware's post-compaction behavior). The cache is bounded (oldest threads evicted first).
• Threshold-first: while the full current conversation is still under the trigger, the request is passed through untouched — no cache lookup, no summary substitution. Compression only ever engages once the real context crosses the configured trigger.
• Stickiness headroom: after keep picks a cutoff, the kept window is forced below ~75% of every trigger before summarizing (_apply_stickiness_headroom). This prevents the every-turn re-compaction that happens when keep is in messages (e.g. 20) but a trigger is in tokens (e.g. 60000) and the recent messages are large — without headroom the "compressed" context would still sit over the trigger and re-summarize on every follow-up question.
• When the genuine (blocking) summarize path actually runs an LLM call, the middleware emits a {"type":"context_compacting","message":"正在压缩上下文…"} event on LangGraph's custom stream channel so the chat UI can show a transient "正在压缩" toast (the cheap cache-reuse path stays silent). Frontend handler: onCustomEvent in frontend-web/src/core/threads/hooks.ts.

Legacy threads: conversations compacted by an older (destructive) build already lost their original messages and carry a persisted summary message. The frontend hides any name: "summary" message, so the stray summary bubble no longer shows — but those already-discarded messages cannot be recovered.

Configuration

Summarization is configured in config.yaml under the summarization key:

summarization:
  enabled: true
  model_name: null  # Use default model or specify a lightweight model

  # Trigger conditions (OR logic - any condition triggers summarization)
  trigger:
    - type: tokens
      value: 4000
    # Additional triggers (optional)
    # - type: messages
    #   value: 50
    # - type: fraction
    #   value: 0.8  # 80% of model's max input tokens

  # Context retention policy
  keep:
    type: messages
    value: 20

  # Token trimming for summarization call
  trim_tokens_to_summarize: 4000

  # Custom summary prompt (optional)
  summary_prompt: null

  # Tool names treated as skill file reads for skill rescue
  skill_file_read_tool_names:
    - read_file
    - read
    - view
    - cat

Configuration Options

enabled

• Type: Boolean
• Default: false
• Description: Enable or disable automatic summarization

model_name

• Type: String or null
• Default: null (uses default model)
• Description: Model to use for generating summaries. Recommended to use a lightweight, cost-effective model like gpt-4o-mini or equivalent.

trigger

• Type: Single ContextSize or list of ContextSize objects
• Required: At least one trigger must be specified when enabled
• Description: Thresholds that trigger summarization. Uses OR logic - summarization runs when ANY threshold is met.

ContextSize Types:

1. Token-based trigger: Activates when token count reaches the specified value

   trigger:
     type: tokens
     value: 4000
   

2. Message-based trigger: Activates when message count reaches the specified value

   trigger:
     type: messages
     value: 50
   

3. Fraction-based trigger: Activates when token usage reaches a percentage of the model's maximum input tokens

   trigger:
     type: fraction
     value: 0.8  # 80% of max input tokens
   

Multiple Triggers:

trigger:
  - type: tokens
    value: 4000
  - type: messages
    value: 50

keep

• Type: ContextSize object
• Default: {type: messages, value: 20}
• Description: Specifies how much recent conversation history to preserve after summarization.

Examples:

# Keep most recent 20 messages
keep:
  type: messages
  value: 20

# Keep most recent 3000 tokens
keep:
  type: tokens
  value: 3000

# Keep most recent 30% of model's max input tokens
keep:
  type: fraction
  value: 0.3

trim_tokens_to_summarize

• Type: Integer or null
• Default: 4000
• Description: Maximum tokens to include when preparing messages for the summarization call itself. Set to null to skip trimming (not recommended for very long conversations).

summary_prompt

• Type: String or null
• Default: null (uses LangChain's default prompt)
• Description: Custom prompt template for generating summaries. The prompt should guide the model to extract the most important context.

preserve_recent_skill_count

• Type: Integer (≥ 0)
• Default: 5
• Description: Number of most-recently-loaded skill files (tool results whose tool name is in skill_file_read_tool_names and whose target path is under skills.container_path, e.g. /mnt/skills/...) that are rescued from summarization. Prevents the agent from losing skill instructions after compression. Set to 0 to disable skill rescue entirely.

preserve_recent_skill_tokens

• Type: Integer (≥ 0)
• Default: 25000
• Description: Total token budget reserved for rescued skill reads. Once this budget is exhausted, older skill bundles are allowed to be summarized.

preserve_recent_skill_tokens_per_skill

• Type: Integer (≥ 0)
• Default: 5000
• Description: Per-skill token cap. Any individual skill read whose tool result exceeds this size is not rescued (it falls through to the summarizer like ordinary content).

skill_file_read_tool_names

• Type: List of strings
• Default: ["read_file", "read", "view", "cat"]
• Description: Tool names treated as skill file reads during summarization rescue. A tool call is only eligible for skill rescue when its name appears in this list and its target path is under skills.container_path.

Default Prompt Behavior: The default LangChain prompt instructs the model to:

• Extract highest quality/most relevant context
• Focus on information critical to the overall goal
• Avoid repeating completed actions
• Return only the extracted context

How It Works

Summarization Flow

1. Monitoring: Before each model call, the middleware counts tokens in the message history
2. Trigger Check: If any configured threshold is met, summarization is triggered
3. Message Partitioning: Messages are split into:
   • Messages to summarize (older messages beyond the keep threshold)
   • Messages to preserve (recent messages within the keep threshold)
4. Summary Generation: The model generates a concise summary of the older messages
5. Context Replacement (transient): The message list handed to the model for this call is rebuilt as [summary, *recent]. The persisted state is left untouched — old messages are not removed, so the frontend still renders the full original conversation.
6. AI/Tool Pair Protection: The system ensures AI messages and their corresponding tool messages stay together
7. Skill Rescue: Before the summary is generated, the most recently loaded skill files (tool results whose tool name is in skill_file_read_tool_names and whose target path is under skills.container_path) are lifted out of the summarization set and prepended to the preserved tail. Selection walks newest-first under three budgets: preserve_recent_skill_count, preserve_recent_skill_tokens, and preserve_recent_skill_tokens_per_skill. The triggering AIMessage and all of its paired ToolMessages move together so tool_call ↔ tool_result pairing stays intact.

Token Counting

• Uses approximate token counting based on character count
• For Anthropic models: ~3.3 characters per token
• For other models: Uses LangChain's default estimation
• Can be customized with a custom token_counter function

Message Preservation

The middleware intelligently preserves message context:

• Recent Messages: Always kept intact based on keep configuration
• AI/Tool Pairs: Never split - if a cutoff point falls within tool messages, the system adjusts to keep the entire AI + Tool message sequence together
• Summary Format: Summary is injected as a HumanMessage with the format:

  Here is a summary of the conversation to date:
  
  [Generated summary text]
  

Best Practices

Choosing Trigger Thresholds

1. Token-based triggers: Recommended for most use cases

   • Set to 60-80% of your model's context window
   • Example: For 8K context, use 4000-6000 tokens

2. Message-based triggers: Useful for controlling conversation length

   • Good for applications with many short messages
   • Example: 50-100 messages depending on average message length

3. Fraction-based triggers: Ideal when using multiple models

   • Automatically adapts to each model's capacity
   • Example: 0.8 (80% of model's max input tokens)

Choosing Retention Policy (keep)

1. Message-based retention: Best for most scenarios

   • Preserves natural conversation flow
   • Recommended: 15-25 messages

2. Token-based retention: Use when precise control is needed

   • Good for managing exact token budgets
   • Recommended: 2000-4000 tokens

3. Fraction-based retention: For multi-model setups

   • Automatically scales with model capacity
   • Recommended: 0.2-0.4 (20-40% of max input)

Model Selection

• Recommended: Use a lightweight, cost-effective model for summaries

  • Examples: gpt-4o-mini, claude-haiku, or equivalent
  • Summaries don't require the most powerful models
  • Significant cost savings on high-volume applications

• Default: If model_name is null, uses the default model

  • May be more expensive but ensures consistency
  • Good for simple setups

Optimization Tips

1. Balance triggers: Combine token and message triggers for robust handling

   trigger:
     - type: tokens
       value: 4000
     - type: messages
       value: 50
   

2. Conservative retention: Keep more messages initially, adjust based on performance

   keep:
     type: messages
     value: 25  # Start higher, reduce if needed
   

3. Trim strategically: Limit tokens sent to summarization model

   trim_tokens_to_summarize: 4000  # Prevents expensive summarization calls
   

4. Monitor and iterate: Track summary quality and adjust configuration

Troubleshooting

Summary Quality Issues

Problem: Summaries losing important context

Solutions:

1. Increase keep value to preserve more messages
2. Decrease trigger thresholds to summarize earlier
3. Customize summary_prompt to emphasize key information
4. Use a more capable model for summarization

Performance Issues

Problem: Summarization calls taking too long

Solutions:

1. Use a faster model for summaries (e.g., gpt-4o-mini)
2. Reduce trim_tokens_to_summarize to send less context
3. Increase trigger thresholds to summarize less frequently

Token Limit Errors

Problem: Still hitting token limits despite summarization

Solutions:

1. Lower trigger thresholds to summarize earlier
2. Reduce keep value to preserve fewer messages
3. Check if individual messages are very large
4. Consider using fraction-based triggers

Implementation Details

Code Structure

• Configuration: packages/harness/deerflow/config/summarization_config.py
• Integration: packages/harness/deerflow/agents/lead_agent/agent.py (_create_summarization_middleware)
• Middleware: packages/harness/deerflow/agents/middlewares/summarization_middleware.py — DeerFlowSummarizationMiddleware, subclassing langchain.agents.middleware.SummarizationMiddleware but overriding compaction to be transient (wrap_model_call) instead of destructive (before_model)
• Tests: tests/test_summarization_middleware.py

Middleware Order

Summarization runs after ThreadData and Sandbox initialization but before Title and Clarification:

1. ThreadDataMiddleware
2. SandboxMiddleware
3. SummarizationMiddleware ← Runs here
4. TitleMiddleware
5. ClarificationMiddleware

State Management

• Configuration is loaded once at startup
• Summarization is non-destructive: the summary replaces older messages only in the transient model request (wrap_model_call), never in persisted state. The checkpointer therefore keeps the full original conversation
• The per-thread summary cache lives in a process-wide, thread-safe LRU (_GLOBAL_SUMMARY_CACHE, bounded), not on the middleware instance. make_lead_agent rebuilds a fresh middleware every run, so an instance cache would be cold each turn and re-run a blocking summary LLM call on every turn of a long conversation; the module-level cache keyed by thread_id lets a summary survive across turns. It is a cost optimization only — a cold/stale cache simply re-summarizes, which is always correct

Example Configurations

Minimal Configuration

summarization:
  enabled: true
  trigger:
    type: tokens
    value: 4000
  keep:
    type: messages
    value: 20

Production Configuration

summarization:
  enabled: true
  model_name: gpt-4o-mini  # Lightweight model for cost efficiency
  trigger:
    - type: tokens
      value: 6000
    - type: messages
      value: 75
  keep:
    type: messages
    value: 25
  trim_tokens_to_summarize: 5000

Multi-Model Configuration

summarization:
  enabled: true
  model_name: gpt-4o-mini
  trigger:
    type: fraction
    value: 0.7  # 70% of model's max input
  keep:
    type: fraction
    value: 0.3  # Keep 30% of max input
  trim_tokens_to_summarize: 4000

Conservative Configuration (High Quality)

summarization:
  enabled: true
  model_name: gpt-4  # Use full model for high-quality summaries
  trigger:
    type: tokens
    value: 8000
  keep:
    type: messages
    value: 40  # Keep more context
  trim_tokens_to_summarize: null  # No trimming

References

• LangChain Summarization Middleware Documentation
• LangChain Source Code