feat: implement token counter with real-time breakdown

Add comprehensive token counting functionality to provide visibility into
context usage:

Backend (Rust):
- Add tiktoken-rs dependency for OpenAI-compatible token counting
- Implement get_token_count command with detailed breakdown
- Count tokens for: system prompt, preset instructions, persona, world info,
  author's note, message history, and current input
- Per-section token breakdown for optimization insights

Frontend (JavaScript/HTML/CSS):
- Add token counter widget in status bar
- Real-time updates as user types (debounced 300ms)
- Expandable breakdown tooltip showing per-section counts
- Automatic update when chat history loads or changes
- Clean, minimal UI with hover interactions

Features:
- Accurate token counting using cl100k_base tokenizer
- Debounced updates for performance
- Detailed breakdown by context section
- Visual indicator with total token count
- Click to expand/collapse detailed breakdown
- Auto-hide when no character is active

This completes the "Must-Have for Basic Roleplay" features from the roadmap:
 World Info/Lorebooks
 Author's Note
 Token Counter
- Message Examples Usage (next)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
2025-10-16 13:24:49 -07:00
parent 828475ae4f
commit 2444ca0811
5 changed files with 332 additions and 0 deletions

View File

@@ -637,6 +637,53 @@
</form>
<div class="status-bar">
<span id="status-text" class="status-text">Ready</span>
<div id="token-counter" class="token-counter" style="display: none;">
<span id="token-count-total" class="token-count">0 tokens</span>
<button id="token-details-btn" class="token-details-btn" title="Show breakdown">
<svg width="12" height="12" viewBox="0 0 16 16" fill="none">
<circle cx="8" cy="8" r="6" stroke="currentColor" stroke-width="1.5"/>
<path d="M8 7v4M8 5h.01" stroke="currentColor" stroke-width="1.5" stroke-linecap="round"/>
</svg>
</button>
</div>
</div>
<!-- Token Breakdown Tooltip -->
<div id="token-breakdown" class="token-breakdown" style="display: none;">
<div class="token-breakdown-header">Token Breakdown</div>
<div class="token-breakdown-list">
<div class="token-breakdown-item">
<span>System Prompt:</span>
<span id="token-system">0</span>
</div>
<div class="token-breakdown-item">
<span>Preset Instructions:</span>
<span id="token-preset">0</span>
</div>
<div class="token-breakdown-item">
<span>Persona:</span>
<span id="token-persona">0</span>
</div>
<div class="token-breakdown-item">
<span>World Info:</span>
<span id="token-worldinfo">0</span>
</div>
<div class="token-breakdown-item">
<span>Author's Note:</span>
<span id="token-authorsnote">0</span>
</div>
<div class="token-breakdown-item">
<span>Message History:</span>
<span id="token-history">0</span>
</div>
<div class="token-breakdown-item">
<span>Current Input:</span>
<span id="token-input">0</span>
</div>
<div class="token-breakdown-total">
<span>Total:</span>
<span id="token-total-detail">0</span>
</div>
</div>
</div>
</footer>
</div>