feat: implement token counter with real-time breakdown
Add comprehensive token counting functionality to provide visibility into context usage: Backend (Rust): - Add tiktoken-rs dependency for OpenAI-compatible token counting - Implement get_token_count command with detailed breakdown - Count tokens for: system prompt, preset instructions, persona, world info, author's note, message history, and current input - Per-section token breakdown for optimization insights Frontend (JavaScript/HTML/CSS): - Add token counter widget in status bar - Real-time updates as user types (debounced 300ms) - Expandable breakdown tooltip showing per-section counts - Automatic update when chat history loads or changes - Clean, minimal UI with hover interactions Features: - Accurate token counting using cl100k_base tokenizer - Debounced updates for performance - Detailed breakdown by context section - Visual indicator with total token count - Click to expand/collapse detailed breakdown - Auto-hide when no character is active This completes the "Must-Have for Basic Roleplay" features from the roadmap: ✅ World Info/Lorebooks ✅ Author's Note ✅ Token Counter - Message Examples Usage (next) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
@@ -637,6 +637,53 @@
|
||||
</form>
|
||||
<div class="status-bar">
|
||||
<span id="status-text" class="status-text">Ready</span>
|
||||
<div id="token-counter" class="token-counter" style="display: none;">
|
||||
<span id="token-count-total" class="token-count">0 tokens</span>
|
||||
<button id="token-details-btn" class="token-details-btn" title="Show breakdown">
|
||||
<svg width="12" height="12" viewBox="0 0 16 16" fill="none">
|
||||
<circle cx="8" cy="8" r="6" stroke="currentColor" stroke-width="1.5"/>
|
||||
<path d="M8 7v4M8 5h.01" stroke="currentColor" stroke-width="1.5" stroke-linecap="round"/>
|
||||
</svg>
|
||||
</button>
|
||||
</div>
|
||||
</div>
|
||||
<!-- Token Breakdown Tooltip -->
|
||||
<div id="token-breakdown" class="token-breakdown" style="display: none;">
|
||||
<div class="token-breakdown-header">Token Breakdown</div>
|
||||
<div class="token-breakdown-list">
|
||||
<div class="token-breakdown-item">
|
||||
<span>System Prompt:</span>
|
||||
<span id="token-system">0</span>
|
||||
</div>
|
||||
<div class="token-breakdown-item">
|
||||
<span>Preset Instructions:</span>
|
||||
<span id="token-preset">0</span>
|
||||
</div>
|
||||
<div class="token-breakdown-item">
|
||||
<span>Persona:</span>
|
||||
<span id="token-persona">0</span>
|
||||
</div>
|
||||
<div class="token-breakdown-item">
|
||||
<span>World Info:</span>
|
||||
<span id="token-worldinfo">0</span>
|
||||
</div>
|
||||
<div class="token-breakdown-item">
|
||||
<span>Author's Note:</span>
|
||||
<span id="token-authorsnote">0</span>
|
||||
</div>
|
||||
<div class="token-breakdown-item">
|
||||
<span>Message History:</span>
|
||||
<span id="token-history">0</span>
|
||||
</div>
|
||||
<div class="token-breakdown-item">
|
||||
<span>Current Input:</span>
|
||||
<span id="token-input">0</span>
|
||||
</div>
|
||||
<div class="token-breakdown-total">
|
||||
<span>Total:</span>
|
||||
<span id="token-total-detail">0</span>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</footer>
|
||||
</div>
|
||||
|
||||
Reference in New Issue
Block a user