Fix comprehensive system issues and implement proper vector database backend selection

- Fix reflection memory spam despite zero active characters in scheduler.py
- Add character enable/disable functionality to admin interface
- Fix Docker configuration with proper network setup and service dependencies
- Resolve admin interface JavaScript errors and login issues
- Fix MCP import paths for updated package structure
- Add comprehensive character management with audit logging
- Implement proper character state management and persistence
- Fix database connectivity and initialization issues
- Add missing audit service for admin operations
- Complete Docker stack integration with all required services

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>

REFACTORING_PROGRESS.md

@@ -0,0 +1,125 @@
+# Discord Fishbowl Refactoring Progress
+
+## Overview
+This document tracks the progress of refactoring efforts to improve security, performance, and maintainability of the Discord Fishbowl bot system.
+
+## High Priority Issues - Security & Performance
+
+### 🔴 Critical Security Issues
+- [ ] **Hardcoded Credentials** - Move all secrets to .env files
+  - [ ] Remove Discord tokens from config files
+  - [ ] Remove database passwords from configs
+  - [ ] Remove JWT secrets from source code
+  - [ ] Remove admin credentials from configs
+- [ ] **Input Validation** - Add validation to admin endpoints
+- [ ] **Client-side JWT** - Fix JWT verification issues
+- [ ] **Default Passwords** - Replace all weak defaults
+
+### 🟡 Performance Critical Issues
+- [ ] **Vector Store Blocking Operations** (`src/rag/vector_store.py:573-586`)
+  - [ ] Fix synchronous embedding generation
+  - [ ] Implement embedding caching
+  - [ ] Add batch processing for embeddings
+- [ ] **Database N+1 Queries** (`src/conversation/engine.py:399-402`)
+  - [ ] Fix character loading queries
+  - [ ] Add proper eager loading
+  - [ ] Optimize conversation retrieval
+- [ ] **Webhook Management** (`src/bot/discord_client.py:179-183`)
+  - [ ] Cache webhook lookups
+  - [ ] Implement webhook pooling
+  - [ ] Optimize webhook creation
+- [ ] **Missing Database Indexes** (`src/database/models.py`)
+  - [ ] Add indexes for foreign keys
+  - [ ] Add composite indexes for frequent queries
+  - [ ] Optimize query performance
+
+## Progress Tracking
+
+### Completed Tasks ✅
+- [x] Comprehensive code review and issue identification
+- [x] Created refactoring progress tracking system
+- [x] Fixed timezone-aware datetime issues in database models
+- [x] Fixed asyncio.Lock initialization issues in vector store
+- [x] Fixed blocking embedding generation in vector_store.py
+- [x] Added embedding caching to improve performance
+- [x] Optimized N+1 query pattern in conversation engine
+- [x] Added webhook caching in Discord client
+- [x] Added missing database index for cleanup queries
+- [x] Created .env.example template for secure deployment
+- [x] Fixed Discord channel ID configuration issue
+
+### In Progress 🔄
+- [ ] Moving hardcoded secrets to environment variables (keeping test values for now)
+
+### Pending ⏳
+- [ ] Update install.py to handle secrets properly
+- [ ] Add comprehensive input validation to admin endpoints
+- [ ] Implement proper error handling patterns
+- [ ] Add health check endpoints
+
+## File Status
+
+### Security Files
+| File | Status | Issues | Priority |
+|------|--------|--------|----------|
+| `config/fishbowl_config.json` | ❌ Needs Fix | Hardcoded tokens | Critical |
+| `.env.docker` | ❌ Needs Fix | Exposed secrets | Critical |
+| `src/admin/auth.py` | ❌ Needs Fix | Weak defaults | Critical |
+| `install.py` | ❌ Needs Update | Missing secret handling | High |
+
+### Performance Files
+| File | Status | Issues | Priority |
+|------|--------|--------|----------|
+| `src/rag/vector_store.py` | ✅ Fixed | Blocking operations | Critical |
+| `src/bot/discord_client.py` | ✅ Fixed | Inefficient webhooks | High |
+| `src/conversation/engine.py` | ✅ Fixed | N+1 queries | High |
+| `src/database/models.py` | ✅ Fixed | Missing indexes | High |
+
+### Code Quality Files
+| File | Status | Issues | Priority |
+|------|--------|--------|----------|
+| `src/mcp_servers/calendar_server.py` | ❌ Needs Refactor | High complexity | Medium |
+| `src/characters/enhanced_character.py` | ❌ Needs Refactor | God class | Medium |
+| Various files | ❌ Needs Fix | Error handling | Medium |
+
+## Metrics
+
+- **Total Critical Issues**: 8
+- **Issues Resolved**: 4 (Performance fixes)
+- **Issues In Progress**: 1
+- **Issues Pending**: 3
+- **Overall Progress**: 50% (4/8 completed)
+
+## Next Actions
+
+1. **Immediate (Today)**
+   - Move all hardcoded secrets to .env files
+   - Update install.py to handle secrets properly
+   - Fix blocking embedding generation
+
+2. **This Week**
+   - Add missing database indexes
+   - Fix N+1 query patterns
+   - Optimize webhook management
+
+3. **Next Week**
+   - Add comprehensive input validation
+   - Implement proper error handling
+   - Begin code complexity reduction
+
+## Notes
+
+- All security issues must be resolved before any production deployment
+- Performance issues directly impact user experience with slow LLM responses
+- Code quality improvements can be done incrementally alongside feature development
+- Testing should be added as each component is refactored
+
+## Estimated Timeline
+
+- **Security Fixes**: 2-3 days
+- **Performance Fixes**: 1 week  
+- **Code Quality**: 2-3 weeks (ongoing)
+- **Production Ready**: 4-6 weeks total
+
+---
+*Last Updated: 2025-07-06*