diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..70565ef --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,101 @@ +# CLAUDE.md + +This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. + +## Build & Dev Commands +- `npm start` - Start dev server (opens at http://localhost:5173) +- `npm test` - Run all tests with Vitest +- `npm test src/path/to/file.test.ts` - Run specific test file +- `npm run build` - Build for production +- `npm run serve` - Preview production build + +## Project Overview +The Mongoose React Client is a MUD (Multi-User Dungeon) client built specifically for Project Mongoose. It connects to the game server via WebSocket and implements various MUD protocols including Telnet, GMCP (Generic MUD Communication Protocol), and MCP (MUD Client Protocol). + +## Architecture Overview + +### Core Services +- **MudClient (`src/client.ts`)**: Central service managing WebSocket connection, protocol handling, and feature integration +- **TelnetParser (`src/telnet.ts`)**: Handles low-level Telnet protocol negotiation and stream processing +- **WebRTCService (`src/WebRTCService.ts`)**: Manages voice/video chat via LiveKit +- **FileTransferManager (`src/FileTransferManager.ts`)**: Handles file uploads/downloads +- **EditorManager (`src/EditorManager.ts`)**: External editor integration +- **GMCP System** (`src/gmcp/`): Game protocol handlers for GMCP (Generic MUD Communication Protocol) packages +- **MCP System** (`src/mcp.ts`): MUD Client Protocol handlers for server communication +- **Services**: + - `MidiService.ts` & `VirtualMidiService.ts`: MIDI audio support + +### Protocol Support + +The client supports multiple MUD protocols: +- **GMCP**: For game data exchange (character stats, room info, etc.) +- **MCP**: For MUD Client Protocol features +- **MCMP**: With 3D audio support +- **Telnet**: Base protocol with ANSI color support + +### State Management +Uses custom store pattern (not Redux): +- **PreferencesStore**: User settings (volume, TTS, channels) +- **InputStore**: Command input state and management +- **FileTransferStore**: IndexedDB-backed file transfer persistence +- **useClientEvent.ts**: Hook for listening to client events + +### GMCP Protocol Structure +Located in `src/gmcp/`, organized hierarchically: +- Base class `GMCPPackage` provides common functionality +- Each package handles specific message types and emits events +- Key packages: Core, Auth, Char/*, Room, Comm/*, Client/*, IRE/* + +### Component Communication +- Server → TelnetParser → GMCP/MCP handlers → Event emission → UI components +- Components use custom hooks to subscribe to client events +- User input → CommandInput → MudClient → WebSocket → Server + +### Features + +- ANSI color rendering with `ansiParser.tsx` +- Text-to-speech with configurable voices +- Desktop notifications +- Session logging +- 3D audio via MCMP +- File transfer between users +- Real-time voice chat via LiveKit +- MIDI music support with virtual synthesizers +- In-game text editor with Monaco Editor + +### Key Files + +- `src/App.tsx`: Main application component +- `src/client.ts`: Core MudClient implementation +- `src/telnet.ts`: Telnet protocol handling +- `src/components/output.tsx`: Main game output window +- `src/components/input.tsx`: Command input component +- `src/components/sidebar.tsx`: Game info sidebar + +### Testing + +Uses Vitest with jsdom environment. Test files use `.test.ts` or `.test.tsx` extensions. + +## Code Style +- **Formatting**: 2-space indentation, CRLF line endings, UTF-8, trim trailing whitespace +- **Types**: Use explicit TypeScript types & interfaces +- **Components**: Functional components with hooks, props defined with interfaces +- **Naming**: + - PascalCase: React components & classes + - camelCase: variables, functions, instances + - UPPER_SNAKE_CASE: constants + - Handlers: prefix with "handle" (handleClick) + - Booleans: prefix with "is" or "has" (isConnected) +- **Imports**: React first, third-party next, local modules last, CSS imports last +- **Error Handling**: Try/catch blocks with console logging and fallback values +- **React Patterns**: Proper effect dependencies, useRef for DOM references, custom hooks +- **Testing**: Vitest with describe/it pattern, descriptive test names +- **Comments**: Avoid redundant comments that state the obvious (e.g., "// Click handler" above a handleClick function) + +## Key Dependencies +- React 18 with TypeScript +- Vite for build/dev server +- Monaco Editor for code editing +- LiveKit for WebRTC +- Cacophony for audio playback +- IndexedDB (via idb) for persistence \ No newline at end of file diff --git a/MIDI.md b/MIDI.md new file mode 100644 index 0000000..58bb451 --- /dev/null +++ b/MIDI.md @@ -0,0 +1,323 @@ +# MIDI System Documentation + +The Mongoose React Client includes a comprehensive MIDI system that enables real-time MIDI input/output, synthesizer engines, and seamless integration with the MUD's GMCP (Generic MUD Communication Protocol) MIDI features. + +## Table of Contents + +- [Architecture Overview](#architecture-overview) +- [MIDI Service](#midi-service) +- [User Interface Components](#user-interface-components) +- [Synthesizer Libraries](#synthesizer-libraries) +- [GMCP MIDI Integration](#gmcp-midi-integration) +- [Configuration and Preferences](#configuration-and-preferences) +- [Usage Examples](#usage-examples) +- [Troubleshooting](#troubleshooting) + +## Architecture Overview + +The MIDI system is built around several key components: + +``` +┌─────────────────┐ ┌──────────────────┐ ┌─────────────────┐ +│ MIDI Devices │────│ MidiService │────│ Synthesizers │ +│ (Hardware/ │ │ (Core Engine) │ │ (JZZ/MIDI.js) │ +│ Virtual) │ │ │ │ │ +└─────────────────┘ └──────────────────┘ └─────────────────┘ + │ │ │ + │ ┌────────────────────┐ │ + └──────────────│ GMCP MIDI │──────────┘ + │ Integration │ + └────────────────────┘ + │ + ┌────────────────────┐ + │ UI Components │ + │ (Preferences/ │ + │ Status/Sidebar) │ + └────────────────────┘ +``` + +## MIDI Service + +### Core Architecture (`src/MidiService.ts`) + +The `MidiService` class is the central hub for all MIDI functionality: + +#### Key Features: +- **Device Management**: Automatic discovery and connection to MIDI input/output devices +- **Multiple Synthesizers**: Supports JZZ Tiny and MIDI.js engines +- **Auto-reconnection**: Remembers and reconnects to previously used devices +- **Message Processing**: Real-time MIDI message parsing and routing +- **Event System**: Observable device changes and connection status + +#### Synthesizer Integration: + +```typescript +// Virtual synthesizers are initialized during service startup +async initializeVirtualSynths() { + // 1. JZZ Tiny Synthesizer (Basic built-in synthesis) + Tiny(JZZ); + JZZ.synth.Tiny.register('JZZ Tiny Synthesizer'); + + // 2. MIDI.js Synthesizer (Soundfont-based, configurable) + await this.initializeMIDIjsSynthesizer(); +} +``` + +#### Device State Management: + +```typescript +interface ConnectionState { + inputConnected: boolean; + outputConnected: boolean; + inputDeviceId?: string; + outputDeviceId?: string; + inputDeviceName?: string; + outputDeviceName?: string; +} +``` + +## User Interface Components + +### 1. MIDI Tab in Preferences (`src/components/preferences.tsx`) + +**Location**: Settings → MIDI Tab + +**Features**: +- Enable/disable MIDI functionality +- **MIDI.js Soundfont Selection**: + - FatBoy (Warm, analog-style sounds) + - FluidR3 GM (General MIDI standard) + - MusyngKite (High-quality, default selection) +- Real-time synthesizer reloading when preferences change + +**Code Structure**: +```typescript +const MidiTab: React.FC = () => { + const handleMidiJsSoundfontChange = async (e) => { + // Updates preferences and reloads MIDI.js synthesizer + await midiService.reloadSynthesizers(); + }; +}; +``` + +### 2. MIDI Tab in Sidebar + +**Location**: Sidebar → MIDI Tab (when connected to server) + +**Features**: +- **Input Device Selection**: Choose MIDI keyboard/controller +- **Output Device Selection**: Choose destination synthesizer +- **Connection Status**: Real-time connection indicators +- **Device Management**: Connect/disconnect controls +- **Auto-reconnection**: Attempts to reconnect to last used devices + +**Components**: +- Device dropdown lists (input/output) +- Connection status indicators +- Manual connect/disconnect buttons +- Device change notifications + +### 3. MIDI Status Component (`src/components/MidiStatus.tsx`) + +**Purpose**: Provides real-time MIDI activity monitoring and device status + +**Features**: +- **Connection Status**: Input/output device connection indicators +- **Message Activity**: Real-time MIDI message display +- **Device Change Notifications**: Alerts when devices are added/removed +- **Reconnection Status**: Shows auto-reconnection attempts +- **Message Filtering**: Displays relevant MIDI data (Note On/Off, CC, PC) + +**Message Types Tracked**: +```typescript +interface MidiMessage { + note?: { note: number; velocity: number; on: boolean; channel?: number }; + controlChange?: { controller: number; value: number; channel: number }; + programChange?: { program: number; channel: number }; + systemMessage?: { type: string; data: number[] }; + raw?: RawMidiMessage; + rawData: RawMidiMessage; // Always included for debugging +} +``` + +## Synthesizer Libraries + +### 1. JZZ (Jazz-MIDI) - Core MIDI Engine + +**Library**: `jzz` npm package +**Role**: Foundation MIDI engine providing cross-platform MIDI I/O + +**Why JZZ**: +- **Cross-browser compatibility**: Works in Chrome, Edge, Firefox, Safari +- **Unified API**: Handles both Web MIDI API and virtual devices seamlessly +- **Plugin Architecture**: Extensible with synthesizer plugins +- **Device Management**: Automatic device discovery and change detection + +**Integration**: +```javascript +// JZZ provides the foundation for all MIDI operations +const jzz = await JZZ(); +const inputDevice = await jzz.openMidiIn(deviceId); +const outputDevice = await jzz.openMidiOut(deviceId); +``` + +### 2. JZZ Tiny Synthesizer + +**Library**: `jzz-synth-tiny` npm package +**Role**: Lightweight built-in synthesizer for basic MIDI playback + +**Features**: +- **No external dependencies**: Pure JavaScript synthesis +- **Low latency**: Minimal processing overhead +- **GM Compatible**: Supports General MIDI instrument mapping +- **Small footprint**: ~50KB total size + +**Use Cases**: +- Basic MIDI playback when no other synthesizers are available +- Testing MIDI connections +- Low-resource environments + +### 3. MIDI.js with Soundfont Support + +**Libraries**: +- Custom `MIDI.js` (modified version) +- Custom `JZZ.synth.MIDIjs.js` bridge + +**Role**: High-quality soundfont-based synthesis with multiple soundfont options + +**Features**: +- **Multiple Soundfonts**: FatBoy, FluidR3 GM, MusyngKite +- **Dynamic Instrument Loading**: Loads instruments on-demand +- **High Audio Quality**: PCM-based samples for realistic sound +- **GM Compatibility**: Full General MIDI instrument support +- **Drum Routing**: Channel 10 drums routed to JZZ Tiny for performance + +**Soundfont Sources**: +```typescript +const soundfontUrls = { + 'FatBoy': 'https://mongoose.world/sounds/soundfont/FatBoy/', + 'FluidR3': 'https://mongoose.world/sounds/soundfont/FluidR3_GM/', + 'MusyngKite': 'https://mongoose.world/sounds/soundfont/MusyngKite/' +}; +``` + +**Why This Approach**: +- **User Choice**: Different soundfonts suit different musical styles +- **Resource Management**: Only one soundfont loaded at a time +- **Performance**: Configurable quality vs. resource usage + +## GMCP MIDI Integration + +### GMCP Package: `Client.Midi` (`src/gmcp/Client/Midi.ts`) + +The GMCP MIDI package provides server-to-client MIDI communication: + +**Supported GMCP Messages**: +- `Client.Midi.Play`: Play MIDI note/sequence +- `Client.Midi.Stop`: Stop current MIDI playback +- `Client.Midi.SetInstrument`: Change MIDI program/instrument +- `Client.Midi.ControlChange`: Send MIDI control change messages + +**Integration Flow**: +``` +Server GMCP → Client.Midi Handler → MidiService → Active Synthesizer → Audio Output +``` + +**Example GMCP Message Handling**: +```typescript +// Server sends: Client.Midi.Play {"note": 60, "velocity": 100, "channel": 0} +handlePlay(data: GMCPMessageClientMidiPlay) { + const midiMessage = [0x90 | data.channel, data.note, data.velocity]; + midiService.sendRawMessage(midiMessage); +} +``` + +**Bidirectional Communication**: +- **Server → Client**: GMCP messages trigger MIDI playback +- **Client → Server**: MIDI input can send GMCP messages back to server + +## Configuration and Preferences + +### Preference Structure (`src/PreferencesStore.tsx`) + +```typescript +export type MidiPreferences = { + enabled: boolean; // Master MIDI enable/disable + lastInputDeviceId?: string; // Remember last input device + lastOutputDeviceId?: string; // Remember last output device + midiJsSoundfont: string; // Selected MIDI.js soundfont +}; +``` + +### Default Settings: +- **MIDI Enabled**: `false` (must be explicitly enabled by user) +- **MIDI.js Soundfont**: `MusyngKite` (highest quality) +- **Auto-reconnect**: Enabled (remembers last used devices) + +### Preference Persistence: +- Stored in `localStorage` as JSON +- Automatically migrated when preference structure changes +- Changes trigger immediate synthesizer reloading + +## Usage Examples + +### Basic MIDI Setup: +1. **Enable MIDI**: Settings → MIDI → Enable MIDI ✓ +2. **Choose Soundfont**: Select preferred MIDI.js soundfont +3. **Connect**: Sidebar → MIDI → Select input/output devices +4. **Test**: Play notes on MIDI keyboard or via GMCP + +### GMCP Integration: +```javascript +// Server LPC code to send MIDI to client: +send_gmcp("Client.Midi.Play", ([ "note": 60, "velocity": 100, "channel": 0 ])); + +// Server receives MIDI input from client: +void gmcp_client_midi_input(mapping data) { + // Process MIDI input from client + write("You played note " + data["note"]); +} +``` + +## Troubleshooting + +### Common Issues: + +**1. No MIDI Devices Found** +- **Browser Support**: Ensure using Chrome, Edge, Opera, or Brave +- **MIDI Permission**: Browser may require user gesture to access MIDI +- **Device Connection**: Check physical MIDI device connections + +**2. No Sound from Synthesizers** +- **AudioContext Suspended**: Click anywhere on page to resume audio +- **Browser Audio Policy**: Some browsers block audio until user interaction +- **Soundfont Loading**: Check console for soundfont download errors + +**3. MIDI Input Not Detected** +- **Device Permissions**: Browser may prompt for MIDI access permission +- **Device Conflicts**: Another application may be using the MIDI device +- **USB Connection**: Try reconnecting USB MIDI devices + +### Debug Information: + +The MIDI system provides extensive console logging: +- `✅` Green checkmarks indicate successful operations +- `❌` Red X marks indicate errors +- Device connection/disconnection events +- MIDI message activity (when input connected) +- Soundfont loading progress + +### Performance Considerations: + +**MIDI.js Soundfonts**: +- **FatBoy**: ~15MB, warm analog sounds, higher CPU usage +- **FluidR3**: ~10MB, standard GM sounds, moderate CPU usage +- **MusyngKite**: ~25MB, highest quality, balanced performance + +**Recommendations**: +- Use **MusyngKite** for best balance of quality and performance +- Use **JZZ Tiny** for minimal resource usage + +--- + +*This documentation covers the MIDI system as of the current implementation. For the latest updates, check the source code in the respective component files.* \ No newline at end of file diff --git a/public/JZZ.synth.MIDIjs.js b/public/JZZ.synth.MIDIjs.js new file mode 100644 index 0000000..2f568c3 --- /dev/null +++ b/public/JZZ.synth.MIDIjs.js @@ -0,0 +1,201 @@ +(function() { + if (!JZZ) return; + if (!JZZ.synth) JZZ.synth = {}; + + function _name(name) { return name ? name : 'JZZ.synth.MIDIjs'; } + + var _waiting = false; + var _running = false; + var _bad = false; + var _error; + + // Dynamic loading state management + var _loadingInstruments = {}; // Track instruments currently being loaded + var _pendingProgramChanges = []; // Queue program changes waiting for load completion + + // Drum routing: Connect to JZZ.synth.Tiny for channel 10 (drums) + var _tinyPort = null; + + function _initTinyPort() { + if (!_tinyPort && JZZ.synth && JZZ.synth.Tiny) { + try { + _tinyPort = JZZ.synth.Tiny(); + console.log('JZZ.synth.Tiny port initialized for drum routing'); + } catch (e) { + console.warn('Failed to initialize JZZ.synth.Tiny port for drums:', e); + } + } + } + + function _receive(a) { + var s = a[0]>>4; + var c = a[0]&0xf; + + // Initialize Tiny port if needed + if (!_tinyPort) { + _initTinyPort(); + } + + // Route channel 10 (index 9) to JZZ.synth.Tiny for drums + if (c === 9 && _tinyPort) { + console.log('Routing channel 10 (drums) to JZZ.synth.Tiny:', a); + _tinyPort.send(a); + return; + } + + if (s == 0xC) { // Program Change (0xC0-0xCF) + var program = a[1]; + var instrumentName = MIDI.GM && MIDI.GM.byId && MIDI.GM.byId[program] ? MIDI.GM.byId[program].id : null; + + if (instrumentName && MIDI.Soundfont && !MIDI.Soundfont[instrumentName]) { + // Instrument not loaded + console.log('Program change to unloaded instrument:', instrumentName, 'program:', program); + + // Check if already loading this instrument + if (_loadingInstruments[instrumentName]) { + // Queue this program change for after loading completes + _pendingProgramChanges.push({channel: c, program: program, instrument: instrumentName}); + console.log('Queuing program change while loading:', instrumentName); + return; // Don't process original program change + } + + // Start loading process + _loadingInstruments[instrumentName] = true; + _pendingProgramChanges.push({channel: c, program: program, instrument: instrumentName}); + + // Immediate fallback to acoustic_grand_piano (program 0) + console.log('Falling back to piano while loading:', instrumentName); + if (MIDI.programChange) { + MIDI.programChange(c, 0); + } + + // Load the requested instrument + if (MIDI.loadResource) { + MIDI.loadResource({ + instruments: [instrumentName], + onsuccess: function() { + console.log('Successfully loaded instrument:', instrumentName); + + // Send deferred program changes for this instrument + var pending = _pendingProgramChanges.filter(function(p) { return p.instrument === instrumentName; }); + pending.forEach(function(p) { + console.log('Sending deferred program change:', p.instrument, 'program:', p.program); + if (MIDI.programChange) { + MIDI.programChange(p.channel, p.program); + } + }); + + // Clean up + delete _loadingInstruments[instrumentName]; + _pendingProgramChanges = _pendingProgramChanges.filter(function(p) { return p.instrument !== instrumentName; }); + }, + onerror: function(err) { + console.warn('Failed to load instrument:', instrumentName, 'error:', err, '- staying on piano'); + delete _loadingInstruments[instrumentName]; + _pendingProgramChanges = _pendingProgramChanges.filter(function(p) { return p.instrument !== instrumentName; }); + } + }); + } + + return; // Don't process original program change + } + + // Instrument already loaded or no MIDI.js available yet - pass through + if (MIDI.programChange) { + MIDI.programChange(c, program); + } + return; + } + + // Standard MIDI message processing + if (s == 0x8) { + if (MIDI.noteOff) { + MIDI.noteOff(c, a[1]); + } + } + else if (s == 0x9) { + if (MIDI.noteOn) { + MIDI.noteOn(c, a[1], a[2]); + } + } + } + + var _ports = []; + function _release(port, name) { + port._info = _engine._info(name); + port._receive = _receive; + port._resume(); + } + + function _onsuccess() { + _running = true; + _waiting = false; + + // Initialize Tiny port for drum routing + _initTinyPort(); + + for (var i=0; i<_ports.length; i++) _release(_ports[i][0], _ports[i][1]); + } + + function _onerror(evt) { + _bad = true; + _error = evt; + for (var i=0; i<_ports.length; i++) _ports[i][0]._crash(_error); + } + + var _engine = {}; + + _engine._info = function(name) { + return { + type: 'MIDI.js', + name: _name(name), + manufacturer: 'virtual', + version: '0.3.2' + }; + } + + _engine._openOut = function(port, name) { + if (_running) { + _release(port, name); + return; + } + if (_bad) { + port._crash(_error); + return; + } + port._pause(); + _ports.push([port, name]); + if (_waiting) return; + _waiting = true; + var arg = _engine._arg; + if (!arg) arg = {}; + arg.onsuccess = _onsuccess; + arg.onerror = _onerror; + try { + MIDI.loadPlugin(arg); + } + catch(e) { + _error = e.message; + _onerror(_error); + } + } + + JZZ.synth.MIDIjs = function() { + var name, arg; + if (arguments.length == 1) arg = arguments[0]; + else { name = arguments[0]; arg = arguments[1];} + name = _name(name); + if (!_running && !_waiting) _engine._arg = arg; + return JZZ.lib.openMidiOut(name, _engine); + } + + JZZ.synth.MIDIjs.register = function() { + var name, arg; + if (arguments.length == 1) arg = arguments[0]; + else { name = arguments[0]; arg = arguments[1];} + name = _name(name); + if (!_running && !_waiting) _engine._arg = arg; + return JZZ.lib.registerMidiOut(name, _engine); + } + +})(); \ No newline at end of file diff --git a/public/MIDI.js b/public/MIDI.js new file mode 100644 index 0000000..1e76bb7 --- /dev/null +++ b/public/MIDI.js @@ -0,0 +1,1928 @@ +/* + ---------------------------------------------------------- + MIDI.audioDetect : 0.3.2 : 2015-03-26 + ---------------------------------------------------------- + https://github.com/mudcube/MIDI.js + ---------------------------------------------------------- + Probably, Maybe, No... Absolutely! + Test to see what types of