Flutter Semantics MCP Server

Enable AI coding assistants to "see" and understand your running Flutter applications through the Model Context Protocol (MCP). Eliminate manual screenshots and UI descriptions when debugging with AI.

What is this?

This package embeds an MCP server inside your Flutter app, allowing AI assistants like Claude Code to:

• See your UI: Query the complete Flutter semantics tree to understand what's on screen
• Debug errors: Access comprehensive error tracking with full context (stack traces, widget paths, user actions)
• Ask questions: "Why is this button disabled?", "What errors occurred?", "What's currently visible?"

All data stays local—no network transmission, auto-disables in release builds.

Features

• Semantics Tree Access: AI can query the complete Flutter UI hierarchy
• Comprehensive Error Tracking: Automatic capture with stack traces, widget context, semantics snapshots, and user action history
• MCP Protocol Integration: Standard protocol for AI tool communication
• Privacy-First: Auto-disables in release builds, privacy-conscious data capture
• Easy Setup: ≤5 lines of code to get started
• Platform Support: macOS and iOS (simulator and physical devices)

Quick Start (< 10 minutes)

Prerequisites

• Flutter ≥3.24.0
• Dart ≥3.0.0
• macOS 13+ or iOS 15+ (MVP platforms)
• Claude Code or another MCP-compatible AI assistant

Installation

Add the package to your Flutter project:

flutter pub add mcp_flutter_semantics

Initialize the Server

Add these lines to your main.dart:

import 'package:flutter/material.dart';
import 'package:mcp_flutter_semantics/mcp_flutter_semantics.dart';

void main() async {
  WidgetsFlutterBinding.ensureInitialized();

  // Start MCP server (auto-disabled in release builds)
  await McpSemanticsServer.instance.start();

  runApp(MyApp());
}

That's it! The MCP server is now running inside your Flutter app.

Configure Claude Code

Follow the Setup Guide to connect Claude Code to your app.

Test the Connection

Run your Flutter app, then ask Claude Code:

• "What's currently visible on screen?"
• "Why is this button disabled?"
• "What errors have occurred?"
• "Show me the last user interaction"

Making Your App AI-Controllable

To enable AI assistants to interact with your app, wrap your widgets with MCP-enabled widgets. The package provides drop-in replacements for common Flutter widgets that expose rich metadata to AI.

MCP Widget Wrappers

McpButton - Interactive Buttons

Wrap any button-like widget to make it AI-tappable:

McpButton(
  id: 'submit-button',              // Unique ID for AI to reference
  label: 'Submit Form',             // Human-readable label
  hint: 'Tap to submit the form',   // Help text for AI
  onPressed: _handleSubmit,         // Your callback
  enabled: _formIsValid,            // Dynamic enabled state
  metadata: {                       // Custom metadata for AI
    'formValid': _formIsValid,
    'disabledReasons': _getDisabledReasons(),
  },
  child: const Text('Submit'),      // Your original button
)

AI can:

• Query button state: widgets://tree shows if enabled/disabled
• Read metadata: See why button is disabled
• Trigger tap: trigger_action("submit-button", "tap")

McpTextField - Text Input Fields

Make text fields AI-fillable:

McpTextField(
  id: 'email-field',
  label: 'Email Address',
  hint: 'Enter your email',
  controller: _emailController,
  enabled: true,
  metadata: {
    'required': true,
    'isValid': _emailController.text.contains('@'),
    'validationMessage': _getEmailError(),
  },
  decoration: const InputDecoration(
    labelText: 'Email',
    hintText: 'you@example.com',
  ),
  validator: (value) => value?.contains('@') == true
      ? null
      : 'Invalid email',
  onChanged: (_) => setState(() {}),
)

AI can:

• Fill field: trigger_action("email-field", "set_text", "john@example.com")
• Clear field: trigger_action("email-field", "clear")
• Read value: Check metadata for current value and validation state

McpDropdown - Selection Dropdowns

McpDropdown<String>(
  id: 'country-dropdown',
  label: 'Country',
  hint: 'Select your country',
  value: _selectedCountry,
  items: const [
    DropdownMenuItem(value: 'US', child: Text('United States')),
    DropdownMenuItem(value: 'UK', child: Text('United Kingdom')),
    DropdownMenuItem(value: 'CA', child: Text('Canada')),
  ],
  onChanged: (value) => setState(() => _selectedCountry = value),
  metadata: {
    'required': false,
    'options': ['US', 'UK', 'CA'],  // AI sees available options
  },
)

AI can:

• Select option: trigger_action("country-dropdown", "select", "UK")
• See options: Metadata includes all available values

McpCheckbox - Checkboxes

McpCheckbox(
  id: 'terms-checkbox',
  label: 'I accept terms and conditions',
  value: _acceptedTerms,
  enabled: true,
  metadata: {'required': true},
  title: const Text('I accept the terms and conditions'),
  onChanged: (value) => setState(() => _acceptedTerms = value ?? false),
)

AI can:

• Check: trigger_action("terms-checkbox", "check")
• Uncheck: trigger_action("terms-checkbox", "uncheck")
• Toggle: trigger_action("terms-checkbox", "toggle")

McpSlider - Value Sliders

McpSlider(
  id: 'satisfaction-slider',
  label: 'Satisfaction Level',
  hint: 'Rate from 0 to 10',
  value: _satisfaction,
  min: 0,
  max: 10,
  divisions: 10,
  onChanged: (value) => setState(() => _satisfaction = value),
  metadata: {
    'unit': 'rating',
    'currentValue': _satisfaction,
  },
)

AI can:

• Set value: trigger_action("satisfaction-slider", "set_value", "8.0")
• Increase: trigger_action("satisfaction-slider", "increase")
• Decrease: trigger_action("satisfaction-slider", "decrease")

Navigation Tracking

Register routes to help AI navigate your app:

void main() async {
  WidgetsFlutterBinding.ensureInitialized();

  // Register routes with rich metadata
  McpNavigationMap.instance.registerRoute(
    path: '/profile',
    name: 'User Profile',
    description: 'View and edit user profile information',
    metadata: {
      'purpose': 'Manage user account details',
      'commonActions': ['Edit name', 'Change avatar', 'Update email'],
      'prerequisites': 'User must be logged in',
    },
  );

  await McpSemanticsServer.instance.start();
  runApp(MyApp());
}

Add navigation observers to your MaterialApp:

MaterialApp(
  navigatorObservers: [
    McpNavigationObserver(sanitizeParams: true),
    McpRoutingObserver(logger: debugPrint),
  ],
  // ... rest of your app
)

AI can:

• Query sitemap: navigation://sitemap shows all routes
• Navigate: navigate_to("/profile")
• Go back: go_back()

Form Validation with Healing Hints

Help AI understand and auto-fix form validation issues:

McpButton(
  id: 'submit-button',
  label: 'Submit',
  onPressed: _isValid ? _handleSubmit : null,
  enabled: _isValid,
  metadata: {
    'disabledReasons': _getDisabledReasons(),  // ["Name is empty", "Email invalid"]

    // Healing hints tell AI how to fix issues
    'healingHints': {
      'canAutoHeal': true,
      'autoHealableFields': ['name-field', 'email-field'],
      'suggestedAction': 'Fill name and email fields',
    },

    // Field status shows current validation state
    'fieldStatus': {
      'name': {
        'id': 'name-field',
        'valid': false,
        'accessible': true,
        'validationMessage': 'Name is required',
      },
      'email': {
        'id': 'email-field',
        'valid': false,
        'accessible': true,
        'validationMessage': 'Must contain @',
      },
    },

    'completionPercentage': 0,  // Progress tracking
  },
  child: const Text('Submit'),
)

AI can:

• See why button is disabled
• Know which fields need fixing
• Auto-fill form intelligently
• Track completion progress

Best Practices

1. Use Unique, Descriptive IDs

✅ GOOD: id: 'user-profile-save-button'
❌ BAD:  id: 'button1'

2. Provide Rich Metadata

McpTextField(
  id: 'age-field',
  label: 'Age',
  metadata: {
    'required': true,
    'minValue': 18,
    'maxValue': 120,
    'validationMessage': _ageError,
    'isEmpty': _ageController.text.isEmpty,
    'isValid': _isAgeValid(),
  },
  // ...
)

3. Keep Metadata Updated

Use setState() to update metadata when widget state changes:

onChanged: (_) => setState(() {}),  // Triggers rebuild with fresh metadata

4. Combine with Semantics

You can use both MCP widgets AND Flutter's Semantics for maximum accessibility:

Semantics(
  label: 'Submit form button',
  hint: 'Tap to submit your information',
  enabled: _formIsValid,
  button: true,
  child: McpButton(
    id: 'submit-button',
    label: 'Submit',
    onPressed: _formIsValid ? _handleSubmit : null,
    // ...
  ),
)

Complete Example

Here's a complete form that's fully AI-controllable:

class MyForm extends StatefulWidget {
  @override
  State<MyForm> createState() => _MyFormState();
}

class _MyFormState extends State<MyForm> {
  final _nameController = TextEditingController();
  final _emailController = TextEditingController();
  bool _acceptedTerms = false;

  bool get _isValid =>
      _nameController.text.isNotEmpty &&
      _emailController.text.contains('@') &&
      _acceptedTerms;

  @override
  Widget build(BuildContext context) {
    return Column(
      children: [
        // Progress indicator
        McpWidget(
          id: 'form-progress',
          type: 'ProgressIndicator',
          label: 'Form Progress',
          metadata: {
            'percentage': _calculateProgress(),
            'fieldsRemaining': _getRemainingFields(),
          },
          child: LinearProgressIndicator(
            value: _calculateProgress() / 100,
          ),
        ),

        // Name field
        McpTextField(
          id: 'name-field',
          label: 'Full Name',
          controller: _nameController,
          metadata: {
            'required': true,
            'isEmpty': _nameController.text.isEmpty,
          },
          onChanged: (_) => setState(() {}),
        ),

        // Email field
        McpTextField(
          id: 'email-field',
          label: 'Email',
          controller: _emailController,
          metadata: {
            'required': true,
            'isValid': _emailController.text.contains('@'),
          },
          onChanged: (_) => setState(() {}),
        ),

        // Terms checkbox
        McpCheckbox(
          id: 'terms-checkbox',
          label: 'Accept terms',
          value: _acceptedTerms,
          onChanged: (val) => setState(() => _acceptedTerms = val ?? false),
        ),

        // Submit button with healing hints
        McpButton(
          id: 'submit-button',
          label: 'Submit',
          onPressed: _isValid ? _submit : null,
          enabled: _isValid,
          metadata: {
            'formValid': _isValid,
            'disabledReasons': _getDisabledReasons(),
            'healingHints': {
              'canAutoHeal': !_isValid,
              'autoHealableFields': _getFixableFields(),
            },
          },
          child: const Text('Submit'),
        ),
      ],
    );
  }
}

Now AI can:

• ✅ See all form fields
• ✅ Fill fields automatically
• ✅ Understand validation rules
• ✅ Track progress
• ✅ Submit when ready

Configuration Options

Customize the server behavior with McpServerConfig:

await McpSemanticsServer.instance.start(
  config: McpServerConfig(
    // App metadata for AI assistants
    appName: 'My App',
    appDescription: 'A demo app showing MCP integration',

    // Logging verbosity
    logLevel: McpLogLevel.verbose,  // none, basic, verbose, debug

    // Error retention
    errorRetentionHours: 48,         // Keep errors for 48 hours
    errorRetentionCount: 1000,       // Max 1000 errors in database

    // User action tracking
    captureUserActions: true,        // Track taps, scrolls, navigation

    // Privacy
    sanitizeRouteParams: true,       // /user/123 → /user/:id
    enableInRelease: false,          // Never enable in production!
  ),
);

Available MCP Resources

The MCP server exposes these resources:

semantics://tree

Returns the complete Flutter semantics tree in JSON format. Includes all SemanticsData fields: labels, values, hints, actions, bounds, transforms, and more.

errors://recent

Returns the most recent errors (last 10 by default). Add ?limit=20 to customize.

errors://all

Returns all errors currently in the database.

errors://unresolved

Returns only errors not yet marked as resolved.

Available MCP Tools

get_semantics_node

Get detailed semantics data for a specific node by ID.

{"node_id": 42}

get_error_details

Get complete error details including all context (stack trace, widget path, semantics snapshot, user actions).

{"error_id": 123}

query_errors

Search and filter errors by type, message pattern, time range, or resolved status.

{
  "error_type": "RenderFlexOverflow",
  "since": "2025-10-21T10:00:00Z",
  "resolved": false,
  "limit": 20
}

mark_error_resolved

Mark an error as resolved or unresolved.

{"error_id": 123, "resolved": true}

clear_errors

Clear errors from the database with optional filtering.

{"filter": "resolved", "older_than_hours": 24}

Error Database Schema

Errors are stored in SQLite (mcp_errors.db) with comprehensive context:

Field	Type	Description
id	INTEGER	Auto-incrementing error ID
timestamp	TEXT	ISO 8601 timestamp
error_type	TEXT	Exception/Error class name
error_message	TEXT	Full error message
stack_trace	TEXT	Complete stack trace
widget_context	TEXT	Widget tree path (JSON)
semantics_snapshot	TEXT	Semantics tree at error time (JSON)
app_state	TEXT	Additional app state (JSON)
user_actions	TEXT	Last 10 user interactions (JSON)
is_fatal	INTEGER	1 if fatal, 0 if non-fatal
is_resolved	INTEGER	1 if marked resolved

Troubleshooting

Server not starting

• Ensure WidgetsFlutterBinding.ensureInitialized() is called before start()
• Check that you're running in debug or profile mode (server auto-disables in release)
• Look for error messages in the console with [MCP] prefix

Claude Code can't connect

• Verify your MCP server configuration in Claude Code settings
• Ensure your Flutter app is running
• Check that the stdio transport is working (look for [MCP] Semantics Server started on stdio in console)
• See CLAUDE_CODE_SETUP.md for detailed troubleshooting

Errors not being captured

• Check that McpSemanticsServer.instance.start() completed successfully
• Verify error capture is initialized (look for [MCP] Error capture initialized in console)
• Try triggering an intentional error to test

Performance issues

• Reduce logLevel to basic or none
• Lower errorRetentionCount if database is very large
• Disable captureUserActions if not needed

Privacy Considerations

This package is designed with privacy in mind:

• Auto-disables in release builds: Server never runs in production unless explicitly enabled
• Text input not captured: User keystrokes and form data are never stored
• Route sanitization: URL parameters are sanitized by default (/user/123 → /user/:id)
• Local only: All data stays on device, no network transmission
• User action tracking: Captures interaction types (tap, scroll) but not content
• Semantics snapshots: Only captures accessibility data, not pixel content

Warning: Never set enableInRelease: true in production apps, as it exposes internal application state.

Platform Support

Platform	Status	Notes
macOS	✅ Supported	Full support (desktop)
iOS	✅ Supported	Simulator and physical devices via USB debugging
Android	⏳ Phase 2	Planned for future release
Web	⏳ Phase 2	Planned for future release
Windows	⏳ Phase 2	Planned for future release
Linux	⏳ Phase 2	Planned for future release

Example App

See the example/ directory for a complete demonstration app showing:

• MCP server initialization
• Comprehensive semantics annotations
• Intentional error triggering
• Error recovery patterns
• User interaction tracking

Run the example:

cd example
flutter run -d macos  # or -d ios

API Documentation

Full API documentation is available:

dart doc
open doc/api/index.html

Or view the generated docs on pub.dev.

How It Works

1. MCP Server: Runs inside your Flutter app using stdio transport
2. Semantics Tree: Accessed via WidgetsBinding.instance.pipelineOwner?.semanticsOwner
3. Error Capture: Hooks into FlutterError.onError and PlatformDispatcher.instance.onError
4. SQLite Storage: Errors persisted to mcp_errors.db in app documents directory
5. User Actions: Tracked via gesture detectors and navigation observers (circular buffer of last 10 actions)
6. MCP Protocol: Resources and tools exposed via mcp_dart package

Performance Impact

The MCP server is designed for minimal performance impact:

• Frame rendering: < 2ms impact (negligible)
• Semantics queries: Complete in < 2 seconds for typical apps (< 1000 nodes)
• Error capture: Asynchronous, doesn't block error handlers
• Memory: ~5-10MB for error database (configurable retention)

Contributing

Contributions are welcome! This is an open-source project.

Repository: https://github.com/moinsen-dev/mcp-frontend-heor

License

MIT License - see LICENSE file for details.

Documentation

• SETUP.md - Installation, configuration, and platform-specific setup
• HOW_TO.md - Usage guides, widgets, testing, and best practices
• KNOW_HOW.md - Architecture, implementation details, and troubleshooting
• Example App - Complete demonstration app

Learn More

• Model Context Protocol (MCP)
• Claude Code
• Flutter Semantics

Support

• Issues: GitHub Issues
• Discussions: GitHub Discussions

---

Made with ❤️ for Flutter developers who love AI-assisted debugging