Capture user feedback on AI responses and app quality

User Feedback Integration

Overseer includes a feedback system to capture user sentiment on AI responses. This helps you understand response quality and identify issues.

Overview

The feedback system provides:

  • Thumbs up/down rating for AI responses
  • Feedback modal for detailed feedback on bad responses
  • Automatic issue creation for negative feedback
  • Analytics on response quality

Quick Start

1. Install Components

npm install @codmir/overseer

2. Add Feedback Buttons

import { FeedbackButtons } from '@codmir/overseer/react';

function AIMessage({ message, conversationId }) {
  return (
    <div className="message">
      <p>{message.content}</p>
      
      <FeedbackButtons
        messageId={message.id}
        conversationId={conversationId}
        onFeedback={(feedback) => {
          console.log('Feedback received:', feedback);
        }}
      />
    </div>
  );
}

3. Handle Negative Feedback

When users click thumbs down, show a feedback modal:

import { FeedbackModal } from '@codmir/overseer/react';
import { useState } from 'react';

function AIMessage({ message, conversationId }) {
  const [showFeedback, setShowFeedback] = useState(false);

  const handleFeedback = (type: 'positive' | 'negative') => {
    if (type === 'negative') {
      setShowFeedback(true);
    } else {
      // Track positive feedback
      trackFeedback({
        type: 'positive',
        messageId: message.id,
        conversationId,
      });
    }
  };

  return (
    <>
      <FeedbackButtons onFeedback={handleFeedback} />
      
      <FeedbackModal
        open={showFeedback}
        onClose={() => setShowFeedback(false)}
        messageId={message.id}
        conversationId={conversationId}
        onSubmit={(feedback) => {
          // Creates an issue automatically
        }}
      />
    </>
  );
}

Feedback Categories

The feedback modal includes predefined categories:

CategoryDescription
incorrect_incompleteResponse is factually wrong or missing information
not_what_askedResponse doesn't address the user's question
slow_buggyPerformance or technical issues
style_toneWriting style or tone issues
safety_legalSafety or legal concerns
otherOther issues with details

Manual Integration

trackFeedback

Track feedback manually:

import { trackFeedback } from '@codmir/overseer';

// Positive feedback
trackFeedback({
  type: 'positive',
  messageId: 'msg_123',
  conversationId: 'conv_456',
});

// Negative feedback with category
trackFeedback({
  type: 'negative',
  messageId: 'msg_123',
  conversationId: 'conv_456',
  category: 'incorrect_incomplete',
  details: 'The code example had a syntax error',
  context: {
    model: 'gpt-4o',
    prompt: 'Write a React component...',
    response: message.content,
  },
});

API Reference

interface FeedbackPayload {
  type: 'positive' | 'negative';
  messageId: string;
  conversationId: string;
  category?: FeedbackCategory;
  details?: string;
  context?: {
    model?: string;
    prompt?: string;
    response?: string;
    [key: string]: unknown;
  };
}

type FeedbackCategory =
  | 'incorrect_incomplete'
  | 'not_what_asked'
  | 'slow_buggy'
  | 'style_tone'
  | 'safety_legal'
  | 'other';

React Components

FeedbackButtons

Renders thumbs up/down buttons:

import { FeedbackButtons } from '@codmir/overseer/react';

<FeedbackButtons
  messageId="msg_123"
  conversationId="conv_456"
  onFeedback={(type) => {
    // Handle feedback
  }}
  size="sm" // 'sm' | 'md' | 'lg'
  showLabels={false}
/>

FeedbackModal

Modal for detailed negative feedback:

import { FeedbackModal } from '@codmir/overseer/react';

<FeedbackModal
  open={isOpen}
  onClose={() => setIsOpen(false)}
  messageId="msg_123"
  conversationId="conv_456"
  categories={[
    'incorrect_incomplete',
    'not_what_asked',
    'slow_buggy',
    'other',
  ]}
  onSubmit={(feedback) => {
    // Feedback submitted
  }}
/>

Issue Creation

Negative feedback automatically creates issues:

┌──────────────┐     ┌───────────────┐     ┌─────────────┐
│  User clicks │────▶│  trackFeedback│────▶│   Issue     │
│  thumbs down │     │  (negative)   │     │   Created   │
└──────────────┘     └───────────────┘     └─────────────┘
                            │
                            ▼
                     feedback.negative event
                            │
                            ▼
                     Issue in /project/[id]/issues

The created issue includes:

  • Feedback category and details
  • Message content and context
  • Conversation ID for reference
  • Model used and prompt (optional)

Example: Full Integration

'use client';

import { useState } from 'react';
import { ThumbsUp, ThumbsDown } from 'lucide-react';
import { trackFeedback } from '@codmir/overseer';
import { FeedbackModal } from '@/components/feedback-modal';

interface MessageActionsProps {
  messageId: string;
  conversationId: string;
  messageContent: string;
  model?: string;
}

export function MessageActions({
  messageId,
  conversationId,
  messageContent,
  model,
}: MessageActionsProps) {
  const [feedback, setFeedback] = useState<'positive' | 'negative' | null>(null);
  const [showModal, setShowModal] = useState(false);

  const handlePositive = () => {
    setFeedback('positive');
    trackFeedback({
      type: 'positive',
      messageId,
      conversationId,
    });
  };

  const handleNegative = () => {
    setFeedback('negative');
    setShowModal(true);
  };

  const handleSubmitFeedback = (category: string, details?: string) => {
    trackFeedback({
      type: 'negative',
      messageId,
      conversationId,
      category: category as FeedbackCategory,
      details,
      context: {
        model,
        response: messageContent,
      },
    });
    setShowModal(false);
  };

  return (
    <>
      <div className="flex items-center gap-1">
        <button
          onClick={handlePositive}
          className={`p-1 rounded hover:bg-muted ${
            feedback === 'positive' ? 'text-green-500' : 'text-muted-foreground'
          }`}
          title="Good response"
        >
          <ThumbsUp className="h-4 w-4" />
        </button>
        <button
          onClick={handleNegative}
          className={`p-1 rounded hover:bg-muted ${
            feedback === 'negative' ? 'text-red-500' : 'text-muted-foreground'
          }`}
          title="Bad response"
        >
          <ThumbsDown className="h-4 w-4" />
        </button>
      </div>

      <FeedbackModal
        open={showModal}
        onClose={() => setShowModal(false)}
        onSubmit={handleSubmitFeedback}
      />
    </>
  );
}

Analytics Dashboard

View feedback analytics at /project/[id]/analytics/feedback:

  • Positive/Negative ratio over time
  • Common categories for negative feedback
  • Model performance comparison
  • Trending issues from feedback

Best Practices

  1. Always show feedback buttons on AI responses
  2. Make positive feedback one-click - no modal needed
  3. Collect context with negative feedback for debugging
  4. Review feedback regularly to improve AI quality
  5. Close the loop - fix issues and track improvement