UI State Tracker

A robust and versatile package for capturing and streaming real-time UI state data from various frontend frameworks to a Supabase real-time database. This package enables the creation of AI agents that can interact with and understand the dynamic state of user interfaces.

Features

• 🎯 Comprehensive event tracking (mouse, keyboard, touch, form, navigation)
• 🔄 Real-time data streaming to Supabase
• ⚛️ Framework support for React, Vue, and vanilla JavaScript
• 🎨 Customizable event sampling and filtering
• 🔍 Detailed event metadata capture
• 🛡️ TypeScript support with full type definitions
• 📊 Efficient event buffering and batching
• 🎮 Custom event support

Installation

npm install @ui-state/core

Quick Start

React

import { UIStateProvider, useUIState } from '@ui-state/core';

// Configure the provider
const config = {
  supabaseUrl: 'YOUR_SUPABASE_URL',
  supabaseKey: 'YOUR_SUPABASE_KEY',
  samplingInterval: 100, // Optional: limit event frequency
  debug: true // Optional: enable debug logging
};

// Wrap your app with the provider
function App() {
  return (
    <UIStateProvider config={config}>
      <YourApp />
    </UIStateProvider>
  );
}

// Use in components
function MyComponent() {
  const { trackCustomEvent, emitCustomEvent } = useUIState();

  // Track custom events
  useEffect(() => {
    trackCustomEvent('my-event', (event) => {
      console.log('Custom event:', event);
    });
  }, []);

  return (
    <button onClick={() => emitCustomEvent('my-event', { action: 'click' })}>
      Click me
    </button>
  );
}

Vue

import { createApp } from 'vue';
import { UIStatePlugin } from '@ui-state/core';

const app = createApp(App);

app.use(UIStatePlugin, {
  supabaseUrl: 'YOUR_SUPABASE_URL',
  supabaseKey: 'YOUR_SUPABASE_KEY'
});

// Use in components
export default {
  setup() {
    const { trackCustomEvent, emitCustomEvent } = useUITracker();

    // Track custom events
    trackCustomEvent('my-event', (event) => {
      console.log('Custom event:', event);
    });

    return {
      handleClick: () => emitCustomEvent('my-event', { action: 'click' })
    };
  }
};

Vanilla JavaScript

import { UIStateManager, trackElements, trackForm } from '@ui-state/core';

// Initialize the manager
UIStateManager.initialize({
  supabaseUrl: 'YOUR_SUPABASE_URL',
  supabaseKey: 'YOUR_SUPABASE_KEY'
});

// Track specific elements
trackElements('.interactive-element', 'my-elements');

// Track forms
const form = document.querySelector('#my-form');
trackForm(form, 'registration-form', (event) => {
  console.log('Form submitted:', event);
});

// Track custom events
const manager = UIStateManager.getInstance();
manager.trackCustomEvent('my-event', (event) => {
  console.log('Custom event:', event);
});

AI Cursor Component

The package includes a customizable AI cursor component that can be used to visualize AI interactions with your UI.

React Usage

import { AICursor, AICursorController } from '@cyrusai/interactions';

function App() {
  // Create a cursor controller instance
  const cursorController = new AICursorController();

  return (
    <div>
      <AICursor 
        size={24}
        color="#007AFF"
        showTrail={true}
        trailLength={5}
        state="idle"
      />
      
      {/* Example of programmatic cursor control */}
      <button onClick={() => {
        cursorController.moveTo({ x: 100, y: 100 });
        cursorController.click();
      }}>
        Control Cursor
      </button>
    </div>
  );
}

Cursor Properties

| Property | Type | Default | Description | |----------|------|---------|-------------| | size | number | 20 | Size of the cursor in pixels | | color | string | '#007AFF' | Primary color of the cursor | | showTrail | boolean | false | Whether to show motion trail | | trailLength | number | 3 | Number of trail elements | | state | CursorState | 'idle' | Current cursor state | | position | {x: number, y: number} | {x: 0, y: 0} | Cursor position |

Cursor States

• idle: Default state with subtle pulse animation
• thinking: Shows loading/processing animation
• acting: Active state during interactions
• dragging: Shown while dragging elements
• selecting: Used during text/element selection
• pressing: Indicates press-and-hold action

Cursor Controller API

The AICursorController provides methods for programmatic cursor control:

// Basic movements
cursorController.moveTo({ x: 100, y: 100 });
cursorController.moveBy({ x: 50, y: 0 });

// Mouse actions
cursorController.click();
cursorController.doubleClick();
cursorController.rightClick();
cursorController.pressAndHold(1000); // duration in ms

// Complex interactions
cursorController.drag({ 
  start: { x: 0, y: 0 }, 
  end: { x: 100, y: 100 } 
});
cursorController.selectText({
  start: { x: 0, y: 0 },
  end: { x: 200, y: 0 }
});
cursorController.scroll({ x: 0, y: 100 });

Configuration Options

| Option | Type | Description | |--------|------|-------------| | supabaseUrl | string | Your Supabase project URL | | supabaseKey | string | Your Supabase API key | | samplingInterval | number | Optional: Minimum time (ms) between events | | debug | boolean | Optional: Enable debug logging | | customEvents | string[] | Optional: List of custom event names to register |

Event Types

The package tracks the following event types:

• Mouse events: click, contextmenu, dblclick, mousedown, mouseup, mousemove, etc.
• Keyboard events: keydown, keyup, keypress
• Form events: focus, blur, change, submit
• Touch events: touchstart, touchmove, touchend
• Window events: resize, scroll
• Custom events: user-defined events

Event Data Structure

Each tracked event includes:

interface UIEvent {
  timestamp: number;
  eventId: string;
  componentId: string;
  componentType: ComponentType;
  eventType: EventType;
  coordinates: {
    x: number;
    y: number;
  };
  text?: string;
  value?: any;
  target?: string;
  metadata?: Record<string, any>;
}

Supabase Setup

1. Create a new Supabase project
2. Create a table for UI events:

create table ui_events (
  id uuid default uuid_generate_v4() primary key,
  timestamp bigint not null,
  event_id text not null,
  component_id text not null,
  component_type text not null,
  event_type text not null,
  coordinates jsonb not null,
  text text,
  value jsonb,
  target text,
  metadata jsonb,
  created_at timestamp with time zone default timezone('utc'::text, now()) not null
);

-- Enable real-time for the table
alter table ui_events replica identity full;

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

License

MIT License - see LICENSE file for details