Streaming API Reference

The Streaming API provides real-time, token-by-token responses for building responsive chat interfaces and interactive applications. The streaming format is OpenAI-compatible, using Server-Sent Events (SSE).

Endpoint

POST https://api.withperf.pro/v1/chat/stream

Authentication

Include your API key in the Authorization header:

Authorization: Bearer YOUR_API_KEY

How It Works

The Streaming API returns responses using Server-Sent Events (SSE), sending text chunks as they’re generated rather than waiting for the complete response.

Benefits

Lower perceived latency: Users see responses immediately
Better UX: Progressive rendering feels more responsive
Real-time feedback: Stop generation early if needed
Streaming UI: Perfect for chat interfaces

Request Body

Same as the Chat API, but responses stream incrementally. This includes full support for multimodal content (images, audio, video, documents) in messages.

Example Request

curl -X POST https://api.withperf.pro/v1/chat/stream \
  -H "Authorization: Bearer pk_test_abc123" \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [
      {
        "role": "user",
        "content": "Explain quantum computing in simple terms"
      }
    ],
    "max_cost_per_call": 0.01
  }'

Response Format

The response uses OpenAI-compatible Server-Sent Events (SSE) format:

data: {"id":"chatcmpl-abc123","object":"chat.completion.chunk","created":1705312200,"model":"claude-sonnet-4-5-20250929","choices":[{"index":0,"delta":{"role":"assistant","content":"Quantum"},"finish_reason":null}]}

data: {"id":"chatcmpl-abc123","object":"chat.completion.chunk","created":1705312200,"model":"claude-sonnet-4-5-20250929","choices":[{"index":0,"delta":{"content":" computing"},"finish_reason":null}]}

data: {"id":"chatcmpl-abc123","object":"chat.completion.chunk","created":1705312200,"model":"claude-sonnet-4-5-20250929","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}

data: [DONE]

Event Types

Content Chunk

Sent for each token or group of tokens. The first chunk includes the role, subsequent chunks only have content:

{
  "id": "chatcmpl-abc123",
  "object": "chat.completion.chunk",
  "created": 1705312200,
  "model": "claude-sonnet-4-5-20250929",
  "choices": [
    {
      "index": 0,
      "delta": {
        "role": "assistant",
        "content": "text fragment"
      },
      "finish_reason": null
    }
  ]
}

Final Chunk

Sent when generation is complete with finish_reason: "stop":

{
  "id": "chatcmpl-abc123",
  "object": "chat.completion.chunk",
  "created": 1705312200,
  "model": "claude-sonnet-4-5-20250929",
  "choices": [
    {
      "index": 0,
      "delta": {},
      "finish_reason": "stop"
    }
  ]
}

Done Signal

After the final chunk, a [DONE] message indicates the stream is complete:

data: [DONE]

Client Implementation

JavaScript/TypeScript

async function streamChat(messages: Message[]) {
  const response = await fetch('https://api.withperf.pro/v1/chat/stream', {
    method: 'POST',
    headers: {
      'Authorization': `Bearer ${API_KEY}`,
      'Content-Type': 'application/json',
    },
    body: JSON.stringify({ messages }),
  });

  const reader = response.body?.getReader();
  const decoder = new TextDecoder();
  let fullText = '';

  while (true) {
    const { done, value } = await reader!.read();
    if (done) break;

    const chunk = decoder.decode(value);
    const lines = chunk.split('\n');

    for (const line of lines) {
      if (line.startsWith('data: ')) {
        const payload = line.slice(6);

        // Check for [DONE] signal
        if (payload === '[DONE]') {
          console.log('Stream complete!');
          break;
        }

        const data = JSON.parse(payload);
        const content = data.choices?.[0]?.delta?.content || '';

        if (content) {
          fullText += content;
          console.log('Partial:', fullText);
        }

        // Check for finish_reason
        if (data.choices?.[0]?.finish_reason === 'stop') {
          console.log('Model:', data.model);
        }
      }
    }
  }

  return fullText;
}

React Hook

import { useState, useCallback } from 'react';

export function useStreamingChat() {
  const [content, setContent] = useState('');
  const [isStreaming, setIsStreaming] = useState(false);
  const [model, setModel] = useState<string | null>(null);

  const streamMessage = useCallback(async (messages: Message[]) => {
    setContent('');
    setIsStreaming(true);
    setModel(null);

    try {
      const response = await fetch('/api/chat/stream', {
        method: 'POST',
        headers: { 'Content-Type': 'application/json' },
        body: JSON.stringify({ messages }),
      });

      const reader = response.body?.getReader();
      const decoder = new TextDecoder();

      while (true) {
        const { done, value } = await reader!.read();
        if (done) break;

        const chunk = decoder.decode(value);
        const lines = chunk.split('\n');

        for (const line of lines) {
          if (line.startsWith('data: ')) {
            const payload = line.slice(6);
            if (payload === '[DONE]') break;

            const data = JSON.parse(payload);
            const deltaContent = data.choices?.[0]?.delta?.content || '';

            if (deltaContent) {
              setContent(prev => prev + deltaContent);
            }

            if (data.choices?.[0]?.finish_reason === 'stop') {
              setModel(data.model);
            }
          }
        }
      }
    } finally {
      setIsStreaming(false);
    }
  }, []);

  return { content, isStreaming, model, streamMessage };
}

Python

import requests
import json

def stream_chat(messages):
    url = "https://api.withperf.pro/v1/chat/stream"
    headers = {
        "Authorization": f"Bearer {API_KEY}",
        "Content-Type": "application/json"
    }
    payload = {"messages": messages}

    with requests.post(url, json=payload, headers=headers, stream=True) as response:
        full_text = ""

        for line in response.iter_lines():
            if line:
                line = line.decode('utf-8')
                if line.startswith('data: '):
                    payload = line[6:]

                    # Check for [DONE] signal
                    if payload == '[DONE]':
                        print("\n\nStream complete!")
                        break

                    data = json.loads(payload)
                    content = data.get('choices', [{}])[0].get('delta', {}).get('content', '')

                    if content:
                        full_text += content
                        print(content, end='', flush=True)

                    # Check for finish_reason
                    if data.get('choices', [{}])[0].get('finish_reason') == 'stop':
                        print(f"\n\nModel: {data.get('model')}")

        return full_text

# Usage
messages = [{"role": "user", "content": "Tell me a story"}]
result = stream_chat(messages)

Python Async

import aiohttp
import asyncio
import json

async def stream_chat_async(messages):
    url = "https://api.withperf.pro/v1/chat/stream"
    headers = {
        "Authorization": f"Bearer {API_KEY}",
        "Content-Type": "application/json"
    }
    payload = {"messages": messages}

    async with aiohttp.ClientSession() as session:
        async with session.post(url, json=payload, headers=headers) as response:
            async for line in response.content:
                line = line.decode('utf-8').strip()
                if line.startswith('data: '):
                    payload = line[6:]

                    if payload == '[DONE]':
                        yield {'done': True}
                        break

                    data = json.loads(payload)
                    content = data.get('choices', [{}])[0].get('delta', {}).get('content', '')

                    if content:
                        yield content

# Usage
async def main():
    messages = [{"role": "user", "content": "Explain AI"}]
    async for chunk in stream_chat_async(messages):
        if isinstance(chunk, str):
            print(chunk, end='', flush=True)
        else:
            print("\n\nDone!")

asyncio.run(main())

Go

package main

import (
    "bufio"
    "bytes"
    "encoding/json"
    "fmt"
    "net/http"
    "strings"
)

type StreamChunk struct {
    Choices []struct {
        Delta struct {
            Content string `json:"content"`
        } `json:"delta"`
        FinishReason *string `json:"finish_reason"`
    } `json:"choices"`
    Model string `json:"model"`
}

func streamChat(messages []Message) error {
    payload, _ := json.Marshal(map[string]interface{}{
        "messages": messages,
    })

    req, _ := http.NewRequest("POST",
        "https://api.withperf.pro/v1/chat/stream",
        bytes.NewBuffer(payload))

    req.Header.Set("Authorization", "Bearer "+API_KEY)
    req.Header.Set("Content-Type", "application/json")

    client := &http.Client{}
    resp, err := client.Do(req)
    if err != nil {
        return err
    }
    defer resp.Body.Close()

    scanner := bufio.NewScanner(resp.Body)
    for scanner.Scan() {
        line := scanner.Text()
        if strings.HasPrefix(line, "data: ") {
            payload := line[6:]

            // Check for [DONE] signal
            if payload == "[DONE]" {
                fmt.Println("\nDone!")
                break
            }

            var chunk StreamChunk
            json.Unmarshal([]byte(payload), &chunk)

            if len(chunk.Choices) > 0 {
                content := chunk.Choices[0].Delta.Content
                if content != "" {
                    fmt.Print(content)
                }
            }
        }
    }

    return scanner.Err()
}

React Component Example

'use client';

import { useState } from 'react';

export default function StreamingChat() {
  const [messages, setMessages] = useState<Message[]>([]);
  const [input, setInput] = useState('');
  const [streamingContent, setStreamingContent] = useState('');
  const [isStreaming, setIsStreaming] = useState(false);

  const sendMessage = async () => {
    if (!input.trim()) return;

    const userMessage = { role: 'user', content: input };
    setMessages(prev => [...prev, userMessage]);
    setInput('');
    setIsStreaming(true);
    setStreamingContent('');

    try {
      const response = await fetch('/api/chat/stream', {
        method: 'POST',
        headers: { 'Content-Type': 'application/json' },
        body: JSON.stringify({
          messages: [...messages, userMessage],
        }),
      });

      const reader = response.body?.getReader();
      const decoder = new TextDecoder();
      let fullContent = '';

      while (true) {
        const { done, value } = await reader!.read();
        if (done) break;

        const chunk = decoder.decode(value);
        const lines = chunk.split('\n');

        for (const line of lines) {
          if (line.startsWith('data: ')) {
            const payload = line.slice(6);

            // Check for [DONE] signal
            if (payload === '[DONE]') {
              setMessages(prev => [
                ...prev,
                { role: 'assistant', content: fullContent }
              ]);
              setStreamingContent('');
              break;
            }

            const data = JSON.parse(payload);
            const deltaContent = data.choices?.[0]?.delta?.content || '';

            if (deltaContent) {
              fullContent += deltaContent;
              setStreamingContent(fullContent);
            }
          }
        }
      }
    } catch (error) {
      console.error('Streaming error:', error);
    } finally {
      setIsStreaming(false);
    }
  };

  return (
    <div className="flex flex-col h-screen">
      <div className="flex-1 overflow-y-auto p-4">
        {messages.map((msg, idx) => (
          <div key={idx} className="mb-4">
            <div className="font-bold">{msg.role}:</div>
            <div>{msg.content}</div>
          </div>
        ))}

        {streamingContent && (
          <div className="mb-4">
            <div className="font-bold">assistant:</div>
            <div className="animate-pulse">{streamingContent}</div>
          </div>
        )}
      </div>

      <div className="border-t p-4">
        <input
          type="text"
          value={input}
          onChange={(e) => setInput(e.target.value)}
          onKeyPress={(e) => e.key === 'Enter' && sendMessage()}
          disabled={isStreaming}
          className="w-full border rounded px-3 py-2"
          placeholder="Type a message..."
        />
      </div>
    </div>
  );
}

Error Handling

Connection Errors

try {
  const response = await fetch(url, { ... });

  if (!response.ok) {
    throw new Error(`HTTP ${response.status}: ${response.statusText}`);
  }

  if (!response.body) {
    throw new Error('No response body');
  }

  // Stream processing...
} catch (error) {
  console.error('Streaming failed:', error);
  // Fallback to non-streaming API
  const fallback = await fetch('/v1/chat', { ... });
}

Timeout Handling

const controller = new AbortController();
const timeout = setTimeout(() => controller.abort(), 30000); // 30s timeout

try {
  const response = await fetch(url, {
    signal: controller.signal,
    ...
  });
  // Process stream...
} catch (error) {
  if (error.name === 'AbortError') {
    console.error('Request timed out');
  }
} finally {
  clearTimeout(timeout);
}

Performance Optimization

Chunking Strategy

Perf optimizes chunk size for balance between latency and throughput:

Small prompts: Sends tokens individually for fastest perceived speed
Large generations: Batches tokens for network efficiency
Adaptive: Adjusts based on connection quality

Buffering

For smoother UI updates, buffer chunks:

let buffer = '';
let lastUpdate = Date.now();

// In your streaming loop:
buffer += data.chunk;

const now = Date.now();
if (now - lastUpdate > 50) { // Update every 50ms
  setContent(prev => prev + buffer);
  buffer = '';
  lastUpdate = now;
}

Rate Limits

Same limits as the Chat API:

Tier	Requests/Minute	Concurrent Streams
Free	60	3
Pro	300	10
Enterprise	Custom	Custom

Best Practices

1. Show Loading State

{isStreaming && (
  <div className="flex items-center gap-2">
    <Spinner />
    <span>Generating response...</span>
  </div>
)}

2. Handle Stream Interruption

Allow users to stop generation:

const abortController = new AbortController();

// Cancel button handler
const handleCancel = () => {
  abortController.abort();
  setIsStreaming(false);
};

// Pass to fetch
fetch(url, { signal: abortController.signal, ... });

3. Graceful Degradation

Fall back to non-streaming if not supported:

const supportsStreaming = 'ReadableStream' in window;

if (supportsStreaming) {
  // Use streaming API
} else {
  // Use regular Chat API
}

4. Optimize for Mobile

Consider connection quality:

// Detect slow connections
const connection = (navigator as any).connection;
const isSlowConnection = connection?.effectiveType === '2g' ||
                         connection?.effectiveType === 'slow-2g';

if (isSlowConnection) {
  // Use regular API or increase buffer size
}

Comparison: Streaming vs Non-Streaming

Feature	Streaming	Non-Streaming
First token latency	~200ms	~2-5s
Perceived speed	Immediate	Delayed
Implementation complexity	Medium	Low
Network efficiency	Same	Same
Error recovery	More complex	Simple
Best for	Chat UIs, long responses	Batch processing, short responses

Chat API - Non-streaming version
Metrics API - Analytics and monitoring
Logs API - Debugging and audit trails

Support

Documentation: docs.withperf.pro
Email: support@withperf.pro
Examples: github.com/perf/examples

​Streaming API Reference

​Endpoint

​Authentication

​How It Works

​Benefits

​Request Body

​Example Request

​Response Format

​Event Types

​Content Chunk

​Final Chunk

​Done Signal

​Client Implementation

​JavaScript/TypeScript

​React Hook

​Python

​Python Async

​Go

​React Component Example

​Error Handling

​Connection Errors

​Timeout Handling

​Performance Optimization

​Chunking Strategy

​Buffering

​Rate Limits

​Best Practices

​1. Show Loading State

​2. Handle Stream Interruption

​3. Graceful Degradation

​4. Optimize for Mobile

​Comparison: Streaming vs Non-Streaming

​Related Endpoints

​Support