Workers AI in Blaze: Text Generation and Streaming

Call Cloudflare Workers AI models from Blaze route handlers via req.env.AI. Supports text generation, streaming SSE responses, and multiple model families.

Cloudflare Workers AI is a serverless inference API that runs machine learning models on Cloudflare’s global GPU network — no model hosting, no infrastructure to manage. Blaze exposes it on req.env.AI in every handler once you add the binding to your Env type. You get access to a broad catalog of model families — text generation, embeddings, image classification, speech recognition, and more — with optional streaming for real-time output.

Configure the binding

1. Add the AI binding to wrangler.toml

   wrangler.toml

   [ai]
   binding = "AI"
   
2. Define the Env type

   src/types/env.ts

   export type Env = {
     AI: Ai;
     // ...other bindings
   };
   
3. Pass Env to createApp

   src/index.ts

   import { createApp } from "blaze";
   import type { Env } from "./types/env";
   
   const app = createApp<Env>();
   // req.env.AI is fully typed as Ai in every handler
   

Text generation

Call req.env.AI.run(model, inputs) to run inference synchronously. Pass a messages array in the OpenAI chat format and set stream: false to receive the complete response in one go.

app.post("/ai/complete", async (req, res) => {
  const { prompt, systemPrompt } = await req.json<{
    prompt:       string;
    systemPrompt?: string;
  }>();

  const result = await req.env.AI.run("@cf/meta/llama-3.1-8b-instruct", {
    messages: [
      {
        role:    "system",
        content: systemPrompt ?? "You are a helpful assistant.",
      },
      {
        role:    "user",
        content: prompt,
      },
    ],
    stream: false,
  });

  res.json({ response: result.response });
});

Streaming responses

Set stream: true to receive a ReadableStream of server-sent events. Pipe it directly to the client using res.stream() so tokens appear in the browser as they are generated — with no buffering in your Worker.

app.post("/ai/stream", async (req, res) => {
  const { prompt } = await req.json<{ prompt: string }>();

  const aiStream = await req.env.AI.run("@cf/meta/llama-3.1-8b-instruct", {
    messages: [{ role: "user", content: prompt }],
    stream:   true,
  });

  res.header("Content-Type",  "text/event-stream");
  res.header("Cache-Control", "no-cache");
  res.header("Connection",    "keep-alive");

  res.stream((writer) => aiStream.pipeTo(writer));
});

Your client can consume the stream with the standard EventSource API or fetch with a ReadableStream reader:

const response = await fetch("/ai/stream", {
  method:  "POST",
  headers: { "Content-Type": "application/json" },
  body:    JSON.stringify({ prompt: "Explain Cloudflare Workers in one paragraph." }),
});

const reader = response.body!.getReader();
const decoder = new TextDecoder();

while (true) {
  const { done, value } = await reader.read();
  if (done) break;
  process.stdout.write(decoder.decode(value));
}

Available models

Workers AI supports multiple model families. Browse the full catalog — including model IDs, input/output schemas, and benchmark scores — at developers.cloudflare.com/workers-ai/models/.

Family	Example model	Use case
Text generation	@cf/meta/llama-3.1-8b-instruct	Chat, completion, summarization
Text embeddings	@cf/baai/bge-base-en-v1.5	Semantic search, RAG pipelines
Image classification	@cf/microsoft/resnet-50	Label images by category
Speech recognition	@cf/openai/whisper	Transcribe audio to text
Text-to-image	@cf/stabilityai/stable-diffusion-xl-base-1.0	Generate images from prompts
Translation	@cf/meta/m2m100-1.2b	Translate between 100+ languages

// Embeddings example — generate a vector for semantic search
app.post("/ai/embed", async (req, res) => {
  const { text } = await req.json<{ text: string }>();

  const result = await req.env.AI.run("@cf/baai/bge-base-en-v1.5", {
    text,
  });

  res.json({ embedding: result.data[0] });
});

// Speech recognition example — transcribe an audio file
app.post("/ai/transcribe", async (req, res) => {
  const audioBuffer = await req.arrayBuffer();

  const result = await req.env.AI.run("@cf/openai/whisper", {
    audio: [...new Uint8Array(audioBuffer)],
  });

  res.json({ text: result.text });
});

Error handling

AI inference can fail due to upstream model errors, invalid inputs, or capacity limits. Wrap every req.env.AI.run() call in a try/catch and throw a BlazeError(502) so your global error handler can return a consistent error shape.

import { BlazeError } from "blaze";

app.post("/ai/complete", async (req, res) => {
  const { prompt } = await req.json<{ prompt: string }>();

  let result: { response: string };

  try {
    result = await req.env.AI.run("@cf/meta/llama-3.1-8b-instruct", {
      messages: [{ role: "user", content: prompt }],
      stream:   false,
    });
  } catch (err) {
    throw new BlazeError(502, "AI inference failed. Please try again.", {
      code: "AI_UPSTREAM_ERROR",
    });
  }

  if (!result.response) {
    throw new BlazeError(502, "AI returned an empty response.", {
      code: "AI_EMPTY_RESPONSE",
    });
  }

  res.json({ response: result.response });
});

Pair this with a global error handler so BlazeError instances are serialized consistently:

app.onError((err, req, res, _next) => {
  const status  = err instanceof BlazeError ? err.status  : 500;
  const message = err instanceof BlazeError ? err.message : "Internal Server Error";
  const meta    = err instanceof BlazeError ? err.meta    : {};

  res.status(status).json({ error: message, ...meta });
});

Workers AI usage is metered by Neurons — Cloudflare’s unit of inference compute. Costs and rate limits vary by model and plan. Monitor your consumption and configure spending limits in the Cloudflare dashboard → Workers AI section. Streaming responses consume the same number of Neurons as non-streaming calls for the same input and output length.