Judge config

How a Broods agent gets a Jev model, and how a hook calls it. Nothing here is built yet.

Config

Set it once on the account. Every agent uses it.

judge: {
  apiKey: "${AI_GATEWAY_API_KEY}",
}

That is the whole common case. The rest defaults to the AI Gateway and typesafe-ai/jev, which is the only evaluation model the Gateway lists from any provider.

An agent that needs something different writes its own block, and that block replaces the account one:

judge: {
  provider: "typesafe",
  modelId: "jev-1.13.0",
  apiKey: "${TYPESAFE_AI_API_KEY}",
}

Replace, not merge. A half-merged credential fails in a way nobody finds until production.

FieldDefaultNotes
apiKeyaccount env${ENV_REF} resolved at sync time, like every other credential. It lands in the encrypted config, never in an env var on the box.
modelIdtypesafe-ai/jevSet it the day jev-1.14 ships and you want to stay on 1.13.
providergatewaygateway, typesafe or custom. The Gateway cannot pin a version (#21213), so pinning means typesafe.
baseURLnoneOnly valid on custom. It typechecks nowhere else, rather than being accepted and ignored.
headersnonePassed through as given.
enabledfalseOff means every call returns null and nothing changes.

Only gateway runs on what is already installed. typesafe and custom pull in @ai-sdk/typesafe-ai, which is one line in resolveJudge and no change to any call site.

Account default plus agent override, rather than a Convex record referenced by id the way sandboxes, workspaces and hook bundles work. A table to hold one row is not worth its CRUD. If judges ever multiply, that pattern is there to copy.

What a hook author writes

Nothing about config. They call it.

import { defineAgent } from "broods";

export const agent = defineAgent({
  name: "guarded-agent",
  model: { provider: "minimax", modelId: "MiniMax-M3" },
  hooks: {
    onToolCall: async (ctx, event) => {
      const { answers } = await ctx.judge({
        state: event,
        questions: {
          risky: {
            type: "boolean",
            instructions: "Destroys data outside the workspace",
          },
        },
      });

      return answers.risky.probability > 0.8 ? { decision: "deny" } : {};
    },
  },
});

Built once

An evaluation model is a stateless object: a provider name, a model id, the question types it accepts, and one doEvaluate method. Nothing to close, nothing to pool. So core builds it once per turn in createTurnContext, next to resolveConfiguredModel, and hands it to whoever asks.

// createTurnContext, next to the language model
const judge = resolveJudge(agentConfig);   // undefined when judging is off

The boundary

That object cannot cross into the V8 isolate. Methods cross as references, which is why ctx.fetch is a bridged host call and not a real fetch. Core keeps the object and the key. The hook gets a function.

CORE (HOST) resolveJudge(agentConfig) keeps the model keeps the API key enforces the timeout HOOK BUNDLE (ISOLATE) ctx.judge({ state, questions }) no key, no provider, no config cannot import `ai` at all bare imports rejected at upload state + questions answers

The one rule

A judge answer may narrow what the agent does. It may never grant something it did not already have.

Core already holds this rule in applyChannelRecord, for the same reason. Jev does not treat its state as hostile, and hook state is a channel message anyone can type into. A classifier that can only subtract capability gives an attacker nothing.

fine deny a tool call

The approval rules said yes, the judge said wait. One blocked call.

never approve past a deny

Auto-approve sits inside the existing rules. It does not replace them.

fine wake on an unmentioned message

The regex said no, the judge says the channel is asking for help. One extra run.

never suppress an explicit mention

Someone typed @agent. Return before the call is even made.

Where it can run

Broods fires twelve hook events. Eight of them let a hook change something, so those are the eight where a judge answer can do work.

HookMay changeWhat a judge call does there
onMessageReceiveddrop, textAnswer a message that forgot the @mention. Drop the noise.
onToolCalldecision, args, denyReasonDeny a destructive call. The example above.
onApprovalRequiredapproveApprove the routine ones so a human sees only what matters.
onMessageSendingdrop, textRead the reply before it reaches the channel.
onStartsystem, messagesAdd a skill hint, or trim what the model is about to read.
onFinishoutputRewrite or withhold the final answer.
onToolResultoutputFilter 300 grep hits down to the ones that matter.
onSubagentFinishvisibleResultScore the child's work before the parent reads it.
agent.step.finished, agent.failed, tool.call.finished and subagent.task.started are observe only. A judge call there can record a score. It cannot act on one.

not available  There is no sandbox hook and no connection hook. Sandbox lifecycle lives in core's account handler, outside the hook system. Channel connections live in the Discord and Matrix forwarders, which have no hook dispatch and forward everything so core decides. Judging either one means adding hook events first, which is a bigger job than this and worth its own decision.

What changes

FileChange
apps/core/src/harness/judge.tsnew. resolveJudge and one judge function
apps/core/src/shared/domain/agent-config.tsAgentJudgeConfig, and judge?: on AgentConfig
packages/convex/model/agentRules.tsnormalizeJudgeConfig(config.judge), beside the other branches
apps/core/src/harness/session.tsbuild it once in createTurnContext
isolate/runner/runner.mjs and isolate/executor.tsone more field on __ctx, beside fetch
packages/broods, apps/docs/docs/hooks.mdctx.judge on the hook context type, one docs section. ctx goes from three things to four.

Still open

  1. Timeout. A judge call in onMessageReceived runs ahead of every inbound message. Fix it at 400 ms in core rather than let a hook author choose, so one slow hook cannot stall a channel.
  2. Failure shape. ctx.fetch throws, and the hook runner already reads a throw as no change. Throwing matches that. Returning null reads better for the rule above, because ?? 0 falls through to the safe answer. I lean null.
  3. Budget. Nothing stops a hook calling judge in a loop. Hook results and state already have byte caps, so a per-run call cap fits.

Broods, 22 September 2026. Mock only, nothing built, working tree clean on dev. Companion artifact: Jev in Broods.