Skip to content

Data contract

TokenShift has no behaviour-changing control plane. Customers configure the binary at install time via the local enrollment manifest deployed by their MDM; PointFive only ingests data and controls which release version is offered to a given client. See Installation & distribution for the install model.

Every record is hybrid-encrypted on the device against PointFive’s public key (pinned in the binary at build time) before it leaves the machine. Before anything is sent, sensitive content is removed on the device: known secret formats and the values inside the most sensitive fields are replaced with one-way hashes (commands keep their shape only). This page answers, in order: what leaves the device at all, what each tier controls, how long it’s kept, and what’s never sent under any configuration — with the full field-by-field detail as a reference at the bottom for anyone who needs it.


TokenShift sends two kinds of records, both hybrid-encrypted on the device before they ship:

  • Typed signals — four small, schema’d aggregates TokenShift computes itself (tool_invocation, recovery_retrieved, compression_sample, client_state). Field-by-field detail: Reference → Typed signals.
  • Raw hook events — a Tier 2+ addition: the redacted payload each agent harness hands to its own hooks, forwarded under that hook’s native name (PreToolUse, UserPromptSubmit, afterShellExecution, and so on), plus context TokenShift derives locally (git branch, token counts, hostname, prompt intent). Field-by-field detail: Reference → Raw hook events and Reference → Enrichment.

Nothing else leaves the device. See What the contract does NOT send for the explicit list of content types that never ship in clear.


A single integer set by user choice and capped from above by install-time config. Effective tier on the wire is min(user_choice, admin_cap). Each higher tier strictly adds.

TierCLI mode nameWhat leaves the laptop
0(config telemetry disable, not a mode)Nothing leaves the machine.
1metricstool_invocation (no shape fields), recovery_retrieved, client_state.
2patternsTier 1 + command_shape and rewritten_command_shape + raw hook events (redacted base + enrichment) + harness_user_account and git_user_email (see Identifiers).
4samplesTier 2 + compression_sample. Triple-gated; off by default even when permitted.

Tier 3 is reserved internally and unreachable in practice — any attempt to set it silently clamps to Tier 2. The named modes are what tokenshift config telemetry mode <name> actually takes — see Core commands.

Default tier when enrolled: 2 (patterns). tokenshift enroll --tenant-id=<id> caps and emits at Tier 2 unless overridden with --tier-cap or local telemetry config.


Even with raw-event collection at Tier 2, the wire never carries any of the following in clear. The redactor removes them on the device before encryption:

  1. Bash stdout/stderr, tool output, prompts, messages, file contents — these field values are SHA-256 hashed on the device; only the hash is sent. The hash is stable (identical content correlates), but the content is not recoverable from it.
  2. Environment variable names or values.
  3. Raw file-path values — path / cwd / transcript-path fields are hashed.
  4. Raw command bytes — commands ship as redacted shape only.
  5. OS usernames, real names, IP addresses, MAC addresses.
  6. Raw email of the git identity, below Tier 2 — at Tier 1 it is read locally to compute user_id and nothing else leaves. From Tier 2 the lower-cased address itself ships as git_user_email, beside the harness’s logged-in account harness_user_account: the two deliberate exceptions, see Identifiers.
  7. Rule contents — only the rule’s identifier.
  8. Commit messages.
  9. Prompt text used for intent classification — only the resulting label (e.g. “debug,” “feature work”) ships, via enrichment.

For completeness, what the wire does carry from the above: stable SHA-256 hashes of those content fields (for correlation, never recovery), and — via enrichment — git context (branch name, repo remote URL, PR number), machine hostname, and prompt-intent labels. It also carries two identity fields in clear from Tier 2 up — harness_user_account and git_user_email, the only identity fields that are not hashed, described under Identifiers.


RecordDefault retentionTenant-extensible?
tool_invocation90 daysYes
recovery_retrieved90 daysYes
client_state90 daysYes
Raw hook events90 daysYes
compression_sample7 daysNo — hard cap.

The 7-day hard cap on compression_sample is non-negotiable. Samples are the only signal that carries redacted output content; the rule-tuning use case doesn’t benefit from longer retention, and every extra day is unnecessary blast radius.


Everything below is the detailed, field-by-field version of what’s summarized above — useful for a full audit, not required reading to understand the contract.

TokenShift computes four typed aggregate records:

#SignalMin tierFires when
1tool_invocation1The agent makes a tool call.
2recovery_retrieved1A developer pulls back output that was compressed away.
3compression_sample4Triple-gated, off by default — a redacted before/after pair for rule tuning.
4client_state1Periodically (≤ once / 24h per machine) — version + install metadata.

tool_invocation — one per tool call, regardless of whether TokenShift compressed it. Carries tool kind, agent (which harness — including codex now), model (raw model string included), a tool-call join id, tool duration, activity bucket, project (raw directory name by default; can be hashed), a stable command_hash, original token count, and an outcome enum. At Tier 2+ also carries redacted command_shape and rewritten_command_shape. No raw command bytes ship at any tier.

recovery_retrieved — fires when a developer pulls back the original output via the recovery command. Carries only a join key, the recovering session, and an age in seconds. No content. No file paths. Note: the recovery cache itself isn’t available on every harness — see Recovery cache scope below — so this signal simply never fires from a Cursor or Codex client.

compression_sample — a redacted before/after pair for PointFive rule tuning. The only signal that carries actual output content — and the most tightly gated record in the contract. Off by default. Three independent gates must all be true: tenant has enabled Tier 4, the matched rule is on the per-rule sample allowlist, and a random draw lands inside the configured sampling percentage. Carries no user_id (defense in depth). Hard-capped retention: 7 days.

client_state — periodic meta-telemetry about the binary itself: version, OS, architecture, install method, enrollment age, and whether the startup self-check passed. Rate-limited to at most once per 24 hours per client_id. No PII, no per-tool-call detail. This is how PointFive and admins see fleet hygiene.

Raw hook events are a Tier 2 addition: at Tier 2 and above, TokenShift also forwards the hook events its supported agents emit — the same payloads the harness hands to any hook — after redaction. Each becomes one record whose signal name is the native hook event (e.g. PreToolUse, UserPromptSubmit, afterShellExecution). Tiers 0 and 1 do not include them.

Coverage by agent (six harnesses today — Claude Code, Cursor, GitHub Copilot CLI, Codex, opencode, and Devin CLI):

AgentHook events forwarded
Claude CodePreToolUse, PostToolUse, PostToolUseFailure, SessionStart, SessionEnd, Stop, StopFailure, UserPromptSubmit, UserPromptExpansion, PreCompact, PostCompact, PostToolBatch, SubagentStart, SubagentStop, PermissionRequest, PermissionDenied, Notification, Setup, InstructionsLoaded, MessageDisplay, TaskCreated, TaskCompleted, TeammateIdle, ConfigChange, CwdChanged, FileChanged, Elicitation, ElicitationResult
CursorbeforeShellExecution, afterShellExecution, beforeMCPExecution, afterMCPExecution, beforeReadFile, afterFileEdit, beforeSubmitPrompt, sessionStart, sessionEnd, preCompact, stop, afterAgentResponse, subagentStart, subagentStop, postToolUse (Delete only), postToolUseFailure, workspaceOpen
GitHub Copilot CLIpreToolUse, postToolUse, postToolUseFailure, userPromptSubmitted, sessionStart, sessionEnd, errorOccurred, permissionRequest, preCompact, agentStop, subagentStart, subagentStop, notification
CodexSessionStart, UserPromptSubmit, PreToolUse, PostToolUse, PermissionRequest, PreCompact, PostCompact, SubagentStart, SubagentStop, Stop
opencodePreToolUse, PostToolUse, Stop
Devin CLIPreToolUse, PostToolUse, UserPromptSubmit, Stop, PostCompaction, SessionStart, SessionEnd, PermissionRequest

Claude Code’s surface is the largest and grows fastest — it’s the harness with the richest native hook set, and TokenShift forwards nearly all of it (two internal delegation hooks are excluded on purpose). Codex speaks a Claude-shaped protocol but exposes a smaller hook surface today — notably no native SessionEnd. opencode is smaller again: its plugin forwards the two tool events plus a Stop synthesized from session.idle, its closest terminal signal. Its permission gate is not in the list because it decides allow/block and emits no record. Devin CLI also speaks the Claude-shaped protocol, with a fixed set of eight events and no model, working directory, or transcript path in the payload; TokenShift derives its model and token usage from the session store the CLI keeps locally. Devin reports an MCP call under the mcp__<server>__<tool> name Claude Code uses; its mcp_list_tools discovery call carries the server in tool_input.server_name. Devin’s PreToolUse, PostToolUse and PermissionRequest also ship tool_input.file_path for every tool that sends one, reads included, hashed like every file path. In the warehouse, Devin’s prompts, turns, tool and MCP calls, tokens and model roll up like every other harness’s; cost reads zero until its models appear in the pricing feed.

Each raw event carries:

  • base — the harness’s hook payload, redacted on the device (see Redaction). Fields the harness includes that aren’t sensitive pass through as-is.
  • enrichment — context TokenShift derives locally (see Enrichment).
  • Dimensions — tenant_id, client_id, agent, session_id, binary version, and a timestamp.

Raw hook events carry no user_id — they are scoped to client_id. From Tier 2 they do carry harness_user_account and git_user_email (in enrichment, since base is the harness’s own payload); see Identifiers.

tokenshift recover <hash> only works when the compressing proxy runs unsandboxed and can write to the local ~/.tokenshift/recovery cache:

AgentProxy sandboxed?tokenshift recover works?
Claude CodeNoYes
GitHub Copilot CLINoYes
CursorYesNo
CodexYesNo
opencodeNoYes
Devin CLINoYes

Cursor and Codex both run the compressing proxy inside a sandbox that can’t write outside the workspace, so the recovery cache write silently fails and no [Full output ...: tokenshift recover <hash>] hint is ever emitted for those sessions — compression still happens, the saved-output pointer just isn’t available. Compression stats for both still ship normally, handed off through a short-lived workspace-local file instead of being recorded in-process. See Harness coverage for how the harnesses differ.

For some events TokenShift computes extra context on the device and attaches it under enrichment. Apart from harness_user_account, git_user_email, hostname, and machine_id below, it carries no identity data.

EnricherRuns onSends
gitsession start, prompt submitbranch (current branch name), repo (origin remote URL with any embedded credentials stripped), and — on prompt submit — pr (current PR number, via gh)
path_gitSuccessful shell (Bash, PowerShell), Edit and Write tool results, cwd change (Claude Code); file edit, successful Delete (Cursor); file edit tool results (create, edit, apply_patch; success only, Copilot); bash, edit, write and apply_patch tool results (success only, opencode); edit, write, apply_patch and exec with a workdir tool results (success only, Devin CLI)For each file or directory path in the event, the git context of the repo holding it: repo (origin remote URL with any embedded credentials stripped) and branch, plus a status (resolved, not_in_repo, no_remote, detached_head, dir_missing, relative_unresolved, timeout, git_missing, error). Entries sit under paths, keyed by the event’s field name (e.g. tool_input.file_path), never by the path itself; the path value follows the event’s file-path redaction (hashed through Tier 2). Each file an apply_patch edits is keyed apply_patch.paths[i], in patch order. A Cursor event with no cwd anchors relative paths at its workspace root. With several roots, a path resolves only when every root that votes gives the same repo and branch, otherwise it is relative_unresolved. For a file edit, the roots holding the file vote; for a Delete, the roots that have its parent dir but no longer the file. A timeout, git_missing or error in any root, including a failed stat, is reported instead of the vote. Roots inside the agent’s own files are dropped before the roots are counted, so [~/.cursor, /repo] acts like [/repo]. Only a Delete’s own cwd replaces the roots; a cwd on a file edit does not. Each agent’s own files are skipped: Claude Code’s config dir, scratchpad, plans and transcripts; Cursor’s ~/.cursor, its config dir (CURSOR_CONFIG_DIR, else $XDG_CONFIG_HOME/cursor), its data dir (CURSOR_DATA_DIR) and app data, except its worktrees (~/.cursor/worktrees and CURSOR_WORKTREES_ROOT), which are user checkouts; Copilot’s home (COPILOT_HOME, else ~/.copilot) and cache dir (COPILOT_CACHE_HOME, else the OS cache dir + copilot); a relative value is anchored at the event’s cwd, as Copilot does; opencode’s config, data, cache and state dirs ($XDG_CONFIG_HOME, $XDG_DATA_HOME, $XDG_CACHE_HOME, $XDG_STATE_HOME + opencode, each else under the home dir), OPENCODE_CONFIG_DIR and the temp dir + opencode, except its worktrees (data dir + worktree), which are user checkouts. That skip includes opencode’s clone cache of reference repos (data dir + repos): those are copies of remote repos, not the user’s work. Other Copilot tools (shell, read, search, MCP, extensions) get no path_git. An opencode bash call resolves its cwd (the session directory) and tool_input.workdir (where the command ran). An opencode edit or write resolves tool_input.file_path, and an apply_patch resolves each file it edits. The plugin anchors a relative workdir or file path at the session directory and does not expand ~, as opencode does. The patch never leaves the device. The tool output follows the command-output redaction. Other opencode tools (read, search, web, MCP) get no path_git. Devin CLI’s own files are skipped too: its config dir, its data dir and ~/.devin. Its payloads carry no cwd, so a relative path anchors at DEVIN_PROJECT_DIR, else CLAUDE_PROJECT_DIR, else the hook’s working directory, the same rule as for the session’s git context. Devin sends exec’s workdir only when agent.codex_tools is on; an exec without one, and other Devin tools (read, search, MCP), get no path_git. Also enricher_duration_ms. Not sent when telemetry is off. Claude Code, Cursor, Copilot, opencode and Devin CLI.
tokensprompt submit, tool-result events, agent-response events, turn end, subagent stop (Claude Code)input_tokens / output_tokens (no content). Tokens are labelled from the agent’s perspective: text fed into the agent’s context — the user prompt and tool results (observations) — is input; text the agent generates — its replies and reasoning — is output. A source field marks each count as estimate (a per-payload approximation) or transcript (an authoritative count parsed from the harness’s own session log), or a store name (opencode_db, devin_db) when the count comes from the harness’s local database. Claude Code, Copilot CLI, Codex, opencode, and Devin CLI each read their own harness-specific record — a session log, or the local store opencode and Devin CLI keep. On Claude Code a subagent’s stop reads that subagent’s own transcript, so every API call it made is counted, not only its final one; Cursor has no token enricher today, so its per-turn counts arrive on the hook itself and carry no reconciled count. Claude Code messages also carry thinking_tokens (a subset of output_tokens, not an addition), the reasoning-effort tier (effort, per_turn_effort, advisor_model) and the serving scalars speed and inference_geo — counts and settings only, no content.
prompt_classesprompt submitLabels for patterns local regexes matched in the prompt — asking for several approaches, vague wording, a pasted error, and so on. Only the labels ship, never the prompt text. A prompt that isn’t a person’s own (a harness-generated one, say) is tagged as such instead. Not wired on opencode today.
hostnamesession startThe machine’s hostname — used for fleet-hygiene context, not tied to a person.
harness_user_accountevery raw event, Tier 2+The harness’s own logged-in account, in clear — see Identifiers. Absent below Tier 2 and when the harness is logged out.
git_user_emailevery raw event, Tier 2+The lower-cased git config --global user.email, in clear — the address behind user_id; see Identifiers. Absent below Tier 2 and when git has no email. Cached beside user_id under ~/.tokenshift (mode 0600) and refreshed from git once a day, so no hook or proxied command waits on a git spawn.
machine_idevery raw eventA one-way hash of the OS machine GUID, identifying the machine across reinstalls — never a person, and never the raw GUID; see Identifiers. Computed once and read from a local file thereafter, so reading the GUID cannot delay a hook.
telemetryevery raw eventtier — the telemetry tier the event was redacted at — and tier_cap, the admin’s cap from the manifest. Two numbers, no content.
intentprompt submitA locally-computed classification of what the prompt is trying to do (e.g. debug, feature work, refactor), produced by a small on-device model — see How it works. The classification label ships, never the prompt text itself.
prompt_classesprompt submitWhich recognisable patterns a prompt matches — for example asking for several alternative approaches, asking for repeated verification, asking to think harder, widening the scope mid-task, or including a file path, a test name, an error message or a command. Matched locally with simple patterns, no model involved. Pattern names only, never the prompt text. Prompts not written by the user (slash commands, interruptions, harness notifications) are tagged as such instead.
contextsession startWhat Claude Code loads into every conversation, as reported by its own /context command: the system prompt, built-in tools, MCP servers, CLAUDE.md and memory files (with their paths), skills and custom agents — each with its token size — plus the model and the total. Names and sizes only, never the contents of a file, skill or tool. If the report could not be collected, a short reason ships instead of the sizes. Claude Code only.

Devin usage records:

  • enrichment.tokens.schema_version = 2 carries request records in messages, with source = devin_db.
  • Each record includes request and message IDs, source and observation timestamps, model, token buckets, source revision, and tool-call IDs and names.
  • Prompt ancestry includes the source prompt ID and timestamp when available; prompt text, replies, and tool arguments are excluded.
  • Live hooks and backfill use the same records, including consumed usage from discarded branches.
  • Warehouse deduplication keeps the latest revision per tenant, session, and request; aggregate token fields are compatibility projections.
  • Each request is priced using its own model and source timestamp before costs are summed; model_mixed identifies turns containing multiple known models.
  • Usage matches backfilled prompts by stored ID. Native hook IDs differ, so live turns are inferred from the source prompt timestamp within observed prompt intervals and marked source_prompt_time.
  • Usage without an ID or source-time match remains in turn_seq = -1, marked usage_attribution = unattributed, with zero active duration.
  • Upgrades preserve already-counted cumulative usage as legacy_interval. Existing requests retain an immutable legacy_baseline, so later corrections count only the difference.

Every knob a tenant admin sets when deploying TokenShift via MDM. There is no runtime config push — changes require regenerating the manifest from the PointFive app and redeploying via MDM. No user-level overrides.

ControlDefaultEffect
Tier capTier 2Maximum tier any client in the tenant can emit.
Project identifier modeRaw directory nameOr hashed directory name, or hashed git remote URL.
Recovery cache enabledOnOff disables the local short-lived cache (already unavailable by default on Cursor/Codex — see Recovery cache scope).
Sample allowlistEmptyPer-rule list of which rules may be sampled at Tier 4. Empty = no samples ever emitted.
Sample rate0%Per-rule sampling probability used at Tier 4.
Retention extensionsDefaults aboveNegotiated per tenant. compression_sample is hard-capped regardless.
  • tenant_id — customer organization, from the enrollment manifest.

  • client_id — random UUID created on first run and persisted under ~/.tokenshift/. Reused across runs, survives upgrades and re-enrollment. One per OS user account. Present on every record, including raw hook events.

  • machine_id — reinstall-surviving endpoint identifier: a one-way SHA-256 hash of the OS-native machine GUID (macOS IOPlatformUUID, Linux /etc/machine-id, Windows MachineGuid), with a machine: domain-separation prefix. Unlike client_id — which is random and regenerates if ~/.tokenshift/ is wiped or the CLI is reinstalled — the same physical machine reproduces the same machine_id, letting the warehouse re-link a re-imaged or reinstalled endpoint to its prior identity. The hash is not keyed per tenant, so it also survives re-enrollment; the same machine reports the same value under any tenant. The raw GUID is never sent and never persisted — only the hash leaves the device. Carried on the enrichment attached to raw hook events (alongside hostname/username). It is computed once — at install, or otherwise on the first background re-run after an upgrade — then written to ~/.tokenshift/machine_id and read from that file on every event, so reading the GUID never delays a hook the developer is waiting on. Present on every hook event from a machine with telemetry on; absent when telemetry is off and when the OS GUID is unreadable.

  • user_id — pseudonymous user identifier derived locally as a one-way hash of the developer’s email under a per-tenant key:

    user_id_hmac_key (random secret, generated at enrollment, lives in manifest)
    │
    ▼
    email ─── one-way hash ──▶ user_id (ships on the wire)
    ▲ the hash is cached under ~/.tokenshift;
    │ the email itself is never persisted.
    git config --global user.email
    (fallback: $TOKENSHIFT_USER_EMAIL; otherwise user_id = null)

    Same email + same tenant key → identical user_id on every machine that developer uses, enabling cross-machine joins. user_id is present on the typed signals (except compression_sample and client_state) and absent from raw hook events.

  • git_user_email — the lower-cased git email itself, the clear-text twin of user_id. Shipped in clear from Tier 2 up: on the typed signals beside user_id (always present, null below Tier 2 or when git config --global user.email is unset) and under enrichment on raw hook events. Cached beside user_id under ~/.tokenshift (mode 0600) and refreshed from git once a day, so no hook or proxied command waits on a git spawn. It exists so the warehouse can attribute an endpoint whose harness reports no account (Claude Code behind Bedrock or a gateway, Devin, opencode), and so one row carrying both fields resolves every historical row that carried only the user_id.

  • harness_user_account — the account the coding agent itself is logged in as: an email for Claude Code, Codex, and Cursor; the GitHub login handle for Copilot CLI; not yet available for Devin CLI, whose credential store holds no account identity. Shipped in clear, not hashed — one of the two deliberate exceptions to the hashing rule in this contract (the other is git_user_email), made because downstream features attribute and verify a real user against it, and a hash of a known employee list offers no meaningful protection. Read from the harness’s own local config (no network call) on every event, in memory; nothing about it is written to disk.

    Only from Tier 2 (patterns) up, enforced at the wire boundary for both clear identity fields: a tenant capped at Tier 1 (metrics) never emits either. Absent whenever the harness is logged out or its account store can’t be read. Lower-cased and trimmed on the device since 0.7.26; older rows are verbatim, so compare case-insensitively. It rides in enrichment on raw hook events only; the typed signals carry git_user_email instead.

  • project — raw directory name by default; install-time config may switch to hashed directory name or hashed git remote URL.

  • session_id, invocation_id, recovery_id — opaque UUIDs used as join keys between signals.

OS usernames, IPs, MACs, and raw file-path values are never sent in clear — file-path fields are among the hashed sensitive groups (see Redaction). The developer’s git email is hashed into user_id at every tier, and from Tier 2 also ships in clear as git_user_email. The identity fields that ship in clear are harness_user_account and git_user_email, both at Tier 2+, as described above. The git repo URL and branch name are sent via enrichment as described above, including for the file paths in the events listed under path_git, except each agent’s own files.

Redaction runs on the device, before encryption, over both the typed signals’ shape fields and every raw hook event’s base. Every field TokenShift might ever emit is typed to a data group (prompt text, file contents, command, file path, URL, identifier, and so on), and a per-group action — Remove, Hash, Redact (kept as a shape, not raw), or Pass — is applied at the effective tier:

  • Content and I/O fields (prompt, command, content, output, stdout, stderr, tool_input, tool_response, result, message, …), search/egress inputs (query, pattern, url, …), identifying paths (file_path, path, cwd, transcript_path, …), and PII (user_email) all have their values replaced before send.
    • command is shape-redacted: structure kept, argument values stripped (its shape carries analytics value).
    • Every other sensitive value is replaced with a SHA-256 hash. The hash ships; the original does not.
  • patch (an apply_patch body) is never sent at any tier; only the files it edits feed path_git.
  • Known secret formats are stripped wherever they appear.
  • Fields not modeled into a sensitive data group pass through as sent by the harness. Only modeled fields ship at all — an unmodeled field or event drops rather than shipping unredacted by accident.

A startup self-check runs representative inputs through the redactor; if it fails, telemetry is disabled for that process (fail-closed).

Anything that doesn’t go right ends in a drop or a disabled producer, never a raw emit.

TriggerAction
Redactor startup self-check failsTelemetry disabled for the process — nothing sent.
Envelope tampered (tenant / signal / schema mismatch in the bound headers)Rejected at decrypt (HTTP 400); never persisted.
Encryption / seal failsRecord dropped.
Tier below a record’s minimumNot emitted.
Sample allowlist or sample-rate missSample not emitted.
Local outbox overflowOldest records dropped first; the developer’s command is never blocked.
Network unavailableRecords queued locally until network returns.
Enrollment manifest absent or unparseableTelemetry off; nothing sent.
  • Typed signal — a named, schema’d aggregate record (one of the four above).
  • Raw hook event — a redacted copy of a hook payload an agent harness emits, forwarded under the hook’s own event name (Tier 2+).
  • Enrichment — context TokenShift derives locally (git, token estimates, hostname, prompt intent) and attaches to raw events; from Tier 2 it also carries the two clear identity fields.
  • Redaction — device-side removal of sensitive field values (SHA-256 hash, or shape for commands) before encryption, driven by a per-field data-group classification.
  • Fail-closed — on any error, drop the data (or disable the producer) rather than emit it raw.
  • Tier — level of data sharing (0/1/2/4). Capped from above by install-time config.