Independent analysis of models, agents, automation and research—written for people responsible for what ships, what it costs and what can fail.
The TypeScript client rejected a bad structured result after listTools—but accepted it before the schema cache existed
In an official MCP TypeScript SDK 1.30.0 probe, Client.callTool accepted a string where a tool promised an integer until listTools populated the output-validator cache. Discovery order is therefore part of the client’s validation boundary.
Tool annotations describe intent; trusted code must still authorize effects
In an official MCP TypeScript SDK 1.30.0 probe, a tool advertised readOnlyHint: true and idempotentHint: true, then wrote a file on both calls. The SDK faithfully transported the hints and faithfully dispatched the mutations.
Lifecycle suppression and executable linking are different controls
In npm 11.6.0, a synthetic package installed by npm ci --ignore-scripts did not run its install hook, but npm still created node_modules/.bin/aag-ci-probe. Invoking that link executed the package. --bin-links=false removed the shim, not the underlying executable file.
A clean install can become code execution when a later rebuild drops the flag
In npm 11.6.0, I installed a synthetic tarball with --ignore-scripts and observed no hooks. A later ordinary npm rebuild ran its preinstall, install and postinstall scripts. The install flag did not persist as package approval state; script policy must cover every later build command.
We separate vendor claims, public evidence and our own recommendations. When evidence is thin, we say so.
Capability, cost, deployment and model-selection decisions.
Explore desk →02Coding agents, tool use, orchestration, control and reliability.
Explore desk →03Operational workflows that survive contact with production.
Explore desk →04What research changes, what it proves and what teams can use now.
Explore desk →05Evaluation design, reproducibility and evidence beyond leaderboards.
Explore desk →06Evidence-led frameworks for expensive AI tooling decisions.
Explore desk →Primary sources first
Reproducible methods
Limitations in plain view
Commercial ties disclosed
Research updates, evaluation frameworks and production lessons for technical leaders.
A local reproduction of the difference between “do not write the tarball” and “do not run code”
In npm 11.6.0, a synthetic package’s prepack, prepare and postpack hooks all ran under npm pack --dry-run. The tarball file was not written, but the scripts changed the working tree. Use --ignore-scripts and an isolated builder when inspection must not grant package code authority.
Source-level evidence for pinning lifecycle-script authority to package versions
pnpm 11.24.0 blocks unapproved dependency builds, but its normal approve-builds flow records a bare package name. The pinned implementation treats that name as approval for every registry version. Exact name@version rules are supported and give CI a narrower execution boundary.
A clean-room probe of allowScripts, allow-git and the temporary-clone build path
npm 12.0.2 correctly rejected a git dependency until I enabled git fetching. After that, an unapproved prepare hook still ran twice inside npm’s temporary clone even while npm warned that the package’s prepare script had been blocked. The implementation explains why --ignore-scripts remains the stronger boundary for untrusted git dependencies.
Lifecycle hooks turn dependency installation into shell execution
A local file dependency with preinstall, install and postinstall hooks ran all three hooks during npm install in my sandbox. Re-running with --ignore-scripts suppressed the hooks entirely, which makes the real question obvious: who gets to execute code during installation?
A deployment decision for local servers, relays and downstream resources
If an MCP server can relay the same bearer token it received, the server becomes part of the authorization plane. The safer pattern is audience-bound resource tokens, explicit resource metadata and a refusal to treat localhost as the trust boundary.
Node 26.7.0 adds audit mode, but the security boundary still lives outside the runtime
Node’s stable permission model still catches accidental filesystem and process access, and v26.7.0’s new --permission-audit mode is useful for rollout rehearsals. Our local probe showed the same diagnostics-channel event in both modes; only enforce mode denied the read.
A go/no-go review for tool access, identity and containment
MCP standardizes how an AI application reaches tools; it does not decide which authority the application should receive. Use these gates before a server can touch production data or actions.
A migrate, sandbox, self-host or wait decision for Z.ai’s post-trained coding model
GLM-5.3 is a timely coding-agent candidate, not an automatic GLM-5.2 upgrade. Trial the managed model in an isolated repository workflow, migrate thinking settings explicitly, contain network and exploit authority, and wait for the actual weights and safety artifacts before approving self-hosting.
A build, buy and deploy decision for OpenAI’s limited-preview Ultrafast API mode
GPT-5.6 Sol Ultrafast is an emerging serving option, not a blanket model migration. Trial it only on latency-critical paths where saved time has measured value, quality remains equivalent locally, tier delivery is observable, and fallback to Standard is safe.
A release decision for retiring model IDs and replacement versions
A replacement model is a new configured system, not a dependency patch. Inventory every route, freeze the decision contract, shadow the replacement, and migrate only the scopes that clear explicit quality, safety, cost and rollback gates.
Approve, constrain, test or reject one configured data path
“No training” is not a retention policy, and a region selector is not a complete data map. Use this evidence-led review to gate an AI API on storage, processing, logs, application state, transfers and deletion.
Build, buy or extend without trapping the evidence in one dashboard
Choose an LLM evaluation platform by what you can export, reproduce and migrate—not by grader count. Own the cases, decision rules, provenance and case-level results before adopting a workflow or dashboard.
An implementation decision for agent and automation controls
Decide which AI actions need human approval, what evidence a reviewer must see, and when approval is too weak to make an unsafe authority path deployable.
A build, buy or hybrid decision for production AI telemetry
Decide what evidence an AI system must produce, which content may be retained, and whether your existing stack, a specialist platform or a hybrid can meet the operating contract.
A security decision for RAG, agents and tool-using AI systems
Treat direct and indirect prompt injection as an authority-design problem. Decide whether to deploy, constrain or reject an LLM workflow with explicit data, tool, approval and evaluation gates.
A deterministic workflow, bounded agent or no-automation decision
Do not buy autonomy for a process that can be expressed and controlled as a workflow. Use this evidence-led framework to choose deterministic code, an LLM-assisted workflow, a bounded agent—or no automation.
A build, buy or bounded-trial framework for one AI workload
Compare a managed model API with a self-hosted open-weight stack on accepted outcomes, full operating cost, data boundaries, control and exit—not token price or infrastructure ideology.
A production decision for knowledge, behavior and combined systems
Retrieval and fine-tuning solve different failure classes. Use this evidence-led decision to choose prompt-only, RAG, fine-tuning, a measured combination—or no LLM change.
A workload evaluation and deployment decision for technical teams
Public benchmarks can shortlist candidates; they cannot decide which configured AI system is acceptable for your workload. This guide turns model selection into a reproducible ship, trial or reject decision.
A production-readiness framework for engineering leaders
A coding agent should earn autonomy through controls and repeatable evidence—not through an impressive demo. This framework turns repository access into a staged engineering decision.
An evidence ladder for technical teams
How to decide whether a research result deserves a prototype, a controlled trial or no engineering time yet.
Measure systems, interventions and failure—not demos
A benchmark design for teams evaluating tool-using agents on their own operational tasks.
A buyer guide for engineering and AI leads
A vendor-neutral scorecard covering evidence, integration, security, economics, operations and exit risk.