--- title: "Token efficiency" output: rmarkdown::html_vignette vignette: > %\VignetteIndexEntry{Token efficiency} %\VignetteEngine{knitr::rmarkdown} %\VignetteEncoding{UTF-8} --- Keep data and computation in R, and send models enough context to choose useful actions. An object around 4 GB can remain in the same R session across turns: the model sees a compact description while its R tool computes on the object. No repeated export and reload cycle is required; R copies, calculation time and model-request latency still depend on the workflow. Four practices help: - Refer to live objects with a compact description rather than printing large tables into a prompt. The model can compute on those objects with the `r` tool. - Compose several deterministic operations in one R call and return a bounded result. - Reuse a session's stable prompt and growing conversation when the provider supports prompt caching. Changing models, tools, compaction or rewind can change that prefix. - Inspect usage and set budgets before starting a costly workflow. The executed examples use fake providers. Their request counts and estimates demonstrate the interfaces; they are not measurements of a real model's speed, quality or bill. ## Inspecting the prompt ``` r library(gptr) view = gptr_prompt(preset = "minimal") attr(view, "tool_names") #> [1] "read" "r" "edit" "write" view$sections #> name tier tokens #> 1 preamble T0 42 #> 2 tools T0 86 #> 3 rules T0 306 #> 4 modes T0 84 #> 5 context T0 152 view$total_tokens #> [1] 1834 ``` `gptr_prompt()` inspects the system sections and tool definitions with estimated token counts; printing it shows their text. It does not query a provider. The `standard` preset describes most capabilities as functions in `peter$`, with the core model-visible `r`, `read`, `edit` and `write` tools and `ask` when applicable. The source benchmark's recorded o200k static-prefix counts are 1,262 for `minimal`, 2,335 for `standard` without optional sections, 2,813 with every section, and 2,956 with `ask` as well. These are measured fixture counts, not a promise about every runtime context or tokenizer. The first user message, project instructions and object descriptions add to them. ## Object descriptions and result size ``` r writeLines(gptr_describe(iris)) #> 150 x 5, 7.1 KB #> $ Sepal.Length min 4.3, median 5.8, max 7.9; no NA #> $ Sepal.Width min 2, median 3, max 4.4; no NA #> $ Petal.Length min 1, median 4.35, max 6.9; no NA #> $ Petal.Width min 0.1, median 1.3, max 2.5; no NA #> $ Species 3 levels, top: setosa (50), versicolor (50), virginica (50); no NA writeLines(gptr_describe(iris, budget = 60)) #> 150 x 5, 7.1 KB #> Sepal.Length:dbl Sepal.Width:dbl Petal.Length:dbl Petal.Width:dbl Species:fct ``` `gptr_describe()` summarizes class, shape and values within a budget (150 tokens by default). Packages can register methods for their own classes. Generative-model context normally uses these descriptions and names; **System 1 questions send their input states**, so their egress and size rules differ. See the system-one vignette. ``` r work = new.env() work$mtcars = mtcars code = paste( "fits = lapply(split(mtcars, mtcars$cyl), function(d) stats::lm(mpg ~ wt, data = d))", "slopes = sapply(fits, function(f) stats::coef(f)[[2]])", "slopes", sep = "\n" ) fake = gptr_fake_provider(list( list(tool = "r", input = list(code = code)), "The fitted weight slopes are stored in slopes." )) s = peter("Fit mpg on weight by cylinder count and compare the slopes.", mtcars, model = fake, envir = work, mode = auto) work$slopes #> 4 6 8 #> -5.647025 -2.780106 -2.192438 ``` The calculation runs in R once the model requests it. Return a small table, statistic or plot rather than a full intermediate object. Long `r` output keeps a bounded head and tail; the full spilled text can be read through `peter$out(id)` when an output ID was supplied. ## Reported usage and estimated components ``` r gptr_usage(s, by = "model") #> group requests input output cache_read cache_write cost #> 1 fake/fake-1 2 3273 48 0 0 0 #> #> # 1 row #> 2 requests, 3.3k tokens in, $0.0000 ledger = gptr_usage(s, detail = TRUE) aggregate(tokens ~ component, data = ledger, FUN = sum) #> component tokens #> 1 attached 130 #> 2 environment 196 #> 3 images 0 #> 4 other 162 #> 5 project 0 #> 6 t0 2604 #> 7 t1 616 #> 8 tool_results 53 #> 9 tools 1942 #> 10 transcript 78 #> 11 workspace 76 ``` The usage summary contains provider-reported input, output, cache reads/writes and cost. `by = "session"`, `"agent"`, `"model"` or `"route"` groups it; child requests are included once. `gptr_usage()` without a session also includes the process-level System 1 log. The detailed ledger estimates components such as tools, workspace, attached objects, transcript and tool results. Its sum can differ from reported usage because tokenizers, provider accounting, images and fake-provider behavior differ. Unknown reported usage or cost stays `NA`, not zero. A local model's known zero metered charge does not imply zero hardware cost or latency. ## Budgets and partial results ``` r small = gptr_fake_provider(list("Done.")) res = tryCatch( peter("Summarize mpg.", model = small, envir = new.env(), budget = list(tokens = 500)), gptr_error_budget = function(e) e ) class(res)[1] #> [1] "gptr_error_budget_tokens" res$session$status #> [1] "budget" ``` The estimated input for the first request already exceeds 500 tokens, so this call stops before that request. A top-level run's defaults are 2 million tokens and 5 US dollars; `budget = list(tokens = ..., cost = ..., turns = ...)` supplies limits for that call. Change settings deliberately when different defaults are needed: ``` r gptr_config(budget = list(tokens = 20000, cost = 1, turns = 10), .scope = "session") s = peter("Review this analysis.", budget = list(tokens = 12000, turns = 6)) ``` Checks happen before requests using estimated next-input tokens and usage already recorded. The next output or final provider charge can exceed a remaining allowance; unknown usage cannot establish a hard monetary ceiling. A budget is a stopping policy, not a billing reservation. At the console the UI can ask to extend it; without a person present the call signals a budget condition carrying the partial session as `e$session`. Inspect that session's status, transcript, workspace and usage before deciding whether to continue. Teams and fan-outs share the root call's budget; launching more children does not give each one a fresh root allowance. With `gptr_parallel()`, member `budget =`, `tools =` and per-run `.opts` are not applied. Set the intended limits through the governing settings or enclosing run, cap concurrency and inspect the returned children's statuses. See `vignette("teams-and-background", package = "gptr")`. ## Three different caches | Mechanism | What is reused | What to inspect | |---|---|---| | Provider prompt cache | A supported repeated request prefix | `gptr_usage(s)` cache columns | | System 1 answer cache | A matching input/question/model decision | Result metadata and `gptr_cache()` | | Document replay and System 2 answer cache | Recorded code and the call's saved answer | `gptr_blocks(path)` and `gptr_cache()` | A stable prefix makes provider cache reuse possible. Providers decide whether a cache hit occurs, subject to their support, minimum size, expiry and pricing; gptr does not guarantee a hit after the first request. Compaction, image elision, changed resources or rewind can break the prefix. Document replay makes no model request for a usable recorded call but still runs its R code. Clearing an answer cache changes what can be reused. ## Explicit ways to reduce context ``` r peter("Summarize the model fit.", fit, .opts = list(context = "names")) peter("Compute mean(x) in R and return only that number.", .opts = list(context = "none")) peter("Compute the mean.", x, .opts = list(preset = "minimal")) ``` `context = "names"` lists workspace names and uses headers for explicitly attached objects. `"none"` omits both built-in workspace and attached-object descriptions; the R bindings still exist and can be used by tool code, as `x` is in the second example. Prompt text, tool results, images and other context blocks can still reach the model. System 1 input states use a separate encoding route. [Execution and memory](execution-model.html) explains these boundaries. `preset = "minimal"` reduces the tool/prompt surface; it does not authorize actions or remove required safety checks. `{identifier}` in a literal prompt can insert a short atomic value when that is sufficient. For a long workflow, continue one session, keep intermediate data in R and request concise reports between stages. Use a new session or an explicit fork when a separate analysis is intended. ## The source benchmark The source repository contains offline golden transcripts, component counts and ratchets for prompt growth and extra requests: ```sh Rscript --vanilla dev/bench/tokens/run.R --check ``` These fixture measurements support regression checks. They do not prove universal token savings or calibration against every provider. Live calibration is a separate, explicitly enabled release check; implementation or offline acceptance alone does not establish it.