Timeline

Milestones are not on this page because they have no dates. See Progress.

Stages

2026-10-24

• 01:31 evaluate, Claude Code (Opus 5.5): Vignette 5 reaches its one-time test. Tasks 1 to 12b are reviewed; the four gpt-6-luna loops are blind (D197); the 15 setups are fixed by an md5 manifest and calibrate with Platt targets (D200). A dry run checked the setups and the 1,028 test rows per model with no call. evaluate_once() now scores the test rows of Jev, gpt-6-luna and Clef-Flash once.
• 03:03 build, Claude Code (Opus 5.5): The package decisions is built and reviewed (Tasks 1 to 13 of the plan): 14 exported functions, six vignettes, four datasets and the agent skill. A final whole-branch review found no critical issue; its fixes passed a scoped re-review. R CMD check --as-cran gives 0 errors and 0 warnings. A clean-session rerun of vignette 5 from its state files sent 0 requests and gave byte-identical results. Claude Code hands the package folder to fmm for the test with the code of vignette 5 (D158).
• 03:55 D209 03:55: fmm stays the only maintainer (cre) of the package decisions; Ying-Ju (Tessa) Chen is an author (aut) with her email (ychen4@udayton.edu) and ORCID (0000-0002-6444-6859).
• 03:55 D210 03:55: The README of this repository names the package decisions (M3), what it does, its vignette that repeats the study, and where its design and plan are; the old name decisionqc stays only for the first draft in docs/.
• 04:43 D211 04:43: The package decisions is public at github.com/fmegahed/decisions; its DESCRIPTION (URL, BugReports), its pkgdown url and the README of this repository point to it, and GitHub Pages serves its website from the branch gh-pages (https://fmegahed.github.io/decisions/).
• 04:58 D212 04:58: The package decisions lists Fadel M. Megahed with his ORCID (0000-0003-2194-5110) and the maintainer email fmegahed@miamioh.edu, as it lists Tessa; the website and the help pages take both from DESCRIPTION.
• 05:05 D213 05:05: Vignette 5 gets a section "Reruns": a second run of its code (fmm, 2026-10-24) gave the same answers for gpt-6-luna at every step; for Jev (jev-1.14.0) the wording and grid AUCs differed by at most 0.03, two close choices changed, and the test AUC on extrusion was 0.739 in place of 0.755; Clef-Flash changed less (wording AUCs within 0.005, one close choice, the same k and image size, test AUCs within 0.007).
• 05:32 D214 05:32: The help of each decider says where to get its key and where to put it, as ellmer does: a sign-up sentence with a checked link, and one shared text for credentials (set the environment variable, best in .Renviron with usethis::edit_r_environ()). Other documentation practices of ellmer are surveyed; the clear, documentation-only ones are adopted and the rest go to fmm as proposals.
• 05:38 D215 05:38: Vignette 5 is renamed "Case study: industrial image inspection" (file industrial-image-inspection.Rmd); its internal script, folder and data names (study_again) stay.
• 05:38 D216 05:38: Before fmm submits the package to CRAN, Claude Code sets the version to 0.1.0, writes cran-comments.md, runs a CRAN policy pass (metadata, help, spelling, URLs) and the final checks, and a fresh reviewer checks the result. fmm runs the checks that send the package to outside services and submits.
• 05:48 D217 05:48: The package README gets an R-CMD-check badge, a lifecycle badge "experimental" and a "Learning more" section; the examples that need a key stay in \dontrun{} (no live examples); the help of decider_typesafe() links https://console.typesafe.ai/ for keys, and decider_openai() keeps https://platform.openai.com.
• 05:48 D218 05:48: The package gets a hex sticker. Claude Code drew ten concepts on a private Design canvas for fmm to choose from: probability bars, a fork with branch probabilities, typed questions, a calibration curve, a dial, tidy output, a signpost, a balance, a probability ring and image inspection, in Miami red with Outfit type.
• 11:09 D219 11:09: The package logo is concept 20 of the hex canvas. On Miami red, a message gives two named answers with bars (refund 0.71, other 0.29). A Corn Yellow line divides the hex. Below it, a fabric image with a circled defect gives defect 0.93 and ok 0.07. The word decisions is at the bottom. The words are outlines, so the SVG does not need the Outfit font. man/figures/logo.svg feeds the pkgdown site and a 240-pixel man/figures/logo.png feeds the README.
• 12:34 D221 12:34: Claude Code runs the outside checks before the CRAN submission: win-builder (R-devel and R-release) with devtools, and R-hub on Linux, Windows, macOS, macOS arm64 and the nosuggests container, from a new rhub.yaml workflow in the package repository. Claude Code shows fmm the lines for cran-comments.md before it writes them. fmm still submits to CRAN.
• 12:50 D222 12:50: The package links the TypeSafe documentation (https://docs.typesafe.ai/introduction) in DESCRIPTION and in the help of decider_typesafe(), and the README Keys table links the TypeSafe console (https://console.typesafe.ai/). The home page https://typesafe.ai is no longer linked.
• 13:06 D223 13:06: The package DESCRIPTION reads VignetteBuilder: knitr, rmarkdown. R-hub then reruns on Linux, Windows, macOS, macOS arm64 and nosuggests, and win-builder reruns on R-devel and R-release, all on the final commit 141eb1d.
• 15:14 build, Claude Code (Opus 5.5): End of milestone M3: the package decisions 0.1.0 passed the final checks on 141eb1d (local --as-cran, GitHub Actions, R-hub and win-builder: only the new-submission note), and fmm submits it to CRAN. The build ledger, with the CRAN preparation and the commit map of D206, is copied to the record (D204).
• 15:22 D224 15:22: The package website shows the part of the record about the package at /record/, with the menu item Record. The script data-raw/record.R of the package repository copies D39 and the decisions from D136 on, the M3 process events, the M3 design, plan and ledger, and the package budgets and call logs from this record; this record stays the source. The Home page of the copy names the larger project (Beyond CLIP, with its repository) and cites the replication-materials skill. Its Contributions page names Ying-Ju (Tessa) Chen as an author: the package came from discussions of fmm and yjc and builds on their joint work (the CLIP paper, the working paper, conformal-clip and conformal-clip-data). The copy is outside the package tarball.
• 1 decision or ruling by Claude Code

2026-10-23

• 00:03 D199 00:03: In vignette 5, a setup whose calibration rows separate the classes perfectly keeps no calibrator and uses the raw probabilities; the vignette says why. This applies to pans (Jev, gpt-6-luna, Clef-Flash) and pipe staple (Jev, Clef-Flash). The other ten setups keep the logistic calibrator.
• 00:43 D200 00:43: Vignette 5 calibrates all 15 setups with Platt scaling as the paper did: a logistic fit on logit(p) with the smoothed targets of Platt, written in the vignette code with glm(). The setups keep no calibrator, so evaluate_once() reports the ranking AUC of the raw probabilities, and the vignette adds the calibrated metrics.
• 00:52 brainstorm, Fadel M. Megahed: M3 (R package decisionqc) starts with a brainstorm, while the paper tasks of Claude Code are on hold (D135). fmm gave the first scope: two routes (ellmer and httr2, or a C or Rust backend), JEV-like models for text and images, open models through Ollama, Hugging Face and llama.cpp, calibration, reports of performance, cost and speed, and a GUI. Outside the package: a vignette, a Shiny app on a Hugging Face Space that runs on a CPU, and an agent skill that optimizes the descriptions.
• 01:08 D136 01:08: The package decisionqc calls System One decision models only: models that return a probability for each allowed answer, such as Jev (TypeSafe), Microsoft-Decision-1 and the OpenAI Decisions API, and open models of this kind on Hugging Face, such as Laya. It does not call chat models.
• 01:14 D137 01:14: decisionqc asks a decision model about one input or about many inputs in one batch, with outputs restricted to the typed answers of the questions. It also calibrates the probabilities and connects to ML-style experiments.
• 01:17 D138 01:17: ML-style experiments with decisionqc run inside tidymodels (data splits, resampling and tuning), with the decision model as one more model there. fmm calls this a plus, not a must.
• 01:24 D139 01:24: decisionqc is a self-contained package, a sister of ellmer for decision models, built with the R ecosystem in mind. It calls the APIs with httr2 and does not need a change in ellmer. The first CRAN release holds the model side, the batch decisions with typed answers, calibration and a report of performance, cost and speed.
• 01:24 D140 01:24: decisionqc publishes 3 to 5 vignettes, each for a different use case. One shows tidymodels on top of the package, and one is about images.
• 01:35 D141 01:35: The first vignette (text) shows the three question types: noul (yes or no), choice and score.
• 01:35 D142 01:35: Image preprocessing is set by arguments of the decision functions, not by user code: at least the size, keeping the resolution and the crop. A hook can take an object detection model that finds the part before the decision, if it makes sense.
• 01:35 D143 01:35: Closed-weight models take their credentials the way ellmer does.
• 01:35 D144 01:35: The tidymodels vignette replicates the comparison of Clef-Flash and JEV in the paper and shows how to tune the descriptions.
• 01:35 D145 01:35: The lightest example runs in the web browser with webR, if possible.
• 01:45 D146 01:45: Name the R package decisions, not decisionqc, because it serves any decision about text or images, not only quality control.
• 01:52 D147 01:52: The GUI of decisions has two modes that share a saved setup file. In the build mode, an R user picks text or images, writes the questions, chooses a model, tries examples, calibrates and sees the report. In the use mode, made from a saved setup, colleagues who do not code give a text or an image and see the answer and its probability.
• 01:52 D148 01:52: The package decisions ships an agent skill that connects a coding agent to the package to tune the descriptions, as the commons package of Posit does: the skill is in the package, and a vignette shows how to copy it to .claude/skills or .agents/skills.
• 01:52 D149 01:52: Add a sixth vignette on short written answers checked against a rubric, from the exam-review game of the isa401 repository (games/exam-review), where each idea of the rubric is a noul question. Its questions can be about the package itself.
• 01:55 D150 01:55: Vignette 6 stays a plain vignette that shows rubric checks of short written answers as a grading option. It credits the source of the idea and the ISA 401 course of fmm at Miami University.
• 02:13 D151 02:13: livelink is not a dependency of decisions, not even in Suggests. The authors use it only to make links that let people run our code, if webR itself is not possible.
• 02:19 D152 02:19: Keep the name decisions for the R package, not decider.
• 02:19 D153 02:19: The live tests of decisions use TypeSafe (Jev), the OpenAI Decisions API, Microsoft-Decision-1 through Foundry or OpenRouter, and local servers (Clef-Flash on the GPU, Laya).
• 02:19 D154 02:19: The first CRAN release of decisions has no fixed date. It goes to CRAN when release 0.1.0 is complete.
• 02:19 D155 02:19: The package guards the test data (score_setup() refuses test rows; a one-time test lock), and the agent skill gives the rules. There is no full sandbox.
• 02:31 D156 02:31: Vignette 5 does the analysis of the study again with the pipeline of the package, with new paid calls for validation, and starts with all available models. It has no CLIP comparison. Its results can differ from the paper.
• 02:31 D157 02:31: Make the Hugging Face Space with Laya. The Space is not part of the package; the documentation of the package links to it.
• 02:31 D158 02:31: The repository of the package decisions gets no replication record, and Claude Code does not push to it. Claude Code tests all of the package and its workflow first; then fmm tests it with the code of vignette 5.
• 02:31 D159 02:31: The package does not manage API spending: no spending cap and no budget, as in ellmer. A user who makes an API key can set its limits with the provider.
• 02:31 D160 02:31: Keep the package small: a few exported functions. Its use comes from the demos (vignettes), which can use other packages.
• 03:03 D161 03:03: Design decisions for model facts that change. The package checks no size limit of a provider; the error of the provider reaches the user word for word in .error. The only model facts in the package are the input types (text, image or both) in one dated table, and each decider takes an input argument that overrides the table.
• 03:03 D162 03:03: The cost column holds only the cost that the provider reports. When a provider does not report a cost, the package says so; it does not estimate the cost from a price table.
• 03:15 D163 03:15: The package, its documents and its vignettes do not call Jev, or any other model, the first decision model.
• 03:20 D164 03:20: The package does not fit calibrators. The probably package (tidymodels) calibrates, in Suggests; calibrate() is not exported. The probability columns of decide() use the names of tidymodels (.pred_<answer>). A setup keeps a probably calibrator and drops it when the questions change.
• 03:22 D165 03:22: Take the GUI out of the package. The Hugging Face Space is the build mode (D157), and vignette 6 has a short Shiny app for the use mode. The setup files stay in the package.
• 03:24 D201 03:24: The README of the package says that decisions is inspired by ellmer, in place of a sister of ellmer.
• 03:24 D202 03:24: The package copies these practices of ellmer: live smoke tests that run only with DECISIONS_LIVE_TESTS=true and a key, and skip otherwise; a clear message for a missing key; GitHub Actions workflows for R CMD check, pkgdown and the live tests (the live workflow runs only by hand, because each run makes paid calls); notes for contributors.
• 03:24 D203 03:24: Claude Code investigates whether vcr cassettes, as in ellmer, can replace the saved results of the text vignettes, with a free experiment on Laya and no paid call, and reports before any change.
• 03:24 D204 03:24: Milestone M3 closes after fmm tests the package with the code of vignette 5; then Claude Code copies the build ledger into the record and sets M3 to done.
• 03:33 D166 03:33: Models, questions and setups are S7 classes, as in ellmer.
• 03:33 D167 03:33: The function names follow ellmer: decider_*() for models, question_*() for questions, and decide().
• 03:33 D168 03:33: decide() takes one model per call. The vignettes compare models with purrr::map() and bind_rows().
• 03:33 D169 03:33: decide() takes path and ignore_hash, as ellmer::batch_chat(): a JSON file keeps the finished answers, and a new run sends only the inputs that are missing or failed.
• 03:33 D170 03:33: decide() retries failed calls with the defaults of httr2 (HTTP 429 and 503, with backoff).
• 03:33 D171 03:33: Each request puts the parts that all inputs share (instructions, then references) first and the input last, so that the prompt caching of a provider can lower the cost.
• 03:33 D172 03:33: A setup keeps its probably calibrator inside the JSON file, with a hash of everything that changes the uncalibrated probabilities: the model (provider, base URL, model, request format), the questions, the references and the image settings; read_setup() drops the calibrator with a warning when any of them changed.
• 03:33 D173 03:33: A setup file is one JSON file, with the prepared reference images inside (base64).
• 03:33 D174 03:33: score_setup() takes one setup per call. Vignette 5 maps over a tuning grid with purrr::map().
• 03:33 D175 03:33: The Space and the app of vignette 6 name the provider that receives each input, accept a credentials function for each session, and have a stop button for long batches.
• 03:33 D176 03:33: magick is in Suggests. When an image arrives and magick is not installed, the package guides the user through the installation.
• 03:33 D177 03:33: On CRAN, each vignette reads small saved results, and the full live code is on the package website.
• 03:33 D178 03:33: The sample answers of vignette 6 are written for the vignette. No student work is used.
• 03:33 D179 03:33: pplx-decider of Perplexity stays out of release 0.1.0 until its request format is known.
• 03:33 D180 03:33: The first CRAN release, 0.1.0, has all 14 functions, the agent skill, the six vignettes and webR example A.
• 03:33 D205 03:33: The help of evaluate_once() keeps its advice for a test row that fails at every call: if a new call cannot fix the row, remove it from the test rows, call evaluate_once() again with the same path, and report the removed row and its error with the result.
• 03:33 D206 03:33: The raw downloads do not ship in the package repository: data-raw/SIRDataDownload.csv (OSHA) and data-raw/openfda_food_enforcement_oh.csv (openFDA) leave every commit of the local history before the first push, and are git-ignored. The data-raw scripts and the help pages give the APA citation and how to download each file.
• 03:33 D207 03:33: pplx-decider of Perplexity is not planned for now; its request format is unclear.
• 03:34 D181 03:34: The code of decisions must be correct, efficient and easy to follow. A simple code base comes before features: this is why fmm chose one model per call (D168) and no GUI in the package (D165).
• 03:34 D182 03:34: Every function of decisions has roxygen2 documentation.
• 03:35 D208 03:35: All six vignettes keep saved results (D177) in release 0.1.0; the package does not use vcr cassettes. If live-evaluated code is wanted later, vignette 6 comes first, then vignette 1.
• 03:36 plan, Claude Code (Opus 5.5): The brainstorm of M3 ends: fmm approved the design of the R package decisions (D183), after D136 to D182. A fresh reviewer read the design as a builder; fmm answered its open questions and 15 open items with multiple-choice questions. Claude Code now writes the implementation plan.
• 03:36 D183 03:36: Approve the design of the R package decisions (revision 3, plans/2026-10-23-decisions-package-design.md). The next step is the implementation plan.
• 03:41 D184 03:41: The code of the package must be cleaner and more efficient than the study code. The study code tells what the package must do, not how to write it.
• 03:44 D185 03:44: Add Ying-Ju (Tessa) Chen (yjc) as an author of the package decisions.
• 03:44 D186 03:44: Vignette 5 stands on its own. Readers may not have seen the paper, which is still in progress, so the vignette explains the question, the data, the sets and the rules itself and does not compare its numbers with the paper.
• 03:50 build, Claude Code (Opus 5.5): fmm approved the plan of the package decisions and chose subagent-driven execution (D187). Claude Code builds the package task by task, with a fresh implementer and a review for each task, in the local folder decisions (back-filled).
• 03:50 D187 03:50: Approve the implementation plan of the package decisions (plans/2026-10-23-decisions-package-plan.md) and run it subagent-driven: a fresh implementer for each task and a review after each task. M3 starts the build.
• 03:59 D188 03:59: Approve three budget rows for Task 2 of the package plan (typesafe, openai, openrouter): at most 400 calls and USD 9.50 for each service.
• 13:23 D189 13:23: The probability columns of decide() get syntactic names in a janitor-like way ("not sure" gives .pred_not_sure), and .pred_class keeps the exact answers as its factor levels. Two answers that give the same column name are an error.
• 13:32 D190 13:32: The cleaning of the probability column names transliterates as janitor does, with stringi::stri_trans_general(x, "Any-Latin;Latin-ASCII") (café gives cafe, да gives da), and a name that starts with a digit gets the prefix X; stringi goes to Imports.
• 21:06 D191 21:06: Approve five budget rows for Tasks 11 and 12 of the package plan: for Task 11, typesafe and openai at most 300 calls and USD 1 each, openrouter at most 150 calls and USD 0.50; for Task 12, typesafe at most 8,000 calls and USD 5, openai at most 8,000 calls and USD 10.
• 21:44 D192 21:44: The package decisions ships three small datasets: ohio_injuries (OSHA Severe Injury Reports, Ohio, 2025, five injury groups) for vignette 3, textiles (50 simulated textile images) for vignette 2, and rubric_answers (the labeled sample answers) for vignette 6. ohio_injuries replaces the modeldata reviews of vignette 3.
• 21:59 D194 21:59: In vignette 5, gpt-6-luna starts from the generic description and scores the candidates of the Jev loops in their order with the 0.02 keep rule, as Jev does.
• 22:02 D195 22:02: The package cites its data sources in APA 7: the OSHA Severe Injury Reports (https://www.osha.gov/severe-injury-reports) for ohio_injuries, and spc4sts (Bui and Apley, 2021, doi:10.1080/00224065.2019.1707730, and its reference manual) for textiles; the help pages, the vignettes and the DESCRIPTION file carry them.
• 22:11 D196 22:11: The package adds a fourth dataset, ohio_recalls, for score questions: the openFDA food recall reports of Ohio firms, all years, with the reason for the recall as the text and the FDA class (Class III, II, I) as an ordered label. Vignette 1 gets a short score section with known answers on about 30 recalls with Jev.
• 22:21 D197 22:21: In vignette 5, gpt-6-luna gets its own description loop, run as the loops of the study: one fresh agent per dataset (extrusion, microstructure, pipe staple, textiles) in a blind sandbox, with the loop instructions of the study, two references per class, at most 6 candidates (candidate 1 is the generic description), and scoring through score_setup() of the package. Pans keep the generic description. The loop log ships with the package, and the wording step of the vignette rescores the logged candidates of each model.
• 22:33 D198 22:33: Vignette 5 has a short note: its setup differs slightly from the paper, because it enlarges the images to 512 or 768 pixels, where the paper used 1,568 pixels with padding. The vignette still does not compare its numbers with the paper (D186).
• 66 decisions and rulings by Claude Code

2026-10-20

• 17:59 D39 17:59: Name the R package decisionqc and the research repository decision-models-for-industrial-image-qc.