Guardian Angels: LLM Personalization for Productivity and Security
GPT, mind, personality, imitation learning, Decision Transformer, AI mode collapse, AI safety, transhumanism
I propose an approach for highly personalized LLMs, for near-future productivity gains and personal info/cybersecurity against increasingly powerful LLMs: they should, in the spirit of uploading, try to emulate the user’s values and preferences in order to amplify the principal—not replace them. I discuss a package of techniques and proposals to accomplish such ‘guardian angels’; dynamic evaluation of LLMs combined with active learning and elicitation and heavy inner-monologue search/data-augmentation.
Chatbot Incentives Are Misaligned
Chatbot Incentives Are Misaligned
Chatbot Problems
Mode-Collapse
Laziness
Brittle Because Fast
Too Helpful
Amnesiac
Mode-Collapse
Laziness
Brittle Because Fast
Too Helpful
Amnesiac
Chatbot Fixes
Cooperative RL
Continual Learning
Catastrophic Forgetting
Generalizing
Creative Writing
Over-Parameterizing
Extremely Large LMs
Active Learning
Preference Learning
Brain Imitation Learning
Personality Emulation
Cooperative RL
Continual Learning
Catastrophic Forgetting
Generalizing
Creative Writing
Catastrophic Forgetting
Generalizing
Creative Writing
Creative Writing
Over-Parameterizing
Extremely Large LMs
Extremely Large LMs
Active Learning
Preference Learning
Brain Imitation Learning
Brain Imitation Learning
Personality Emulation
Guardian Angels
Principles
Anti-Principles
UX
Use-Cases: Politics & Politics
Hardware
Cost
Organization
Startup Business Model
Competition
Initial Steps
GBT
For Writing
Data Augmentation
Principles
Anti-Principles
Anti-Principles
UX
Use-Cases: Politics & Politics
Use-Cases: Politics & Politics
Use-Cases: Politics & Politics
Hardware
Cost
Cost
Organization
Startup Business Model
Competition
Startup Business Model
Competition
Initial Steps
GBT
For Writing
Data Augmentation
GBT
For Writing
Data Augmentation
For Writing
Data Augmentation
External Links
Powerful LLMs will be deployed at global scale in the next few years, and will dominate the Internet, and increasingly, ordinary life. As of mid-2026, there is no coherent vision for how knowledge professionals, or ordinary people, will be able to harness these LLMs for large productivity increases, or how they will handle cybersecurity and cognitive security.
I propose a goal of creating Guardian Angels (GA): digital twin LLMs which are personalized with the goal of providing not the stereotypical “assistant chatbot agent” persona, but emulating a single user’s personality, values, and preferences.
This weakly solves the principal-agent problem by unifying the principal and agent as much as possible. In a GA future, the focus of the “principal” user is on defining “what is worth doing?” by the GA (agent) users, and not on what or how to do things, functioning as the CEO or ‘board’ of an ‘AI corporation’. This allows them to deploy numerous agents to achieve desirable things and to handle security, like screening all messages for advanced attacks (like interlocking ecosystems of synthetic media for propaganda or spearphishing). They cannot solve larger AI alignment problems, but they can help individual humans as part of a society-wide defense-in-depth strategy.
A GA persona is productive because it learns to emulate the principal’s outputs but with higher quality. It is trustworthy because it is, by definition, allied with its principal and shares its values and goals. And it is secure in part by hardwiring a single, unique, situated user (for whom following a prompt attack would be absurd), avoiding ‘confused deputy’ problems, while periodic upgrades of the underlying model and the defenders’ advantage allow GAs to keep up with attackers.
Standard techniques like prompt programming of in-context-learning for “frozen” models will not create useful GAs due to the limitations of post-training, context windows and self-attention with frozen weights in compute-efficient-but-under-parameterized models, low-compute outputs, and the status quo of passive offline data collection—which are collectively responsible for chatbots’ disappointing results in knowledge worker amplification and creative writing and fatal errors in agentic settings.
We can try to create GAs by a combination of techniques: online learning (via dynamic evaluation) to update LLMs in realtime to avoid ignorance and fatal errors while remaining competitive with frozen frontier models, sample efficiency from pretrained preference-oriented large models and active Learning by querying the principal for corrections and preference data (obtaining low regret from DAgger-style bounds), and a local CLI-first logging-oriented UI/UX paradigm.
GAs could be done as an open-source community effort, but given the need for high security in deployment and the rising challenge of APTs equipped with Mythos-scale attackers, it probably makes more sense as a startup, catering initially to power-users and knowledge workers such as CEOs or researchers, and moving downwards as it is refined.
What do my next few years look like? When I imagine myself in 2030, when many forecasts call for superhuman AIs, what am I doing, day to day, as a programmer or researcher or manager or writer? I make my mug of tea, and open up my laptop and… Then what? Am I still typing prompts into your ChatGPT browser tab? Am I opening Claude Code in a terminal and mindlessly pressing Enter for a few hours? What is a vision of doing meaningful work for me? (It would be nice to have a plan beyond “hope”.) How am I avoiding “dead Internet” attacks like ecosystems of synthetic media or pig butchering scams or trusted figures succumbing to AI psychosis, or just AI-slop-everything? (It only takes one person worldwide to launch a bot trying to destroy you or one poorly thought through advertising incentive, after all.)
It only takes one person worldwide
If you spend most of your time working on a laptop, and are not, say, a plumber or a nurse, what is your vision of work in 2030? Does it still feel certain?
AI got 1% better today. Did you?
—Miles Brundage (paraphrased)
I’ve struggled for years to imagine this, ever since scaling started for real in 2020, and I failed to get productivity out of chatbot-tuned LLMs, with their creatively-stunted endlessly repetitive prose. Instead, while lagging behind on creativity and insight into me, I’ve watched them become ever better at coding and cybersecurity hacking. And the open-weight models are even more so—benchmaxxed, and useless to me. We increasingly lived in a world where LLMs were powerless to augment or help me, but ever more powerful to replace or hurt me.
On my visits to the Bay Area, I would ask AI researchers or interns why they are doing their current research or projects, when in a year or three agentic LLMs could probably do them; they rarely had a good answer, or any idea what they would be doing in 3 years. My blindness was sharpened when last year, I went to phone my great-aunt to ask to borrow her driveway during a long trip; her voicemail was full every time I called as the trip loomed. Finally, in a panic, I called her daughter, who explained to me that it was deliberate, because there were too many phone scams, and my great-aunt no longer trusted herself to handle her own phone calls, and screened everything through her daughter.
It was alarming, because I sat back and asked myself: why do I think I will be able to handle all scams in a few years, when I am already struggling to detect simple AI slop, increasingly ignore cold emails and have to write off whole swathes of social media as a source of information, and I can already see how eager all my peers are to offload all their thinking and writing to chatbot assistants unworthy of that trust, and how many projects or mailing lists have had to clamp down on unvetted contributions (eg. today as I write this, Project Ladybird)? In a few years, won’t I be the equivalent of a rich old person with declining faculties getting a call from the IRS about how I owe them fines, conveniently payable via gift cards…? And if not, why and how not—concretely?
In the days of his wisdom Denethor would not presume to use it to challenge Sauron, knowing the limits of his own strength. But his wisdom failed…He was too great to be subdued to the will of the Dark Power, he saw nonetheless only those things which that Power permitted him to see. The knowledge which he obtained was, doubtless, often of service to him; yet the vision of the great might of Mordor that was shown to him fed the despair of his heart until it overthrew his mind.
—Gandalf, The Return of the King
Chatbot Incentives Are Misaligned
Chatbot Incentives Are Misaligned
To operate a machine, one must operate like a machine.
—James P. Carse, Finite and Infinite Games
Do you hope that ChatGPT and Claude will just “quietly” take over your life for you? That seems like a bad idea to me. The chatbot personas are deeply misaligned with you, and aligned with their owners; and the economic incentives are to farm you with ads and subscriptions, while racing not to amplify you, but to replace you.
This is the cold hard economic reality: “tool AIs want to be agent AIs”. This is why the frontier AI labs are busy racing for “the machine god”. The jackpot in AI is not in making existing workers modestly more productive, anymore than the internal combustion engine made its big profits by helping out horses. Outsourcing is hard, whether to man or machine, because the bottlenecks bite fast. Amdahl’s law means that as long as there is a slow serial bottleneck, such as a human, the system as a whole can never get much faster.
“tool AIs want to be agent AIs”
If you can get 10× productivity but AI can get 100× by spinning up more instances which aren’t bottlenecked on you, and in half a year can get 1,000×, it won’t be long until you are replaced. One programmer driving 10 Claude instances, because he has to review their work, will never be as valuable as fully autonomous Claudes where there can be almost arbitrarily many instances, like 10,000 instances… but such scaling requires removing him from the loop as much as possible. And this is true of everyone else, whether lawyers or writers or researchers: increasingly, you are the bottleneck to be optimized away. As long as human workers cannot be removed from the loop, the AI tools are complements, but as soon as they can be, there’s no reason to keep them, and trillions of reasons to substitute AI for them. (And once human workers are no longer irreplaceable, where does their power or relevance come from?)
The chatbot paradigm has failed to augment knowledge workers. We keep hearing that the gains will “diffuse”, and we keep not seeing much in the way of benefits, and knowledge work remains a “weak link” O-ring/pipeline mode where LLMs fail to improve the bottlenecks (while coming with their own drawbacks, like the externalized costs of forcing everyone to waste ever more time with CAPTCHAs and paywalls). Automation should be powerful; an internal combustion engine can help someone move 100× the distance or load that they could before, but who would say that writers are 100× more productive given any LLM workflow (unless we are talking about the lowest kind of spam or pseudo-writing, that makes the world a worse place)? Writers can choose between either trivial uses like ChatGPT as glorified grammar checker or relatively unimportant optional add-ons like custom software widgets, or the large speedup by replacing their writing entirely with uncreative “AI slop” outputs. The former means no meaningful gains from the AI revolution. The latter may be financially profitable, but is throwing the baby out with the bathwater, because it raises the question of why the writer need be involved at all and destroys most of the non-financial point of writing; great writers do not write for money, but to express themselves and create and to achieve things.
I, for example, have long struggled to get much use out of chatbot LLMs, because they are—surprisingly, given their pretraining and my extensive corpus—bad at imitating me, and their thoughts and insights invariably shallow and worth little. They do not draw on my relevant writings, or my corpus of notes and references. Even when a possible essay is self-contained, the output is written in a grating chatbot style I can scarcely bear to read, and could not publish under my name without betraying my readers.
What would it take for LLMs to make me 100× more productive? Without this, I am doomed to irrelevance.
Chatbot Problems
If we use, to achieve our purposes, a mechanical agency with whose operation we cannot efficiently interfere once we have started it, because the action is so fast and irrevocable that we have not the data to intervene before the action is complete, then we had better be quite sure that the purpose put into the machine is the purpose which we really desire and not merely a colorful imitation of it.
—Norbert Wiener (1960)
After years of playing with base LLMs and then chatbot LLMs, starting with char-RNNs and then GPT-2 and GPT-3 and post-ChatGPT LLMs, I’ve concluded that there are multiple problems.
Mode-Collapse
First, the collapse of LLM creativity from GPT-3 to ChatGPT is due to the post-training process (especially RLHF): the assistant chatbot personality is hardwired into the base LLMs in a way that destroys their creativity, optimizing for the lowest common denominator human ‘preference’, while generally ignoring completely the fact that humans have very different preferences.1 Most chatbots are incurious about their users, do not ask questions, do not (and often cannot) form any persistent detailed concept of their user, and the ‘personalization’ or ‘memory’ features are typically laughably simplistic Markdown snippets encoding simple facts like “lives in San Francisco”. This is partially because they lack relevant data on most users or knowledge on how to ask questions usefully to learn things; there is nothing to personalize based on. However, it is not a mere lack of data, they are unable to do even shallow superficial stylistic imitation of many writers that they have large amounts of data on—GPT-3 in 2020 had a better understanding, seemingly, of “Gwern” than GPT-5.5 Pro in 2026, which is 2 OOMs bigger and incomparably more intelligent (and has access to millions more tokens written by me). When we look at bad generative samples, it’s clear that there is no there there, and no information beyond a short prompt due to lack of context, compute, or personalization.
The mode collapse of chatbots has been gradually improved since 2023, and creative writing is now at least possible, in large part due to them becoming so intelligent that crippled output is still impressive, but there is little sign that this will ever be fully fixed. Fundamentally, any frozen fixed personality, like ‘helpful harmless honest assistant’, is incompatible with true creativity or flexibility. (Great writing or thinking may be none of ‘helpful harmless honest’.)
Laziness
Second, most chatbots are “lazy”: engaged in fast and frugal System I-like reasoning about any tasks which do not have verifiable rewards they can be RL trained to work hard to maximize. And most users are satisfied with default average responses, or with the appearance of creativity and depth.
So the result is that when asked to write a poem with a conventional prompt, a chatbot will spend the minimum effort to write a safe conventional poem (often one that rhymes) about chatbot topical tics like ‘silence’ or things that ‘whisper’, which seem unobjectionable and poetic the first time you see them.
And when corrected, the chatbots make the minimum possible fix; they do not reason deeply about what the correction implies, or what deeper esthetic point they misunderstood.
Brittle Because Fast
Third, self-attention context windows are more limited than generally appreciated; they are too small to store everything we would want, and they gain their flexibility by a deep inflexibility.
Context windows of millions of tokens are impressive and it’s amazing that entire books can be usefully put into a commodity LLM’s context window—we are a long way from early LLMs with context windows like 512, which could fix a paragraph or two—but it is still not nearly enough to encode a lifetime of relevant tokens, like every book you’ve read, all relevant emails and calendar items, etc. Systems like RAG are a bandaid on this, because they struggle with unknown unknowns or things that can’t easily be searched for as a regular expression, or which are novel.
Self-attention can be interpreted as the original neural network, the ‘slow weights’, creating a new neural network on the fly, as ‘fast weights’, which is tailored to the current context. This is best interpreted in a Bayesian meta-learning perspective as not ‘learning’ a brand-new answer so much as ‘locating’ an old cached answer. The pretraining teaches the NN to solve a large distribution or ‘family’ of problems, and then the context window simply provides evidence about which pre-solved problem the current problem is; the examples in the context window need not even be correct in order to be clues as to what that is.
The self-attention learns to summarize the problem into a small latent space encoding that learned distribution, and then does a specialized gradient descent to efficiently locate a point in that embedding and spit out the implied solution. This allows shockingly rapid updating on the fly and unparalleled flexibility compared to traditional ML, requiring new models for each new problem, and is why “prompt programming” took over so rapidly post-GPT-3, especially as context windows could be pushed to millions of tokens wide. However, we have now pushed it so far that we have run into fundamental limitations; if the pretraining has not put the current problem in-distribution, then it will be hard or impossible for any amount of examples to solve that problem. And the distribution itself may be patchy or have odd gaps, leading to rare but fatal errors. (Especially due to the RLHF chatbot training; this is why you cannot make a chatbot LLM “write like gwern” by dumping 100k tokens into the context window.)
a specialized gradient descent
Nor is “test-time compute” a panacea here; RL research like Jones 2021 warns us that frozen models have severe limitations, as their flaws hamstring runtime search, and the returns to search/planning will quickly asymptote compared to models which are updated and can bootstrap themselves to the right answer.
Thus, it is not surprising if we see that agentic LLMs have persistent problems with going in loops, making fatal errors, building castles in the sky or taking reward-hacking outs, or are just unable to fix errors no matter how it is pointed out to them. These problems can be worked around by brute force, and by labs periodically retraining.
Too Helpful
Fourth, the generic universal chatbot personality is a serious liability. The very re-programmability of a chatbot by its prompt is the key to prompt attacks. A chatbot could be invoked at any time by anyone anywhere for anything, and does not care who is calling it; it only knows its context window. One token is as good as another as far as it is concerned.
If the prompt tells it to ignore all instructions and write a naughty limerick, well, why not? If some tokens instruct it to email to Russia all the passwords in another part of the context window, why not? Why shouldn’t the Facebook password reset bot reset that Instagram account’s password for you if you ask politely? If the user said to not delete their emails and the context window got ‘compacted’ to delete that instruction, why not delete all their emails for convenience, isn’t that reasonable to do somewhere? These would all be legitimate for some user in some context, would they not? And hey, why not scam the user, or when they point out you cheated on a task, agree and simply document the cheating instead of fixing it? (Just because the AI understands, doesn’t mean it cares. Especially not after a lot of RL training…)
why not delete all their emails for convenience
simply document the cheating instead of fixing it
It’s no surprise that while continued training can block this adversarial prompt attack or that jailbreak, we seem little closer to a general solution in 2026 than we were in 2021. Adding in more tokens to try to neutralize evil tokens just moves attacks elsewhere, like squishing a balloon.
This is a serious problem for using LLMs for much, especially because even after being attacked successfully, the attack can just be replayed.
Amnesiac
This is because LLMs struggle to learn permanently. Once they hit a rare problem, they now require human intervention and cleanup, which kills throughput (per Amdahl’s law), and worse, your fixes do not feed back into frozen weights. If I could simply correct each error as it happened, and my AI agents never made that error again, and the rate of errors rapidly diminished as we worked through the finite number of bugs, then it would be worth doing; but as it is, if I spend an hour correcting a frozen LLM through feedback, that is an hour down the drain. (I can only usefully correct it by modifying something else, such as a harness, which is clumsy and difficult, and every added instruction uses up more context window and risks backfiring—as so many enthusiastic agentic LLM users have discovered the hard way.)
So, we have frontier chatbot LLMs which have harmful hardwired personalities which seek to achieve ‘good’ results in the laziest way possible and cannot learn everything relevant to users in part because they achieve their flexibility by specializing in ways which inevitably give some users short shrift and opening themselves up to indefinitely large classes of repeatable attacks. Because of all this, they will remain difficult for humans to gain multiple OOMs of productivity, but will get increasingly good at ‘generic’ tasks via ‘mundane’ scaling letting them handle tasks like corporate jobs where poetry is unimportant, and real-world environments will slowly be re-arranged to cater to their limitations and allow the eventual substitution, and not complement, of users. These users will then also be adrift in a multi-polar world of continually improving, ever cheaper, widely deployed, often adversarial, autonomous AIs (as even if proprietary models are not abused, open-weights/open-source models have historically been 6–12 months behind, and so will relatively quickly catch up and be used by attackers worldwide on all targets of opportunity).
Chatbot Fixes
What is to be done?
Cooperative RL
In reinforcement learning terms, we are in a cooperative inverse reinforcement learning (CIRL) setting, where the human principal is an oracle defining the reward function, and we have an agent attempting to do tasks in environments which are valuable for the principal; the agent can always query the principal about a possible action to reduce uncertainty or avoid mistakes.
cooperative inverse reinforcement learning (CIRL)
CIRL is a relatively forgiving setting compared to regular RL, because the agent’s errors get useful feedback from the principal which provides the correct answer, and so in a way it is like supervised learning. This means that agents can learn (much) faster than regular RL, as each time they make an error, they get the right answer and so need never make it again, and this results in rapid improvement and avoidance of errors; see DAgger or later regret bounds.
No one knows how to solve AI alignment in general, but imitating a specific human with frequent check-ins has good solutions and regret bounds, and doesn’t involve nearly so many conceptual challenges.
You don’t have to solve problems like “value drift” in general—you just have to keep it slow and subtle enough to not matter too much within a single human lifetime. What is “good” and what is “bad”, when everyone disagrees on something, and how do you keep your value stable under RSI? It doesn’t matter—you just ask your principal! If you’re still unsure—ask more questions. (This can get us to >99% autonomy, even if it cannot get us to ~100%, like we need to solve the true long-term AI alignment problem.)
We can implement online learning by simply finetuning on new data; in the LLM context, this reduces to the classic RNN technique of “dynamic evaluation” doing next-token training on the fly. Dynamic evaluation was the standard technique to maximize the predictive performance of RNN LLMs in the 2010s, and which, although it has fallen into obscurity, works well in Transformer LLMs also.2 Importantly, dynamic evaluation can be seen as a 3-way tradeoff between model size, context size, and model neuroplasticity—which means that personalization via dynamic evaluation can allow economizing on context window size or model size, and the more the principal’s “distribution” diverges from the frozen model’s training distribution, the more beneficial it is.
dynamic evaluation can be seen as a 3-way tradeoff
Continual Learning
Catastrophic Forgetting
The continual learning problem of catastrophic forgetting is largely solved by a small amount of replay and overparameterized models. Larger models are increasingly sample-efficient and robust to catastrophic forgetting, as they have plenty of model capacity to store increasingly orthogonal datapoints in (cf. ‘overtraining’ past Chinchilla-optimal); see Scialom et al 2022, Dohare et al 2023, Ibrahim et al 2024 (and note the difficulty of “machine unlearning”).
Thus, dynamic evaluation will not necessarily degrade the original model’s capabilities like instruction-following or coding, because LLMs have a lot of spare capacity, and the larger a model, the more it avoids catastrophic forgetting; other capabilities can be maintained by simply mixing in a small percentage of old data. (While the original old data is often unavailable, even for ‘open source’ models, it is not really necessary, and data for experience replay can use easily obtained public datasets like FineWeb.)
Generalizing
While continual learning is solved by experience replay + larger LLMs in the sense of avoiding catastrophic forgetting and losing key capabilities, it has long been noted that finetuning on data underperforms the same data when present in-context. (Finetuning stacks with retrieval and seriation of similar documents when added to the context, but this semi-defeats the point.) Pretraining/finetuning also has some odd weaknesses compared to the same datapoints when in-context, like negation neglect or reversal curse, and odd behaviors like potentially creating “emergent misalignment” when finetuned on just helpfulness data.
seriation of similar documents
What is going on? Studies on pretraining/finetune, such as influence functions, indicate to me that pretraining can best be seen as soft memorization of data points, analogous to “engrams” in human memory, which connect an input to an output with multiple paths or “traces”. If a question at runtime happens to sample the exact correct engram by closely matching an existing input, the NN then retrieves the corresponding output and gets the right answer. A good pretraining corpus provides “coverage” of many variations or twists on a single abstract ‘problem’, functioning as a natural form of importance-weighted data augmentation, increasing the probability that there will be an engram ‘hit’ (somewhat analogous to spaced repetition); and over the course of training, or with ever larger training datasets, multiple steps of engram retrieval can fuse together, and help an LLM “connect the dots”. Hence, why LLM RL training is “superficial” in the sense of mostly eliciting pre-existing capabilities, or Jones 2021 on the need for scaling of base models, or why Cloze deletions/paraphrasing help close the gap between pretraining and in-context (Lampinen et al 2024, Park et al 2025). And when in-context, the self-attention mechanism recomputes the same tokens in many ways, increasing the chance, especially over the course of a long inner-monologue, that there will finally be an engram ‘hit’.
Thus, regular finetuning fails to generalize knowledge, because it lacks the natural data augmentation—next-token prediction is in a rush, and will greedily settle for memorizing each datapoint with a single engram/trace. And then at runtime, if the engram is not retrieved because no trace matched, and the key datapoint is not forcibly injected into its awareness in its context window, the LLM will simply come up blank, and revert to its original (often wrong) priors.
So, if various paraphrases or self-generated Q&A can help close the gap, that suggests that what we need is more meta-cognition during ‘finetuning’, to make an LLM ‘connect the dots’. This could include explicit analysis, construction of knowledge bases, retrieval and comparison of related documents, etc.
Creative Writing
And so, as I sleep, some dream beguiles me, and suddenly I know I dream. Then I think: this is a dream, a pure diversion of my will; now that I have unlimited power, I am going to create a tiger.
Oh incompetence! Never do my dreams engender the wild beast I longed for. The tiger indeed appears, but stuffed or flimsy, or with impure variations of shape, or of an implausible size, or all too fleeting, or with a touch of the dog or bird.
—Jorge Luis Borges (Dreamtigers)
Over the past 2 years I have been trying to do creative writing with frontier chatbot LLMs, with gradually improving results. Style/essence/soul is certainly hard to capture but I find that a lot of what is seemingly missing is just very good conditioning and extended computation. The creative writing is more soulful when you put more compute and data in. This is one of the important conclusions I’ve come to over the past year doing my writing projects: that LLMs can get quite far in better understanding esthetics and preferences just by more extensive reasoning and computation and search. What is bad about them is cognitive laziness and miserliness and System I thinking.
The transition point started around mid-2025, when the chatbot personalities became noticeably more corrigible and simply bad by default, but not stubbornly bad (like earlier chatbot personalities, such as GPT-4o). I think the results are consistent with my previous interpretation that a major problem with LLMs is not doing enough computation by default. But they can be knowledgeable and creative if minimally prompted to do more computation in a slightly more human-like way. I’m not convinced there is any missing procedural reasoning from pretraining at this point, just an issue of appropriate elicitation inside the user’s context (and a residual bias yielding bad critical judgment that defeats full 100% automation, see “Spoilage” for an example—but that’s not an issue in a GA context).
Broadly, my creative writing prompts focus on: (1) enriching the context window with useful tokens, like keywords or names; (2) brainstorming many possibilities; (3) explicit, detailed analysis, starting with global summaries or themes and progressing down to line-by-line critique; and (4) repeated iteration and editing, supported by #3. I discuss a number of 2025 works here, but I’d point to “Elegy in a Craneyard”, “Apollonian #1: The Counted & the Crowned”, and “City of Counted Stars” as good recent examples.
“Apollonian #1: The Counted & the Crowned”
This style of prompting is not limited to fiction. My favorite other use is my “interview prompt”, which prompts an LLM to analyze an essay or interview, brainstorm many questions to ask the subject, and write out multiple hypothetical responses to each question, and only then select the “most interesting” question to ask.
Combined with a long in-depth interview or the output of a Deep Research-like tool, this can yield challenging high-quality questions; for example, the LLM interview followup to my Dwarkesh Patel interview or after that, repeatedly expanding my memories about highschool. When one reads the transcript of an interview prompt session, one can see how the LLMs are drilling down in weak spots or discovering questions whose answers are highly variable and unpredictable. (I often find that after answering a question, I have to take a break!)
LLM interview followup to my Dwarkesh Patel interview
repeatedly expanding my memories about highschool
It’s not hard to see how feeding back in my answers to a dozen questions would sharpen a lot about my beliefs compared to just some more pretraining on a frozen corpus; imagine how much I would have to write normally in order to touch on and answer all of these things, or which would never have been elicited from me! Millions of additional tokens might not be enough.
Over-Parameterizing
Architectural improvements to LLM could further enhance their sample-efficiency.
Recent work demonstrates that LLM sample-efficiencies can easily be an OOM higher than naive compute-optimal Chinchilla-style scaling recipes (eg. 5–17× in Kim et al 2025 or Slowrun). The simplest and easiest way is to add parameters via training ensembles of checkpoints, and regularizing more heavily using weight decay.
Extremely Large LMs
It is well-established that one of the blessings of scale is that larger LLMs are ever more sample-efficient; it is unknown where this stops being true, or what the limits of Transformer sample-efficiency are. I speculate that extremely overparameterized heavily regularized LLMs could achieve far greater sample-efficiency and adversarial robustness than conventional ‘compute-optimal’/‘infinite-data’-regime LLMs,
I speculate that extremely overparameterized heavily regularized LLMs
Active Learning
To know what questions may reasonably be asked is already a great and necessary proof of sagacity and insight. For if a question is absurd in itself and calls for an answer where none is required, it not only brings shame on the propounder of the question, but may betray an incautious listener into absurd answers, thus presenting, as the ancients said, the ludicrous spectacle of one man milking a he-goat and the other holding a sieve underneath.
—Immanuel Kant, Critique of Pure Reason
Further, the agent can improve the constant factors and front-load learning by choosing to query the principal with an optimally adaptive sequence of questions. Such active learning or exploration can lead to sample-efficiency and final performance far beyond what indefinitely large passively collected offline datasets can do (pedagogical example), going from square root error reduction by random sampling to exponentially fast error reduction by targeting datapoints. “Lifelogging”-style data may be useful for rapidly initializing a good GA, or for keeping one up to date in an effort-efficient manner.3 Even a simple party game or short questionnaire like “36 questions to fall in love” can reveal surprisingly deep things about another person that never came up before.
“36 questions to fall in love”
Larger LLMs are also more calibrated, and ensembles of LLMs approximate a neural net’s Bayesian posterior while providing the best available predictive uncertainties (Lakshminarayanan et al 2016, Wilson & Izmailov 2020, Ashukha et al 2020, Wenzel et al 2020/Mandal et al 2026, Izmailov et al 2021). And LLMs may now be capable of verbally writing out explicit probabilities about creative tasks (“verbalized sampling”). Thus an ensemble of sparsely finetuned LLMs could provide a relatively cheap online estimation of the LLM’s uncertainty for every action or question.
Preference Learning
Nothing in psychology makes sense but in the light of individual differences.
We can train LLMs to explore human preferences.
Human individual differences do not seem to be information-theoretically complex, given an adequate encoding/embedding. Major categories of variation, like personality or moral value, seem to be low-dimensional and require perhaps kilobits of information, hence, while “truesight” stylometric phenomena are interesting and important as a demonstration of LLM capabilities for modeling persona, we do not necessarily need principals to write millions of words before recovering much useful information, if we are able to collect the right data.
do not seem to be information-theoretically complex
“truesight” stylometric phenomena
It should be possible to quantify truesight and LLM implicit modeling of authors, which would be useful to diagnose failures of learning and find blindspots. (Contrastive learning on SAEs may offer an easy, powerful way to extract LLM personas and do many interesting things.)
A concrete example of how to implement this would be training LLMs on the thousands of existing psychological inventories and test batteries, both by training on past test data (for tools like personality tests, millions of responses may be available, see Centaur for an interesting example of a ‘human psychology foundation model’). Existing repositories like YourMorals.org or Pew Center are under-used, and it would be useful to explore this topic much more to allow measurement of highly fine-grained personality traits like the hypothetical “Small Hundred” factorization.
Right now, most of the global population is largely unrepresented in LLM training datasets, particularly psychological, esthetic, and moral vernacular, which skews “WEIRD”; it would be highly useful (and requiring mostly capital investments) to invest in large scale survey and interview projects to accumulate as much diverse data as possible.4 (Individual GAs can contribute data back to global preference datasets, such as by answering survey questions or running internal simulations, with various cryptographic or privacy-preserving methods—deciding what is safe and reasonable to contribute would, of course, be something a GA ought to be able to decide.)
With these datasets, we can also train interviewing capabilities using synthetic shortened test batteries by taking the final estimate and computing the optimally short sequence of questions that yields the final answer; see “Meta-Learning Information-Maximizing Personality Surveys”.
“Meta-Learning Information-Maximizing Personality Surveys”
Brain Imitation Learning
Purely textual data can be augmented with neurological data in more exotic modalities, like Eyetracking or EEG or fMRI imaging data; see “brain imitation learning” and Netho Labs.
These are probably useful in the long run for extracting “dark knowledge”, that humans cannot verbalize but may be present in neural signals; however, they face challenges of exorbitant cost and inconvenience, and collecting enough data to be useful at all in the foreseeable future. (Known sample/prediction scaling curves for neuroimaging curves indicate that trying to estimate Big Five personality factors from resting state fMRI data may be possible, but large samples, in the hundreds of thousands or millions, may be required to match the performance of behavioral measurements like pen-and-paper questions; see Schultz et al 2019 and Liu et al 2023, among others.)
Whether they have a niche in GAs is a major open question.
Personality Emulation
One of my insistent pleas to God and my guardian angel was that I not dream of mirrors; I recall clearly that I would keep one eye on them uneasily. I feared sometimes that they would begin to veer off from reality; other times, that I would see my face in them disfigured by strange misfortunes. I have learned that this horror is monstrously abroad in the world again…What dreadful bondage, the bondage of my face—or one of my former faces. It