No. 033
Asking AI for creative ideas kept falling short across models. When I gave it three months of my logs and asked it to build something on its own, I got a tool I use every day. The difference was giving it evidence first and judging the result.
12 min readaideveloperreflection
Whenever Claude Code or Codex ships a new model, I send it the same request: “Please brainstorm creative ideas.”
There are two reasons: one is “wanting to build something open source, or something other people would find worth using,” and the other is “something useful for me to use.”
But every time, the results went in a loop.
So I kept getting dozens of ideas from these sessions, and none of them ever turned into something I built.
This time I asked differently. Instead of asking for ideas, I asked it to look at my logs (the last three months of Claude Code and Codex history) and build something on its own. The result was satisfying. The tool it built now sits in my menu bar, and I use it.
What surprised me was that the model this time was Opus 5.5, one of the models I'd been asking for ideas all along. I didn't organize context to feed it myself or put special care into the prompt, yet it gave me a satisfying result right away.
Below, I'll walk through how it went, step by step.
At first I thought the problem was how I was asking. So in early September I ran experiments, changing the instructions each time. I had the model generate ideas on the same topics several times and compared whether the results improved depending on the instructions.
| Experiment | What I compared | Result |
|---|---|---|
| Ideation techniques | Plain instruction vs. six ideation techniques, 108 ideas | 3/54 to 7/54 passed the bar, within chance |
| Divergence skill | Plain request vs. adding a skill with a divergence procedure | Plain request won 5 of 6 comparisons |
| Divergence methods | Fictional backgrounds, asking for non-obvious answers, borrowing distant concepts | No method clearly beat the plain request |
| Generation procedures | Independent repeats, splitting directions, adding unexplored directions | None of the three passed the promotion criteria |
Counting the experiments not in the table, I ran seven, and none of the modified instructions consistently beat the plain request.
Of course, the judging was also done by models from the same family, and the number of runs was small, so this doesn't prove that instructions are useless.
Still, what I learned was at least this much: refining my instructions wasn't producing any clear improvement.
Then I made this request.
~/.claude and ~/.codex hold every conversation I've had with Claude Code and Codex.
Over three months, that came to 528 sessions and about 3,800 requests I'd typed.
From these logs, the AI found things I had said again and again.
Then it built a tool to fix these annoyances in one go. I didn't decide what to build. The logs did.
What the AI built was a weekly dashboard called “Mirror.” With one script command or one skill call, it reads my conversation logs and repositories and shows the past week on a single page. It shows when I studied in each study area, where my time went, and which requests I kept retyping over several days.
Since there's no status document to update by hand, nothing goes stale. It recalculates everything from the logs that are already being written. (I've hidden a few items I'd rather not make public.)
My first thought when I saw it was, “Why didn't I build this sooner?”
Once I had Mirror, one thing still bothered me. A dashboard that only shows you things tends to end once you've looked at it. You learn what the problems are, but handling them is still on you. So the next day I asked this.
Again, I didn't specify a format. What came back was a macOS menu bar app.
The menu bar at the top of the screen always shows how many actions are left, like 거울 1 (Mirror 1). Clicking it shows the last study date for each study area at the top, with this week's actions below. Each action comes with things you can do in a single click.
The app reanalyzes the logs every three hours and sends a notification when a new study gap or deployment drift shows up. It's registered to start at login, and since it's always running, I measured its resource use too. Idle, it used 36 MB of memory and almost no CPU.
This is what surprised me most. All I said was “something that lets me act on it.” A menu bar app never crossed my mind. The trip to open the dashboard is gone, and every step of handling an action is now a single click. If I'd seen “menu bar app” as one line in a list of ideas, I probably would have skipped right past it. Only after actually using it did I realize it was the right call.
The action loop runs behind the menu bar. Each signal Mirror finds becomes an action with a ready-to-paste request and a working folder. Instead of using the menu bar, I can tell an agent “do number 3,” and it handles the action in that conversation and marks it done.
Another part I liked is that marking something done and confirming it resolved are separate. Marking an action done doesn't close it. Only when the same signal is gone in the next analysis does it become “resolved.” If the signal is still there, it comes back as “needs review.”
In practice, the action asking me to decide whether to promote four failure-log entries into rules was confirmed resolved in the analysis the day after I handled it. The action to cut down a repeated request, on the other hand, is marked done but still shows needs review. Its signal only goes away after the same request stops appearing for four weeks. Doing the work and making the problem go away are different things, and the tool keeps track of that difference for me.
The first version wasn't good right away. The AI ran what it built against my real logs, found the places where the results looked wrong, and fixed them itself.
If I'd received the first design as it was, I'd probably have been half disappointed. I think a good part of why I ended up satisfied came from these fixes. And fixes like these can't come out of the idea stage. They're problems you only see after running it on real data.
After I had everything, I got curious. It was the same model, so why was I satisfied this time? So I asked the AI directly. The conversation below is shortened and edited from the actual exchange to make it easier to read.
After reading the answer, I noticed one more thing. When I asked for ideas and when I asked it to just build something, the point at which I made judgments was different.
When I asked for ideas, I was judging at the very front. I took the list, imagined how each one would turn out, and picked one myself. With only imagination to judge with, the outcomes were always vague.
When I asked it to just build something, my judgment moved to the very end. I judged by actually using what was built. Judging with my own data on the screen is far easier and more certain.
But if you only push judgment back, the front seat goes empty. That's the seat for deciding “what to build.” Hand that seat to AI with nothing in it, and you'll get the same average answer you get when you ask for ideas. This time, my logs filled it. What I kept repeating, what bothered me, and what I wanted were already in them.
So the conclusion isn't “just tell it to build something.” Give it evidence first, then judge the result. Both had to be there.
Until now, my mindset was “people should evaluate AI's ideas and plans.” This experience makes me wonder, though it may be a hasty conclusion, whether the place for evaluation has moved to “people should use and review what AI builds.”
In my last post, If AI Can Do It All, What Can I Add?, I wrote that I wanted to be a developer who sets the bar for the result I want and builds toward it together with AI. At the time, it was still closer to a resolution.
This time I got a glimpse of what that actually looks like. I didn't come up with ideas, and I didn't write a single line of code. Instead, I pointed to where the evidence was, and I judged by using what was built. And I asked a question that pushed the AI's answer one step further. The thought that judgment had moved back was one I raised first.