Stop trying to make one AI model do everything

Stop trying to make one AI model do everything only to get maybe 80% consistency on a task. You do not need a genius model for your breakthrough work. You need lots of small ones that are 100 percent consistent. Here is what I mean.

Automation, when do I do it?

Models SUCK at work when the prompting is loose. The better the language, the more specific and expert the analysis, the better the adherence.

Anthropic showed a tutorial of Claude classifying blame on traffic accident reports. During the demo they kept adding more and more to the prompt to make Claude better at assigning blame. The better the prompt got, the better the model got at the task. By the end it seemed like Claude was pretty darn good at assigning blame from a simple visual of the scene. They even made it stick to a specific structured output. But that was one example. Now what if you need to do this at scale with lots of edge cases? What are you going to do? Edit the prompt every time? hahahah. wait. no! no don't do that...

DONT LET HIM PROMPT THE MODEL TO EVERY EDGE CASE MURPH!

If you have done any work at all making /skill.md files or modifying your AGENTS.md or CLAUDE.md, you have seen this same behavioral improvement. You can even tack on more layers to get better consistency with a specified output.

At the company where I work we have to classify a ton of patient medical tests and documents with AI models, because doing it by hand can be... cumbersome. Example: a patient gets a Flow Cytometry test result. We take that result, classify which markers were present, and need a specific output in code for what the test found, percentages of visibility in the patient, and a few more classifications.

Normally a doctor on our team is clicking through a menu, entering numerical values and yeses and nos. We take that shape of response and put it in our database. It is glorified copy paste, a waste of this doctor's time, and a perfect candidate for agentic automation. So we "automated" it. We followed the lead engineers at Claude and prompted our own classification model to handle the millions of documents we consume. It sucked. No matter how much we improved the prompt, specified the output structure, or yelled at it, it would fail to adhere, and the doctors still had to classify everything by hand.

Permanent edge cases, and no time to handle them all, meant someone had to edit the prompt every. single. day. to make the AI just work. I have more of these stories and more frustrations to rant about, but the main takeaway is: You need to be able to train your model rapidly only through usage of the application. This is not true for everything but it is true here.

So how do you do this? You could take the data and have a second model review the issue and change the language of the prompt. I tried this. It worked out ok. The model training the new agent from the feedback would change important parts of the classification method and make the model worse in an area it was already performing well. A new test arrives, or a new format, or a broken file, the model cannot get it right, a new model changes its prompt, it degrades in some new area. You get the idea. This is not addressing the root problem of the model.

It's time to leave your /skill.md and .sh script son

Take a small agent and make two data sets. One set is the agent on the task and its output. One set is the expert's correction.

Now we take the context of the document, the corrected method, and the AI's attempt, and save that as a training segment. Our team does this about 100 times a day naturally, because they are correcting the AI anyway. They have to. Then we send that off for training on a model from a frontier lab, or an open source model we can host locally. We do not modify the /skill, the prompt, or anything else quite yet.

We need to literally reprogram the brain of this model. Handing a child an instruction manual will not make a kid good at a task. Repetition of the material, the take-home exams, and homework is what makes the kid good at a task. The more they practiced, the better they got. So why are we treating AI differently? If we want it to be good at a task, we should be making new "synapses" for it. Not giving it another book to ignore.

We need to modify the weights. Eventually we have a highly skilled, trained document-analysis AI that can improve on its own. No manual prompt retraining required. Now when we give it an instruction list or a toolbox like /skill.md files or scripts and an output shape in a system prompt, we know it will be very consistent.

That lets us take work that required a frontier model with heavy prompting and mixed accuracy, and pass it to a tiny model with a specialized skill, on a very unique set of examples, with pinpoint precision.

30 small models of AI on the wall, 30 small models of AI! Take one down, pass it around, now we have better trained models on the wall.

Now take that document analyzer and span it out. Maybe it's good at the other tests too now, and if it is, great, have it work those. If it starts to break down, repeat the cycle and put a model on the one part of the flow that's failing. I know this isn't new. This is just RLHF. Doing it on purpose, on one job, from the corrections your team is already typing anyway, is the part people keep skipping.

I personally love the model training community. I can go find an open-weight text-to-speech model that fits on a tiny hosted server and it works amazing. Yes it is a bit slow, but it is really freaking good! Now I can build my own audiobooks from PDFs and articles for practically nothing. Screw you Audible! People have already been making full content farms like this by chaining a vision model, a voice model, a writing model, and whatever else they need. One job, stay consistent, and when the new job wrecks the old one you don't write a longer prompt. You make another model.

I don't have a fancy rule for when to split because I haven't trained this thing yet. I'm not specifically trained in this area. I'm looking into it. From where I'm sitting, you split when teaching it the new thing makes it worse at the old thing. That's as far as I've gotten.

What I think this could look like

While I know I am not an subject matter expert I am excited to do more work on this and continually post about it. All I know is the current method of using general intelligence to perform specified actions is not the right way, and the skill dumps are not the right way either. That is the biggest waste of human effort I can see right now. Everybody keeps writing another instruction file for a model that still doesn't know the job.

Hugging Face is an open library where you can store the weights, the datasets, spaces, and a model card with a bit of discussion. I have used it a bit for personal projects. However the thing I actually want is more like git for models. I go to the library, I find a model that almost does my job, I make a branch or a version of it, I train that version on my specific thing, and I put it back. Somebody else can grab mine, make their own version, and keep going. If their version is better, great. If teaching it a second job makes the first one worse, they split it, and now there are two models instead of one bloated one.

I am not claiming I invented that. Hugging Face is already git for storing the files. DVC has been calling itself git for data and models since 2020. There is even a research tool called Git-Theta that tried to make git understand the model itself, not just a giant file. What I cannot find is one public place where I branch a model, train it on my corrections, and leave a better version for the next person. If that already exists and I missed it, I want to hear it.

That's the future I'm trying to poke at. One place people can train and post their own models, collaborate on them, and actually improve the thing. Some of those training sets would be public and some would stay private, and that's fine. I don't have this built. I haven't trained the medical-document model yet. I just know prompting and skills are not getting us there, and I would rather be wrong in public than keep adding more skills.

If you have already been doing this, or if this is dumber than I think, I want to hear it.