What a prompt cannot build
A model will write you a working system faster than any team could. It will not tell you whether it is correct, and that is not a gap that closes as models improve — the missing information is not in the model. This is where the line actually falls, with an example from building this site.
What a prompt does produce, and what it does not
- It produces code. Good code, quickly, for problems that resemble problems it has seen. That part is real and it is not going backwards.
- It does not produce access. Credentials, a position inside the network, permission to touch a production system. No amount of capability earns a company's trust.
- It does not produce ground truth about your environment. Which version is actually installed, what that 2011 system actually returns, what the business rule really is once you account for the exception nobody wrote down.
- It does not produce consequence. When something fails, someone has to answer for it. That cannot be delegated to a system that has nothing at stake.
- It does not produce agreement. What the system should do is a negotiation between people with different interests, not a question with a correct answer.
Why these limits are structural, not temporary
The information is not on the internet
A model learns from public text. The behaviour of an undocumented machine, an internal process, a legacy system's actual quirks — that knowledge lives in equipment and in people's heads. There is no corpus to train on, so capability does not help.
Verification requires the thing being verified
A model can check whether code is internally consistent. It cannot check whether it matches a reality it has no access to. Those are different questions, and only the second one determines whether your system works.
Accountability cannot be automated by definition
The reason a signature matters is that something is at risk for the signer. A system with nothing at stake can produce the artefact but not the assurance, no matter how good the artefact is.
The hardest input is what people want
Most of the difficulty in a real project is deciding what to build, which is a disagreement between a finance team, an operations team and a founder. A model can write anything; it cannot settle that.
Where this shows up in practice
The example from this site
The contact form on this site was written with AI assistance. The tests were written too. They passed. The form was broken for every real visitor for two days: the tests posted JSON, and browsers post multipart. The code was correct; the assumption about the world outside it was not.
The version that is installed, not the one in the example
A configuration directive that is valid in nginx 1.25 and fatal in 1.24. Syntactically perfect, and it stopped the server from starting. That difference is not in the code; it is on the machine.
The rule that is correct and still wrong
A redirect rule that did exactly what it said, and interacted with an internal rewrite to produce an infinite loop. Reviewing the rule reveals nothing. Requesting the page reveals it immediately.
The input nobody questioned
A rate limiter counting requests per IP address, working perfectly, and counting the wrong thing: behind a CDN every request arrives from the CDN. Correct logic, wrong input, and no test would have caught it.
Five questions to ask about anything generated
- Was this exercised the way a real user will use it, or the way the author imagined it?
- Does it match the versions actually installed, or the ones in the documentation?
- If it fails at three in the morning, is there enough recorded to find out why?
- Who answers for it — and do they understand it well enough to answer?
- Was the question ‘is this correct?’ asked by someone who could tell?
Questions this raises
Is this an argument against using AI to write code?
No. Devsitia uses it constantly and it is a genuine gain. The argument is narrower: generating code and knowing whether it is right are different problems, and only the first one got cheaper. Treating them as the same is what produces systems that work in the demo.
Can someone with no software background build a real system this way?
They can build something that appears to work, and that is a real change worth taking seriously. What they cannot do is evaluate it. The person who could not write the code also cannot read it, so they have no way to tell a working system from one that has not failed yet. The barrier that moved was producing, not knowing.
Will this change as models get better?
Some of it, yes — and quickly. Anything that is a matter of capability will fall. What will not fall is the part where the required information does not exist in any training corpus, and the part where someone has to be accountable. Those are not difficulty problems.
So what is the practical advice?
Use the generation. Then verify against the actual environment: the real request format, the installed version, the running system. And keep a person who can tell the difference between code that is correct and code that is merely plausible, because plausible is what a model produces by construction.
How do you reduce the risk without giving up the speed?
Assemble verified components for the parts that are expensive to get wrong — authentication, permissions, payments, data integrity, audit — and generate freely everywhere else. The failure modes concentrate in a small number of places, and those are the places worth not improvising.
Start a conversation
Tell us what you are building, or what is currently breaking. You will get a straight answer from an engineer, not a sales script.