You test the test by mutating the code. Once you have 100% condition/decision branch coverage you can automatically sweep the code with changes and the tests will fail.
I have a little harness I use for this steps through each non-comment line of code changes signs, comparison directions, offsets values, swaps variables, adds negations, replaces computations with constants, etc. basically changes that are more or less guaranteed to compile.
Then it runs the compiler to produce a stripped optimized output, if the compiler is successful, it checks that the resulting md5 is different from all the prior results, runs the tests. If the tests pass, it saves the passing code (which, to be clear is a meta-test failure), and then later I sweep through and either determine that it managed to produce functionally equivalent code or I improve the tests (and fix the resulting bugs they expose).
The identical compiled binary test eliminates a lot of false positives.
But that kind of approach doesn't work unless you get to ~100% condition/decision branch coverage since obviously any condition that isn't tested will be free to mutate.
Hm. Maybe if I updated it I'll have some attempt at sampling LLM rewrites of functions. :P
Interesting approach. I hadn't considered actually implementing such "brute force" methods. I guess it's similar to fuzzing.
I think the problem is you are then moving your "real" condition/decision branch documentation into your tests. The tests are then basically a guard rail for just in case someone modifies some abstract bit of code and it changes the behaviour of some otherwise opaque decision branch. The approach in OP seems to be to just move the "real" logic/documentation into the code itself and do away with the abstractions (and perhaps the tests too).
Yes, I agree though the issue there is that changes which aren't believed to change the behavior might, as there isn't a way to tell except by being a very careful programmer and reviewer.
I have a little harness I use for this steps through each non-comment line of code changes signs, comparison directions, offsets values, swaps variables, adds negations, replaces computations with constants, etc. basically changes that are more or less guaranteed to compile.
Then it runs the compiler to produce a stripped optimized output, if the compiler is successful, it checks that the resulting md5 is different from all the prior results, runs the tests. If the tests pass, it saves the passing code (which, to be clear is a meta-test failure), and then later I sweep through and either determine that it managed to produce functionally equivalent code or I improve the tests (and fix the resulting bugs they expose).
The identical compiled binary test eliminates a lot of false positives.
But that kind of approach doesn't work unless you get to ~100% condition/decision branch coverage since obviously any condition that isn't tested will be free to mutate.
Hm. Maybe if I updated it I'll have some attempt at sampling LLM rewrites of functions. :P