So far none of these models can write even a slightly complicated function well for me. I tried Mistral, ChatGPT, Qwen Coder 2, Claude, ... they apparently all fail when the solution requires to make use of continuations and such. Probably, because they don't have enough examples in their training data or something.
Example: Partition a linked list in linear time. None of these models seems to be able to get, that `reverse` or converting the whole list to a vector are in themselves linear operations and therefore forbid themselves. When you tell them to not use those, they still continue to do so and blatantly claim, that they are not using them. Á la:
"You are right, ... . The following code avoids using `reverse`, ... :
[code that still uses reverse]"
And in languages like Python they will cheat, because Python's list is more like an array, where random access is O(1).
This means they only work well, when you are doing something quite mainstream, where the amount of training data is a significantly strong signal in the noise. But even there they often struggle. For example I found them somewhat useful for doing Django things, but just as often they gave bullshit code, or it took a lot of back and forth to get something useful out of them.
I think it is embarrassing, that with sooo much training data, they are still unable to do much more than going by frequency in training data when suggesting "solutions". They are "learning" differently than a human being. When a human being sees a new concept, they can often apply that new concept, even if that concept does not happen to be needed that often, as long as they remember the concept. But in these LLMs it seems they deem everything that isn't mainstream irrelevant.
I just asked Gemini 2.5 Pro to write a function in Haskell to partition a list in linear time, and it did it perfectly. When you say you were using ChatGPT and Claude, do you mean you were using the free ones? Plain GPT 4o is very poor at coding.
-- | Takes a predicate and a list, and returns a pair of lists.
-- | The first list contains elements that satisfy the predicate.
-- | The second contains the rest.
partitionRecursive :: (a -> Bool) -> [a] -> ([a], [a])
partitionRecursive _ [] = ([], []) -- Base case: An empty list results in two empty lists.
partitionRecursive p (x:xs) =
-- Recursively partition the rest of the list
let (trues, falses) = partitionRecursive p xs
in if p x
-- If the predicate is true for x, add it to the 'trues' list.
then (x : trues, falses)
-- Otherwise, add it to the 'falses' list.
else (trues, x : falses)
My Haskell reading is weak, but that looks like it would change the order of elements in the 2 lists, as you are prepending items to the front of `trues` and `falses`, instead of "appending" them. Of course `append` is forbidden, because it is linear runtime itself.
I just checked my code and while I think the partition example still shows the problem, the problem I used to check is a similar one, but different one:
Split a list at an element that satisfies a predicate. Here is some code for that in Scheme:
(define split-at
(λ (lst pred)
"Each element of LST is checked using the predicate. If the
predicate is satisfied for an element, then that element
will be seen as the separator. Return 2 values: The split
off part and the remaining part of the list LST."
(let iter ([lst° lst]
[index° 0]
[cont
(λ (acc-split-off rem-lst)
(values acc-split-off rem-lst))])
(cond
[(null? lst°)
(cont '() lst°)]
[(pred (car lst°) index°)
(cont '() (cdr lst°))]
[else
(iter (cdr lst°)
(+ index° 1)
(λ (new-tail rem-lst)
(cont (cons (car lst°) new-tail)
rem-lst)))]))))
For this kind of stuff with constructed continuations they somehow never get it. They will do `reverse` and `list->vector`, and `append` all day long or some other attempt of working around what you specify they shall not do. The concept of building up a continuation seems completely unknown to them.
I get this kind of lying from Gemini 2.5 Flash sometimes. It's super frustrating and dissolves all the wonder that accumulated when the LLM was giving useful responses. When it happens, I abandon the session and either figure out the problem myself or try a fresh session with more detailed prompting.
Example: Partition a linked list in linear time. None of these models seems to be able to get, that `reverse` or converting the whole list to a vector are in themselves linear operations and therefore forbid themselves. When you tell them to not use those, they still continue to do so and blatantly claim, that they are not using them. Á la:
"You are right, ... . The following code avoids using `reverse`, ... :
[code that still uses reverse]"
And in languages like Python they will cheat, because Python's list is more like an array, where random access is O(1).
This means they only work well, when you are doing something quite mainstream, where the amount of training data is a significantly strong signal in the noise. But even there they often struggle. For example I found them somewhat useful for doing Django things, but just as often they gave bullshit code, or it took a lot of back and forth to get something useful out of them.
I think it is embarrassing, that with sooo much training data, they are still unable to do much more than going by frequency in training data when suggesting "solutions". They are "learning" differently than a human being. When a human being sees a new concept, they can often apply that new concept, even if that concept does not happen to be needed that often, as long as they remember the concept. But in these LLMs it seems they deem everything that isn't mainstream irrelevant.