LLMs can’t even stay on topic when specifically being asked to solve one problem.
This happens to me all the damn time:
I paste a class that references some other classes which I have already tested to be working, my problem is in a specific method that doesn’t directly call on any of the other classes. I tell the LLM specifically which method is not working, I also tell it that I have tested all the other methods and they work as intended (complete with comments documenting what they’re supposed to do). I then ask the LLM to only focus on the method I have specified, and it still goes on about “have you implemented all the other classes this class references? Here’s my shitty implementation of those classes instead.”
So then I paste all the classes that the one I’m asking about depends on, reiterate that all of them have been tested and are working, tell the LLM which method has the problem again, and it still decides that my problem must be in the other classes and starts “fixing” them which 9 out of 10 times is just rearranging the code that I already wrote and fucking up the organisation that I had designed.
It’s somewhat useful for searching for well-known example code using natural language, i.e. “How do I open a network socket using Rust,” or if your problem is really simple. Maybe it’s just the specific LLM I use, but in my experience it can’t actually problem solve better than humans.
Yeah I find LLMs most useful to basically read the docs for me and provide it’s own sample/pseudocode. If it goes off the rails, I have to guide it back myself using natural language. Even then though it’s still just a tool that gets me going in the right direction, or helps me consider alternative solutions buried in the docs that I might have skimmed over. Rarely does it produce code that I can actually use in my project.
LLMs can’t even stay on topic when specifically being asked to solve one problem.
This happens to me all the damn time:
I paste a class that references some other classes which I have already tested to be working, my problem is in a specific method that doesn’t directly call on any of the other classes. I tell the LLM specifically which method is not working, I also tell it that I have tested all the other methods and they work as intended (complete with comments documenting what they’re supposed to do). I then ask the LLM to only focus on the method I have specified, and it still goes on about “have you implemented all the other classes this class references? Here’s my shitty implementation of those classes instead.”
So then I paste all the classes that the one I’m asking about depends on, reiterate that all of them have been tested and are working, tell the LLM which method has the problem again, and it still decides that my problem must be in the other classes and starts “fixing” them which 9 out of 10 times is just rearranging the code that I already wrote and fucking up the organisation that I had designed.
It’s somewhat useful for searching for well-known example code using natural language, i.e. “How do I open a network socket using Rust,” or if your problem is really simple. Maybe it’s just the specific LLM I use, but in my experience it can’t actually problem solve better than humans.
Yeah I find LLMs most useful to basically read the docs for me and provide it’s own sample/pseudocode. If it goes off the rails, I have to guide it back myself using natural language. Even then though it’s still just a tool that gets me going in the right direction, or helps me consider alternative solutions buried in the docs that I might have skimmed over. Rarely does it produce code that I can actually use in my project.