I think i’ve only once flat out told one it was wrong about a specific assertion I quoted and it immediately was able to find its way to what I knew to be the correct claim.
I just wonder what would happen if i was in fact mistaken and I told it confidently it was wrong without elaborating
What’s the difference between converting the characters and numerals into a response based off their syntax vs understanding them? How can you establish, as the person submitting the notes, whether whoever is in the Chinese room is the former or the latter?
I remember reading something about LLMs not being able to learn “x is y” equivalence relations. Can’t find it now but limitations like this are what make differences clear between what humans do and what we’ve managed to teach the neural network (which will be used to iterate and improve the model further, of course)
In the Chinese box analogy, this would be like them knowing cats are considered cute but not whether considered-cute animals include cats (if I remember the limitation type correctly). If you happen to slip the right instructions/questions, something they’ve seen before or something they’re capable of extrapolating, then nothing seems off; but if someone can say in one paragraph that cats are cute but they know of no cute animal, you’d not think they’re understanding what they’re saying, and so don’t really understand the language even if they give you plausible words in all other cases
(For cats it’ll work because there’s a billion example sentences out there. LLM vendors are also trying to sidestep such problems by having it generate a bunch of tangential text (in which it might happen to regurgitate the tokens it needs to piece together the answer) before answering the prompt, but that’s still not being able to apply logic)
Intentionality is the key difference. You can eventually tell a Chinese room’s nature by giving it new variables that it hasn’t encountered before. New problems lead to algorithmic breakdown.
That said, there’s deeper conversations you can have about what consciousness truly is, of course. My personal view is that it requires a level of complexity that we are still very far away from architecturally, and a level of scalability that we may not even be able to support ecologically. This thought experiment is mainly to show you what the inner workings of a computerized process can look like, and works to provide a demystified perspective.