@Olivier Galibert @Charlie Stross In this circumstance they would do pretty well in terms of hallucinating. The problem with hallucination is that we haven't solved the problem of how to get it to say "I don't know". So basically any time the correct answer is some form of "I don't know" it just invents something that looks like an answer.

When trained off of your own works and using it to inspect those same works, the only time it would hallucinate is when you ask it about something that's nowhere in your works... at which point it kinda incidentally functions for helping do worldbuilding. Just so long as you're knowledgeable enough of you own works to recognize what is actually there.

It's important to recognize that AI was developed mostly as a solution without any more than a vague problem. Practically everything we're doing with these AIs is unintended functions... and that creates a lot of problems people eventually need to understand (hallucinating being one of them).

(The worst being the information apocalypse that is already underway...)

@Charlie Stross @Pseudo Nym It's hard to argue with "autocorrect on steroids" but that doesn't mean there isn't intelligence.

One of the things I think makes all of this so much messier is the fact that everyone is talking about "intelligence" as a concrete well defined thing, as opposed to a word that the only acceptable usage is just as a synonym for human (because you will never find a better definition of intelligence that won't exclude many human beings and include many things you wouldn't call intelligent).

It's hugely important, especially for people with cognitive disabilities, to recognize that it does in fact have some limited cognitive functions.

There are a great many people already using it to help them accommodate disabilities like executive dysfunction (because it very clearly has executive functioning, and notably better executive functioning that most people with ADHD and many autistics).

It has contextual understanding that is enough to help assist autistics trying to make sense of allistic bullshit, and can basically translate communication between different neurotypes. Let alone the fact that it can perform literal high quality translation between languages when it wasn't even designed to do that.

Or the fact that it literally can debug code.

I argue that it's genuinely intelligent, but it currently possesses only a very very limited set of cognitive functions.

We, on the other hand, will literally never accept any AI as intelligent (let alone anything else for that matter) until our own thinking changes. Because we ultimately consider intelligence magical and human. We fundamentally believe that if we even vaguely understand how it works, it can't be intelligent (as if all the arguments there didn't apply to human minds as well). And we refuse to believe that non-human-like intelligence is valid (ie. that animals have actual thoughts and feelings, let alone that they can even understand rudementary language... while they show rudimentary skill at sign language or talk to us using essentially the same tools that many non-verbal autistics on for communication).