I ask 100 information questions to four digital assistants. All of them fail at least half.
Vlad / 2 days ago
vlad.d2dx.com
snip:
The best performing assistant, on information queries, is Google. This is perhaps not surprising, given that they are a search engine. The same crutch is used by Alexa to squeeze ahead of Alexa by one answered question, even though it gave more partial answers. (On the scoring methodology, Alexa would come out ahead, as it had a greater proportion of adequately answered questions). Siri, which never read a search result, came last in number of questions answered, and even managed a greater proportion of half-answered questions than Alexa, usually because of requiring additional input for things Alexa could understand.
Before we proceed, let’s acknowledge that I asked freeform questions the way a human would. The fact that I get an answer, let alone an answer at least a quarter of the time is impressive, from a pure technological standpoint. It is all the more impressive when considered from the standpoint of the hypothetical Amalgam assistant: a theoretical construct that takes the best answer from the four assistants. With 47 correct and 26 partial answers, this gives an amazing 73% success rate. |
