11/2/99 article on ACVC vs. others. Very good on the technicals.
November 2, 1999
--------------------------------------------------------------------------------
By James Gifford, Computer Telephony Magazine
--------------------------------------------------------------------------------
Unified Messaging Shootout
Active Voice Picked As Overall Winner
The following content is gleaned directly from last month's issue of Computer Telephony Magazine (to subscribe online, go to: telecomlibrary.com.
It should be duly noted that all tests and product analysis for this review were performed by CT Labs, an independent third-party CT product testing and certification lab. CT Labs works under contract to Computer Telephony to provide these testing services and is not directly affiliated with the magazine or its publisher, Miller Freeman/CMP.
CT Labs clients include CT product manufacturers and call centers. The company offers a range of in-lab and on-site services that draws from the considerable CT experience of its management, staff and consultants. For further information, you may contact CT Labs via phone at 916-784-7870 or through its Web site at www.ct-labs.com.
For a complete overview of CT Labs testing methodology for this review, go to: ct-labs.com
It's pretty cool. You can even listen to email text-to-speech audio samples for each product there.
Meanwhile, here's the online recap of the conclusion to Computer Telephony's Unified Messaging Shootout:
BASIC DEFINITIONS
Unified messaging is the attempt to make three widely differing message types easy to receive, reply to, and manage. Most current solutions work by providing the back-room technology to put voicemail and fax messages into an e-mail-like manager, such as Microsoft Outlook.
There are a number of ways to accomplish this merger, from using a single server with truly unified messaging management to combining the contents and features of three separate servers behind a single user interface. We decided it was time for a steely-eyed look at a representative sampling of unified messaging systems.
THE CONTENDERS
The term "unified messaging" has been slopped around quite a bit in recent years and applied to everything that even loosely combines multiple message types. Even if you're strict in your definition of the term -- voicemail, fax, and e-mail managed from a single user interface -- there's no shortage of vendors, and offerings vary immensely.
We wanted to keep the test to a manageable size so that the results would be meaningful. The starting point, then, was to select a representative subset of the unified-messaging spectrum. Limiting participants to those who offered a UM solution that followed the above definition (which could include some integrated messaging solutions) was the first step. But after that, the list was still immense.
We then looked at markets and decided that the industry's main thrust was to appeal to midsize businesses. That cut out the SOHO and small enterprise-only vendors, as well as the heavyweight-only companies -- interesting classes in themselves, but our goal was a manageable test scenario.
That reduction still left a lot of potential players, so we filtered the list again by unifying technology. We selected systems that used the most common unifying element, Microsoft Exchange Server. That brought the list to an acceptable length and, once again, it represents the mainstream of what midsize companies will be evaluating. From there, it was a matter of lining up the following four vendors whose wares represented this arena well.
ACTIVE VOICE
Active Voice ( www.activevoice.com) is one of the largest sellers of voice-processing platforms. Best known for its OS/2-based Repartee messaging system, Active Voice moved into the NT world with Unity, a completely new messaging platform that embraces NT, TAPI, and Exchange Server. Unity is a truly unified solution, with voice, e-mail, and fax managed in traditional single-directory fashion.
Designed to be the MS BackOffice telephony component, Unity ships bundled with Microsoft's Exchange Server and Internet Information Server (IIS). Fax services are provided by Active Voice's Lighting Fax fax-server product.
AVT
AVT (www.avtc.com) produces CallXpress, a scalable enterprise-class unified messaging system. CallXpress is unique among the products tested in that it supports unified and integrated messaging models. After evaluating the features of both, CT Labs elected to test CallXpress in integrated messaging mode.
The fax component of AVT's offering is provided by its RightFax product.
KEY VOICE
Key Voice (www.keyvoice.com) makes a complete line of voice and unified messaging products, from its tiny standalone Debut through Lite and Small Office versions of the product we tested, Corporate Office NT. Key Voice's offering is unique among the products tested in that the fax services are part of the basic bundle, instead of being an add-on component.
LUCENT TECHNOLOGIES
Lucent (www.lucent.com) is no stranger to any area of telephony. If it didn't develop it (remember, Lucent founded Bell Labs), it picked up the best through careful acquisitions. Lucent's messaging products evolved from the 1996 acquisition of Octel. Octel Unified Messenger is one of the most feature-rich and robust messaging platforms available.
Fax services for the system were provided by the OEMed Faxination fax server software.
THE TEST SYSTEMS
Each vendor was asked to provide the following components:
A unified messaging phone server with eight voice ports, eight text-to-speech resource ports and the minimum amount of speech recording space available
Compatible fax services, as an integral component or a separate server, with four fax-line resources (configured as two send/two receive, if appropriate)
PBX integration for the CT Labs test switch (a Lucent Partner ACS)
Two days of on-site assistance with setup of the system at CT Labs
TESTING OBJECTIVES
For this half of the test, CT Labs set out to answer the following questions about these Unified Messaging systems:
How easy are these systems to maintain -- that is, how easy is it to add, move and delete users? Are multiple user directories required, e.g., separate ones for voice, fax, and email servers or can a single directory entry handle all message types?
How effective are remote fax access and email-reading features of these products?;
What is the quality of the voice server's speech prompts, recordings, and text-to-speech engine?
How seamless is the integration with MS Outlook?; Is there email integration support for email clients other than Exchange? How do they compare, feature-wise?
CT LABS EVALUATIONS
CT Labs evaluated three areas of these products in addition to the formal performance testing. Evaluation is necessarily subjective, so various end users may have different opinions of this material's quality and suitability. CT Labs attempted to make their evaluations from the perspective of a mainstream user. All rankings are on a 0 to 10 scale.
USER GUIDES & ONLINE HELP EVALUATION
CT Labs evaluated each product's user guides and online help for both administrators and end users. Documentation was assessed for quality and usability, both in a general evaluation and as it was used during the setup and testing process.
Results were as follows:
Admin User Guides: Active Voice -- 7.5; AVT -- 7.5; Key Voice -- 7.6; Lucent -- 7.1
User Guides: Active Voice -- 8.2; AVT -- 9.0; Key Voice -- 7.1; Lucent -- 8.7
Admin Online Help: Active Voice -- 9.0; AVT -- 8.5; Key Voice -- 8.0; Lucent -- 7.0
User Online Help: Active Voice -- 9.0; AVT -- 10.0; Key Voice -- 0.0; Lucent -- 10.0
Overall: Active Voice -- 8.4; AVT -- 8.7; Key Voice -- 6.1; Lucent -- 8.1
Note: Key Voice was hurt by the fact that they did not provide online help for its client interface.
Active Voice had four admin guides totaling 631 pages, plus an end-user guide of 72 pages. All received high marks from the reviewer. Online help was also good, but the point-and-click help feature did not work in the version tested, resulting in a minor loss of points.
AVT includes a 167-page admin guide and a CD-ROM with a large quantity of supplementary admin material. The evaluators had trouble locating relevant information on the disk, since it appears to include documentation for all of AVT's products, but eventually found about 100 additional pages of related material in PDF format. Reviewers thought the 17-page end-user guide was excellent, but the accompanying multimedia training and tutorial Coach CD-ROM got even higher marks for creativity, content and production.
Key Voice supplies two admin guides with a whopping 830 pages, but reviewers found the organization of the guides to be excellent, with liberal use of section dividers making it easy to locate needed information. Reviewers found the end-user guides, a 14-page booklet and a fold-up quick-reference, to be a bit lean, but Computer Telephony editor James Gifford's own examination found them complete and easy to follow. Again, testers were unable to locate any help topics for the user client, resulting in a score of zero in this category.
Lucent had three administrator guides totaling 535 pages. Although the information was complete, reviewers found it was harder than necessary to locate desired information and felt that the guide could have been better organized. The troubleshooting sections were also somewhat thin and vague. The admin online help was good, although no help was found for fields on the fax server administration tab. The three end-user guides (a total of 180 pages) were judged very good, as was the end-user online help.
Active Voice and AVT come away with a near-tie win for this category.
OUTLOOK CLIENT EVALUATION
Most users of these products will interact with the messaging system through the dial-up TUI or the desktop MS Outlook GUI. All of these products modify the basic Outlook interface to add more tabs, fields and features for message management. CT Labs thoroughly evaluated the modified interfaces for the seamlessness of their voice and fax message integration and the smoothness of their message management tools.
Active Voice racked up a perfect score for its Outlook integration. The reviewers found it to be very seamless, with the cleanest and most intuitive interface of those evaluated. Message retrieval was also very quick.
AVT also scored very well with its GUI. It lost points for a one-time anomaly encountered when switching from the local soundcard to virtual soundcard operation: The switch caused the interface to hang, with an hourglass wait cursor, for about a minute.
Key Voice scored significantly below the other products for a number of reasons. Most significantly, it lacks the virtual soundcard feature, requiring a soundcard in the local PC for voice messaging. It also uses the same icon for both fax and voice messages, requiring the user to examine the Subject field to determine the message type. The general assessment was that iNTerchange is a young product and flaws such as these will disappear with time.
Lucent had good Outlook integration, but a few glitches lowered their score. The fax integrated display window was too small and did not display faxes legibly in most cases. (Users can easily switch to the default Windows fax viewer, but the additional step should not be necessary for normal fax viewing.) Switching from soundcard to virtual soundcard and back was more difficult than the testers felt it should have been. Each such change required the message review dialog to close and reopen. Testers also felt that there were some unnecessary restrictions in the message reply options (for example, there was no way to reply to a voice message from an outside caller -- not even via email).
Active Voice is the clear winner in this evaluation.
ADMINISTRATIVE CLIENT EVALUATION
The admin client is a critical part of any networked, multi-user application. In many companies, a system administrator will need to monitor and make changes to the system on a daily basis, or even more frequently. The ease of use, especially for common and repetitive functions, is important.
Active Voice uses the only web-style administration GUI of those tested. Testers found it easy to use, requiring eight fields to be filled out to install a new user. An import utility lets administrators create a block of new users in one operation. The browser-like interface resulted in rather slow screen redraws, which made usage a little clumsy.
AVT has a very good admin client that requires a total of 11 fields to be filled out for each new user. It also has an import feature to allow creation of a block of new users in one operation. Testers particularly liked the well-implemented search feature, which assists in finding user records quickly.
Key Voice uses text-file import for a lot of configuration steps. One such utility lets administrators create a block of users by importing a four-column text file, which can be easily prepared in a number of ways. Testers found this import process to be efficient for some steps but somewhat clumsy for most configuration, and would have preferred more configuration access via the form-based GUI. Single users can be added to the system via a GUI with a total of 10 fields.
Lucent tightly integrates its configuration utility by adding two tabs to the Exchange Server admin client. Administrators with Exchange Server experience (which should be nearly all of them) will find this utility easy to use. New users can be added by filling in a total of 11 fields.
Lucent gets the tick for this category.
RECORDING BANDWIDTH & QUALITY TESTS
Recorded speech quality and recording bandwidth are important measurements of any voice-messaging system. Low speech quality can result in misunderstood messages. Even if speech quality is not so poor as to distort messages, harsh or crackling audio quality quickly becomes irritating to users and callers alike.
CT Labs applied the very sophisticated PAMS (Perceptual Analysis Measurement System) speech analysis technique to evaluate recording quality on each system. The sample recordings are made by playing a very high-quality standard reference recording into the system, recording the sample as played back by the system, and comparing the two.
In prior tests of speech recording systems, CT Labs has established that the automated PAMS scoring technique is the next best thing to using the industry-standard MOS (Mean Opinion Score) testing. MOS scoring, provided by companies like Dynastat, uses a live listening panel and consequently can be expensive and time-consuming. An automated process that can be used in its place under less-demanding circumstances is a boon.
PAMS evaluates both speech listening quality and speech listening effort.
The amount of disk space required to store speech messages is also important, since storage capacity is limited. However, a smaller recording bandwidth often translates to a reduction in recording quality. Most speech recording systems permit the selection of the recording bandwidth. Lower settings maximize available recording time, while higher settings increase speech quality. CT Labs performed all speech quality testing using each manufacturer's default bandwidth setting.
Active Voice showed the highest speech quality scores while using a recording bandwidth that was double or more that of the two most efficient systems.
AVT scored only a tick behind Active Voice, and managed to do so with a recording bandwidth almost three times narrower -- an astounding feat.
Key Voice scored slightly lower than AVT and Active Voice while using a relatively high recording bandwidth -- a disappointing showing.
Lucent came in a close last in speech quality scoring, but did so with the second-smallest bandwidth usage, just above that of AVT.
The winner in this critical category is AVT -- by a wide margin.
EMAIL TO TTS QUALITY TEST
When users need to access text messages email and fax header information from the TUI, the text-to-speech conversion quality is crucial. A slow, erratic, monotonous read-back (the infamous drunken Swede voice) can make use of a TTS interface intensely annoying.
Manufacturers have been working hard on this technology, developing and licensing faster, more powerful and more intelligent TTS engines. Sven's interpretation and diction have gotten much better as a result, and he doesn't seem to be hitting the aqvavit quite so hard these days either.
In a Unified Messaging system, TTS engines are particularly overworked by the peculiarities of email acronyms, abbreviations, strange codes, quote markers and generally weird words abound. The ability to read back email quickly, understandably and with a minimum of translation errors greatly increases the value of the messaging system to the everyday user.
Ideally, the TTS engine should be able to deal with common email abbreviations and read them out in expanded form (for your information, by the way, etc.). It should also be able to skip over system header information and quote marks.
CT Labs fed a complex email message into each system and had live listeners evaluate the result for quality. A single instance of TTS readback was recorded and the samples were played in double-blind fashion for each of the listeners. The scores were then averaged. CT Labs also counted the number of expansion errors, where the system either failed to expand a common email acronym or abbreviation or did so incorrectly.
Active Voice had the lowest quality scoring for TTS readback, but tied with AVT for the lowest number of expansion errors.
AVT virtually tied with Key Voice for quality scoring, but had two fewer expansion errors.
Lucent had the highest quality score by far, but also had twice as many errors as Active Voice and AVT.
While these tests might seem to indicate that TTS quality comes at the expense of expansion errors (i.e., that they're tied to shared CPU/DSP cycles; thus inversely related in the literal sense) this doesn't reflect the way most text-to-speech systems actually work. More likely, certain manufacturers (or the developers who sourced their OEM text-to-speech software) chose to expend more effort and time in building clever parsers than in tweaking output speech quality -- or the reverse.
This begs the question: Which is more important? And we're afraid the answer is very subjective. Interested readers should listen to the TTS readback recordings on CT Labs' website and see for themselves which factor (basic quality or number of expansion errors) is more important.
CT Labs' assessment is that all of the vendors could use improvement in this area. Fortunately, TTS technology is advancing rapidly; this year's marginal systems will likely be much more acceptable in future revisions.
FAX RECEIVE TEST
The ability of the Unified Messaging systems to receive faxes for individual users is of paramount importance. To test these systems' fax receiving capabilities, CT Labs dialed into each system's phone server TUI using the keys of a fax machine, and directed a test fax to a selected mailbox. The process was then repeated with a different mailbox.
The system response was monitored to determine the delay between the end of the fax transmission and the appearance of a new message icon on the associated user's workstation. The received fax quality was also evaluated.
The results were as follows: Active Voice had delays of 10 and 17 seconds in each of the boxes; AVT had delays of 30 and 50 seconds; Key Voice -- 21 and 19; and Lucent only 4 and 5 seconds.
Although Lucent was by far the quickest system in generating new-message notification and thus gets the nod as winner for this test, none of the systems showed excessive delay in this area.
Received fax quality was identical and excellent for all systems.
EXCHANGE SERVER LAN DISCONNECT TEST
In any multiple-server system, it is possible for one of the servers to go offline. How the system handles the loss and restoration of the server is a critical performance issue.
If the accidental disconnection of a server from the network will necessitate a shutdown and restart of the whole system, it's a black mark indeed for the manufacturer. If the system can tolerate the momentary loss of the server and recover quickly when it returns, it's a very favorable mark.
While Exchange Server is unavailable, the Outlook client will be unusable. The TUI client, though, should not necessarily be affected. CT Labs evaluated how well the TUI interfaces of these systems function in the absence of Exchange Server. To evaluate these systems' ability to handle the loss of Exchange Server, the following test was performed:
A call was placed into the phone server via a PBX station phone, and the caller logged into a mailbox
The Exchange Server system had its LAN connection unplugged;
The caller attempted to review as many of the pending voice and email messages as possible;
An attempt was made to record a voice message for a selected mailbox.
Then, after the Exchange Server LAN connection was restored, the testers monitored how long it took the system to restore proper call and message handling, and how long it took the sent message to appear at the destination user's mailbox.
Active Voice recovered quickly after reconnection. Attempts to review an email or voice message via the TUI while Exchange Server was disconnected resulted in a one-minute delay, followed by an error message suggesting the caller try again later.
AVT recovered quickly after reconnection. Attempting to review a voice message via the TUI while Exchange Server was disconnected resulted in a one-minute delay followed by a disconnection of the call, without any error message. Attempting to review an email message resulted in the same one-minute delay, after which the message header information was read out. There was then a 50-second pause, after which the system disconnected the call without an error message.
Key Voice was the hands-down star of this test. It recovered quickly after reconnection and exhibited no errors when checking voice and email messages while Exchange Server was disconnected. Key Voice manages this feat by maintaining duplicate copies of all email and voice messages on the phone server, minimizing the impact of an Exchange Server problem. (Key Voice, however, is the only system tested that cannot run in true unified mode, with all messages managed from a single server.)
Lucent took slightly longer to recover after reconnection, but only slightly so, and none of the systems was judged to have a problem in this area. Attempting to check a voice or email message resulted in the familiar one-minute delay, followed by an error message asking the caller to try again later.
The one-minute delay encountered by three of the systems is due to the phone server waiting for Exchange Server to respond, then timing out when no response is received. AVT loses points here for its awkward hangup without informing the caller of a problem or giving them a chance to make a different selection. Key Voice's flawless performance makes it the clear winner in this test.
PHONE SERVER LAN DISCONNECT TEST
Failure of the phone server will result in an inability of the system to handle any telephone access to messages. The failure of the phone server should not, ideally, cause message access problems for Outlook users.
To test these systems, CT Labs sent three new voice messages to a selected mailbox. While checking these messages from a network workstation, the LAN connection to the phone server was disconnected.
Since messages should be stored on the Exchange Server, this interruption should cause no problem for the workstation user. And indeed, that's just what CT Labs found. All four systems passed this test without fault.
PHONE SERVER POWER OUTAGE RECOVERY TEST
Although hefty UPSs are recommended for all critical network components, especially servers, the response of systems to power loss and recovery is a useful indicator of robustness. For this test, CT Labs disconnected power to each phone server for 30 seconds, then monitored how long it took the system to return to operation after power was restored. Testers also watched for operating system or server errors during and after startup.
All of the phone servers handled power loss and restoration well, with no detected errors. The Key Voice unit returned to service most quickly, at just over three minutes, while the others took just under five minutes each. Most users will find all these units' performance acceptable in this area.
BRASS TACKS: ACTIVE VOICE WINS!
When the dust settled, it was Active Voice's Unity that had racked up the winning score. Active Voice's experience with the older and very solid Repartee system seems to have given them a head start when developing an NT/Exchange Server-based Unified Messaging system.
What's astonishing is how close the four systems were at the finish line. A bare half-point separates the top and bottom scores (see the table for the various overall categories and relative weights given to the system areas).
Given that Unified Messaging systems are complex and require a number of components to integrate and work together smoothly, it would not have been surprising to see one or more of these systems tank badly under CT Labs' rigorous testing. None did. If you're ready for Unified Messaging, these systems are ready for you.
CONCLUSION
The days of dealing with three different interfaces to manage voice, fax and email are dead -- or at least should be. If a company's staff needs fast access to all three types of messages, at their desks and in the field, Unified Messaging offers a way to streamline communications with its single, easy-to-juggle inbox.
Not surprisingly, our testing shows that there is still room for improvement in the fringes. But those minor fixes will surely come, and they aren't reasons to stay with that creaky voicemail system any longer. If your business depends on fast, efficient command of communications ... unify 'em!
I remember a PR discussing it, but this is the actual article.
Jack |
