Language Weaver is pleased to announce that its software showed the highest accuracy among software programs that are commercially available, on the official evaluation metric used in government-conducted evaluations. The evaluations, called MT-06, were scored by the National Institute of Standards and Technology (NIST), a non-regulatory agency of the U.S. Commerce Department’s Technology Administration. NIST conducts these evaluations in order to support machine translation (MT) research and help advance the state of the art in MT technology.
For those who haven’t heard of the MT-06 evaluations before they are basically the “World’s Strongest Man” competition of the machine translation space. Participants (more than 40 this year) are given two translation tasks. This year it was translating a chunk of content from Arabic to English and then another chunk from Chinese to English.
Participants are classified into different subcategories based on several factors, including whether or not it is a commercially available system or research oriented and secondly the processing horsepower running in the background.
Both are important distinctions as they can heavily affect the final result, imagine if you will a car race. You can give everyone the same track but some guys may have a Honda Civic while others bring their Ferrari F1 car. It’s clear the F1 will beat the Civic hands down when it comes to going around the track but it’s entirely impractical for any other purpose.
Research vs. Commercial represents the “practical” considerations whereas the background infrastructure represents speed & performance. By creating subcategories it allows companies like Language Weaver, who build a practical piece of software that performs exceptionally on “minimal” hardware shine while still being compared with a research driven project that doesn’t necessarily need to worry about being broadly usable and can instead focus purely on performance.
Language Weaver appears to have performed well across the board but they had their biggest success in the Arabic to English task where they were rated top of the pile in the large data and unlimited data tracks and well as many other sub-categories.
Congratulations all around to the team at Language Weaver.
For the data geeks out there the results are here. It’s worth noting Google ranks top of the pile on just about everything. At the end of the day I don’t believe they’re considered “Commercially available” and from what I’ve heard they also have an insane amount of computing horsepower backing up their system – and from what I’ve been told it can make a huge difference in the results.