3.1.1. Unit selection generates the most natural speech

Consistently, in every challenge, the system that has been rated as the most natural by listeners has always generated the speech signal by concatenating recorded samples of speech. The size of these units has varied somewhat, as have the methods for selecting and concatenating them, but it is striking that listeners consistently rate recorded speech containing inevitable concatenation artefacts as sounding more natural than speech generated using a vocoder. Nevertheless, whilst listeners might say such speech is more natural, they generally find it harder to understand than speech for a vocoder driven by a statistical parametric model.

The Challenge has seen many ‘classical’ unit selection systems that closely follow Hunt and Black (1996) The most prototypical of these is the Festival system, with its ‘multisyn’ unit selection engine (Clark et al., 2005, 2006; Richmond et al., 2007). This system was adopted as a benchmark in later challenges, allowing some limited comparisons across different years of the challenge to be made (e.g., was a system better or worse than Festival?). Another classical unit selection, which like Festival has its roots in earlier ATR systems, is Ximera (Toda et al., 2006).

Many, many other similar unit selection systems have been entered into the Challenge, with varying degrees of success. This variation points to the fact that unit selection, whilst appearing to be fairly simple, requires a great deal of engineering skill to obtain really good results. These classical unit selection are those described in Weiss et al. (2007), DRESS (Rosales et al., 2008), ILSP’s system (Raptis et al., 2010, 2011, 2012; Chalamandaris et al., 2013), MILE (Kumar et al., 2013), Nokia’s NTTS (Ding and Alhonen, 2008), Ogmios (Bonafonte et al., 2007, 2008), RACAI (Boros et al., 2013) SVOX (Wouters, 2007), VoiceText (Jun et al., 2007), and WISTON (Tao et al., 2008, 2009, 2010).

The inevitable creep of statistical techniques. As soon as the good performance of HMMbased (Section 3.1.2) and later hybrid (Section 3.1.3) synthesisers was demonstrated, many unit selection systems entered into the Challenge started to adopt statistical methods. Jess was initially a classical unit selection system in its first appearances (Cahill and Carson-Berndsen, 2006, 2007) but later added an HMM-based prosody model (Cahill et al., 2011). OpenMary also evolved from unit selection (Schroeder et al., 2006; Schroeder and Hunecke, 2007) by adding a statistical join model (Schroeder et al., 2008) and continues to participate in the Challenge with both unit selection (Schroeder et al., 2009; Charfuelan et al., 2013) and HMM-based (Section 3.1.2) systems. The I²R system likewise has evolved from classical unit selection (Dong et al., 2008, 2009, 2010) to a system employing HMM-guided unit selection (Dong et al., 2011; Lee et al., 2013). Predating these though, is the clunits system Black and Taylor (1997), first entered in 2008 – see Section 3.1.3.

Newcomers can build great unit selection systems too. Many unit selection systems entered into the Challenge do not actually perform any better than Festival, so have to be seen mainly as as learning exercise for the participating teams and not a contribution to knowledge. However, the ability to build excellent unit selection systems can be developed independently, as demonstrated by a couple of ‘newcomers’ (from a speech synthesis community point of view). One notable entry into three of the Blizzard Challenges is the classical unit selection system IVONA (Osowski and Kaszczuk, 2006; Osowski, 2007; Kaszczuk and Osowski, 2009) from a previously little-known Polish company. This system achieved out-standing results; the company was subsequently acquired by Amazon. Another previously little-known company has also entered very respectable unit selection systems into the Challenge: Lessac’s method uses a unit called the Lesseme (a kind of phonetic/prosodic-contextdependent unit) to very good effect (Nitisaroj et al., 2010, 2011). The reason that the Lesseme works is probably that it hardcodes some of the key target cost features into the unit type, rather than being radically different from more common units like diphones. What do we learn from such systems? We see that unit selection continues to be the obvious choice if building a commercial product; that, with the right engineers, it delivers very high naturalness. The executive summary is pretty clear: if we don’t care about controllability, expressivity, or having a library of many voices, and we have the time, money and the right people to do the engineering, then we should choose unit selection every time.

Taking a little more risk. The above systems were entered into the Challenge principally to benchmark them against other systems, although typically participants that take part more than once do generally report that the Challenge has helped them improve their systems. On the other hand, some participants in the Challenge use it as an opportunity to take a little more risk and try new ideas. Cerevoice experimented with compressed waveforms (Aylett et al., 2007) in one Challenge, and a form of data cleaning based on genre pruning in another (Andersson et al., 2008). Some have even used Blizzard as a way to develop research methodology (Kominek et al., 2005).

Voice conversion. Blizzard requires that the entered voice sounds close to the provided speaker, which usually means building a voice on that data. Only two unit selection entries have done differently, by starting from an existing voice. The IBM system of 2005 used speaker transformation (Hamza et al., 2005), and a system based on the Festival front end with the AhoTTS waveform concatenator, which first entered in 2008 (Sainz et al., 2008), also applied voice conversion in 2009 (Sainz et al., 2009).

Non-uniform units. For engineering simplicity, most systems employ a single unit type such as the diphone or half-phone, but a few try to extend this to non-uniform units. Examples from the Blizzard Challenge include Ding and Alhonen (2007), Yang et al. (2006) which also employs an HMM-generated prosody target, the DSSP system (Latacz et al., 2008) which later added a statistical target cost (Latacz et al., 2009) and trainable context-dependent target cost weights (Latacz et al., 2010), and a system using syllable-sized units plus back-off (Raghavendra et al., 2008).

Learning the unit type and constructing synthetic units. Almost all unit selection systems used expert-defined types (e.g., diphones) as the acoustic unit. Two exceptions to this are the IBM unit selection systems which use HMM state-sized units (a fraction of a phone) and employ HMM state clustering to identify classes of interchangeable units (Eide et al., 2006; Fernandez et al., 2008).

Another departure from the usual type of unit is Toshiba’s ‘plural unit selection and fusion’ approach which constructs units by automatically merging together several recorded instances (Buchholz et al., 2007; Li et al., 2008, 2009). Other systems also try to overcome the limitations of units available in the original recordings by constructing additional units either through concatenation (Aylett et al., 2006) or using HMMs (Aylett and Pidcock, 2009), in an offline procedure known as ‘bulking’. It’s worth re-iterating at this point that we are only concerned in this paper with systems entered into the Blizzard Challenge, and are not attempting to trace ideas back to their inventors.

3.1.2. Statistical parametric methods generate the most intelligible speech

In contrast to the unit selection approach, systems which employ statistical parametric models to drive a vocoder are generally rated as less natural-sounding by listeners. Nevertheless, the same listeners can transcribe this ‘less natural’ speech more accurately than unit selection output. The Blizzard Challenge has witnessed the most important period of progress for statistical parametric models. The first Challenge already saw the use of the high-quality vocoder that has become the most widely used (STRAIGHT) and explicit duration models (hidden semi-Markov models: HSMMs) (Zen and Toda, 2005) and subsequent years saw systems employing a vast array of enhancements such as MGC-LSP acoustic features which combine the benefits of cepstral and all-pole representations of the spectral envelope, and global variance (GV) (Zen et al., 2006), minimum generation error training (MGE) (Ling et al., 2006, 2007), formant enhancement (Oura et al., 2009), trajectory training (Maia et al., 2009), the use of GV during training along with trainable mixed excitation (Shiga et al., 2010), minimum generation error linear regression (MGELR) model adaptation (Oura et al., 2010), adjustments to the perceptual scales used to represent acoustic features (Yamagishi and Watts, 2010), deterministic annealing expectation maximisation (Hashimoto et al., 2011) and ‘chapter-adaptive training’ to cope with changes in recording conditions within audiobook training data (Takaki et al., 2013).

Adaptive models. High intelligibility might be a very attractive property, but was discovered in the course of evaluation and was not specifically designed or claimed as a feature of these systems. On the other hand, a ‘killer feature’ of the statistical parametric framework, that is designed right into the system and is one of the main claims of proponents of the statistical approach, is the ability to modify the underlying model parameters. This is most commonly achieved using adaptation techniques borrowed from ASR, then subsequently extended for TTS. Blizzard entries have used supervised speaker adaptation (Yamagishi et al., 2007, 2008) as well as unsupervised adaptation (i.e., with word transcripts obtained using ASR) (Yamagishi et al., 2009), as an effective way to leverage pre-existing recordings of other speakers when constructing a voice for that year’s target speaker.

The spread of statistical parametric synthesis. Because almost all of the incremental advancements in statistical parametric modelling techniques are implemented in the HTS toolkit, they are available to everyone. This has spawned a great number of entries from what we might call ‘HTS users’ – groups that use the toolkit in an essentially unmodified form. Entries to the Challenge in this category include Scholtz et al. (2008), Liao and Wu (2009); Liao et al. (2010, 2012); Liao and Pan (2013), Louw et al. (2010, 2013), Nokia’s system combining their front end with HTS (Zhang et al., 2010), Cotescu (2011), some recent MARY entries (Charfuelan, 2012; Charfuelan et al., 2013), one of the entries from I²R (Lee et al., 2013) and a system which combined the flite front end with HTS (Dinh et al., 2013).

As with unit selection, not all of these are better than the HTS benchmark (employed alongside the Festival benchmark, to give an addition point of calibration from year to year). So, whilst statistical parametric methods might rightly claim to be more ‘automatic’ than unit selection, nevertheless a high degree of expertise and engineering skill is still required to obtain good results.

Improvements to the vocoder through source modelling. The hypothesis that the vocoder is the limiting factor in the naturalness of statistical parametric speech synthesis has led to various attempts to construct improved vocoders. Within the Blizzard Challenge, the most prominent strand of research in this area has focussed on improving the excitation source either by modelling residual signals (Maia et al., 2008, 2009), with a parametric glottal waveform model (Andersson et al., 2009) or by using sampled glottal pulse waveforms as in the GlottHMM system (Suni et al., 2010, 2011, 2012).

3.1.3. Hybrid systems: unit selection guided by a statistical parametric model

In the first few years of the Challenge, it became clear that statistical parametric systems consistently had the better intelligibility, whereas unit selection systems consistently had better naturalness. Although never formally proven, it is widely thought that this better naturalness was a result of using recorded waveforms – in other words, it is a local property of the signal that is partly independent of concatenation artefacts. Conversely, it is widely thought that the intelligibility of statistical parametric systems is a result of their ability to more accurately generate context-dependent speech units (as opposed to the out-of-context units of unit selection). An obvious next step was then to retain unit selection as the method for waveform generation – thus ensuring a natural-sounding signal – but to select the units using a statistical parametric model – thus taking advantage of its ability to predict the acoustic properties of units-in-context that did not occur in the available recorded corpus but that were needed at synthesis time.

Probabilistic models for unit selection. The hand-crafted nature of the join and target cost functions used in classical unit selection are often seen as unsatisfactory, since they must be tuned by ear and it is not possible to be sure that optimal values of the various parameters (e.g., weights on linguistic features) have been found. Overcoming this limitation has been a longstanding goal in unit selection research. Within the Blizzard Challenge, we have observed a number of systems tackling this problem. Sakai and Shu (2005) and Sakai (2006) describe a system evolved from MIT’s Envoice in which probabilistic models replace almost all hand-tunable parameters. Likewise, the ‘clunits’ method, first entered to the Challenge in 2008 (Black et al., 2008; Oliveira et al., 2008) builds clustering trees which group together acoustically interchangeable units which share a subset of linguistic feature values. Other attempts at trainable unit selection include the two early entries from µXac (Rozak, 2007, 2008) followed by the much improved system described in Rozak (2009) Lessac also entered systems in which an acoustic target, in this case from a Hierarchical Mixture of Experts, guided the selection of units (Wilhelms-Tricarico et al., 2012, 2013).

The weakness of most attempts to employ learned-from-data models in unit selection is perhaps that they pay attention only to acoustic similarity and do not involve human perceptual judgments. This is probably why a hand-tuned target cost is still better, if correctly constructed and tuned by an expert: it accounts for perceptual judgements.

Hybrid systems. We define ‘hybrid’ systems as those which employ a statistical parametric model – which is in itself capable of generation speech in conjunction with a vocoder – to guide the selection of units from the database, which are subsequently concatenated. There is of course not a clear dividing line: for example, the unit selection system described by Wilhelms-Tricarico et al. (2012, 2013) uses a powerful statistical model to predict an acoustic target trajectory, but without any intention of generating speech from it.

The first proposal of a hybrid system observed in the Blizzard Challenge was from Kominek and Black (2006), who mentioned both ‘clunits’ and HTS as candidates for the statistical parametric model, but actually used their own ‘ClusterGen’ method as the statistical parametric component; this is rather similar to decision tree-clustered HMM states, as used in HTS. The system was refined and entered again in 2007 (Black et al., 2007).

Subsequently, the ‘hybridistion’ of HMMbased synthesis with unit selection was developed and placed on a formal mathematical foundation in which the probabilistic nature of the HMMs was made use of. The sequence of highly-successful entries from USTC and their spinout iFlytek are strong evidence that this technique does indeed combine benefits of unit selection and statistical parametric models (Ling et al., 2007, 2008; Lu et al., 2009; Jiang et al., 2010). Subsequent systems of theirs experimented with Lessemes as the modelling unit (Chen et al., 2011), channeland expressiveness-related labels for audiobook data (Ling et al., 2012), automatic weight learning based on an objective quality model (Chen et al., 2013) and vocal tract resonance (VTR) trajectory-guided unit selection (Zhang et al., 2009).

In parallel to the USTC/iFlytek system evolution, Microsoft Research Asia (MSRA) have entered similar systems. The rather elegant name of ‘trajectory tiling’ was coined by them and featured in their 2010 entry to the Challenge (Qian et al., 2010). It alludes to a method used in computer graphics in which a parametric model (e.g., a wireframe or skeleton) is given a ‘skin’ composed from sampled images. The skeleton is convenient for the artist to manipulate and is flexible enough to produce any desired pose, whilst the detailed skin convinces the viewer that the object is real and not computer-generated. In speech, the corresponding advantages are that the underlying statistical parametric model is able to generate any speech sound in any context (the ‘trajectory’), whilst the overlayed samples (‘tiles’) provide the necessary details to convince the listener that the signal is natural speech.

In latter years, more groups have adopted various forms of the hybrid approach, including the NTNU (Meen and Svendsen, 2010), BUCEADOR (Sainz et al., 2011), and SHRC-Ginkgo systems (Yu et al., 2013).

4.1.1. Advances in objective measures

Although not directly used to rank the systems with the Challenge, objective measures of speech quality have made some progress over the last decade. Most notable is the work of Falk et al. (2008), Hinterleitner et al. (2010) and Norrenbrock et al. (2012) who have collectively pursued instrumental (that is, signal-based rather than listener-based) measures; these have begun to show useful results. These measures attempt to replicate the judgements that listeners would provide for a given set of speech signals. The Blizzard Challenge has been able to provide a substantial training set of signals-plus-listener-ratings on which object measures can be tuned and additional independent data sets on which their effectiveness can be tested.

4.1.2. Spinoffs and related evaluations

The Blizzard Challenge was itself inspired by the long tradition of common evaluation tasks from the field of ASR, and has in turn inspired others to use this methodology to measure (and hopefully promote) progress in other fields. The Hurricane Challenge (Cooke et al., 2013) evaluated methods for improving the intelligibility of natural or synthetic speech in the presence of additive noise, and its organisation closely followed the Blizzard model, with an open invitation to the community to participate, a common data set and set of rules, and a large centralised listening test run by the organisers. The Albayzin Challenges in 2010 (Díaz et al., 2011) and 2012 included a replication of the Blizzard Challenge, using a Spanish corpus.

4.2.1. What to evaluate

Naturalness and intelligibility remain the main evaluation criteria for speech synthesis, with judgements being elicited from listeners on a Lickert scale (Likert, 1932). Naturalness remains poorly defined, although listeners do seem to have a clear idea of what is being asked of them given the consistency of their judgements. Intelligibility is measured, as noted in Section 4.2.2, in a particularly unrealistic, or ‘ecologically invalid’, way.

Blizzard also adds an evaluation of speaker similarity to the mix. This was introduced initially only as a check that participants were using the provided recordings and not entering pre-built systems. With the advent of speaker adaptive approaches, and for unit selection en-tires employing voice conversion, speaker similarity became a useful dimension of the evaluation in its own right.

Despite continued calls by the organisers, few researchers in the community have risen to their challenge to propose new and better listening test designs, and in particular to propose what to evaluate. The only exception to this is Hinterleitner et al. (2011), who proposed a multi-dimensional test for evaluating synthetic audiobooks. Their method was adopted by the Blizzard Challenge organisers in those later years where audiobook data was used.

4.2.2. How to evaluate

Playing synthetic speech to listeners and asking them to make some response (e.g., provide a rating for a specified property) or perform a task (e.g., transcribe the words they heard) is the bread and butter of synthetic speech evaluation. Whilst objective measures have their place in single-system tuning or in identifying gross differences between systems, a listening test remains the only sure way to demonstrate the superiority of one’s proposed new method.

The problem of evaluating synthetic speech via listening tests is not a solved one. It is intrinsically difficult for two reasons. First, it is not clear exactly what properties to evaluate. Second, it is hard to know how to evaluate the chosen properties, and one can never be certain that all of the listeners have correctly performed the task you expected of them.

Blizzard takes a simple approach to alleviating these worries. The instructions given to listeners are generally simple and do not require any training or high level of knowledge on the listeners’ part. A large number of listeners is employed, thus minimising the effect of individuals who fail to follow these instructions. The statistical tests for significant differences are deliberately conservative (Clark et al., 2007) in order to avoid false claims. Of course, the flip-side of this is that it is possible Blizzard fails to identify interesting differences some of the time.

The listening tests typically used by the TTS research community lack ecological validity in many ways. They take place in an unusual setting – quiet, comfortable listening booths with high-quality sound reproduction and no distractions – and ask listeners to perform tasks they would never do in everyday life. For example, in order to test the intelligibility of systems, listeners are asked to transcribe – by typing on a computer keyboard – the individual words they heard. It is hard to think of a real application where this would be done. Worse, the sentences played to listeners are deliberately hard to comprehend, often being devoid of meaning (Benoit and Grice, 1996). This is done to remove the ceiling effect: in other words, many synthesisers could be close to 100% intelligible if predictable, meaningful sentences were used.

Does the lack of ecological validity matter though? In some respects it certainly is not a problem: if our synthesiser is as intelligible as natural speech when using difficult, meaningless sentences then we would be confident that it would be at least as intelligible using normal sentences. That is, the laboratory testing situation can uncover effects that would shrink into insignificance in the real world and the only danger is that we are identifying rather small differences. We still have confidence that we can identify the best system, although we may over-estimate how much better than the next system it actually is.

But in other respects the lack of ecological validity is much more serious. The idealised environment is the most serious issue: real end users do not operate in quiet environments free of distractions. The 2009 Challenge included a condition in which the synthetic speech was corrupted by a simulated telephone channel (King and Karaiskos, 2009) and the Hurricane Challenge mentioned in Section 4.1.2 addressed the problem of speech-in-noise much more rigourously. The tasks used are also a problem, since listeners are allowed to perform them under no significant constraints on their attention or time. There is doubtless still much to learn from experimental psychology, including the use of distractors to disguise to true purpose of the experiment, or methods which can introduce realistic levels of cognitive load into our subjects.

Despite these widely-recognised potential problems with how TTS is generally evaluated, there have been few attempts to innovate. Perhaps this is for the simple reason that any alternative would almost certainly yield far fewer data points per hour of testing time than current paradigms, and so be less practical and more costly. But perhaps it is just plain laziness: researchers prefer to spend their time inventing exciting new methods for synthesising speech, not worrying about whether they are actually measuring the quality of their work in the best way, especially when the burden of some of that evaluation can be offloaded to an external Challenge.

4.3.1. Whole system vs. component-level evaluations

As we mentioned in Section 2, Blizzard only attempts end-to-end system evaluations. Moreover, it also bundles in the data preparation stages such as alignment with the text and optional hand-corrections performed by some participants. In other words, it evaluates the totality of the systems components and the engineering skill and effort needed to make it work well on a new database. Conclusions about which method is “best” are therefore inevitably filtered through the level of expertise and available resources of the team implementing that method. This may be a partial explanation of the “failure” of some entries: the idea had merit, but the implementation was flawed. The availability of resources for checking and correcting the data varies widely between participants. To quantify the effect this has on overall quality, one year’s Challenge did release hand-checked alignments but this was found to be of limited use because it does not guarantee consistency across systems, since some may use a different phonetic inventory or pronunciation dictionary. Some participants have themselves investigated the benefits of manual annotations (Chu et al., 2006).

Providing linguistic specifications may appear to be one way to isolate the waveform generation component, but it would not be possible for some participants to modify their systems to use an externally-provided linguistic specification.

4.3.2. Common data, but what else?

The core of the Blizzard Challenge is the shared corpus which all participants are required to use. Its size has varied over the years, generally getting larger over time, and several years have seen specific sub-challenges involving restricted corpus sizes. As we have mentioned a number of times throughout this paper, a common corpus only ‘levels the playing field’ to some degree and there remain many other uncontrolled factors which may explain differences between systems. It is probably impossible to entirely separate out the effectiveness of a proposed technique from the skill of the engineer who implements it. Simple techniques, implemented by experts, can perform very well. Certainly, complex techniques poorly implemented are not likely to succeed. Within a single year of the Challenge then, it is hard to say for sure that one technique is better than another.

But, by looking over several years of Challenges, as we have done here, we can start to find independently-constructed systems being entered that use a common technique. When we see several of these performing well, then it becomes more reasonable to say that this is a good technique. Clear examples of this (if implemented skilfully) include unit selection, which almost guarantees a good naturalness score, HMM-based methods, which almost guarantee good intelligibility, and hybrid systems which maintain the high naturalness of unit selection and start to approach the intelligibility of HMM systems.

4.3.3. Too much at stake leads to too little risk

As the Challenge became more and more established, and a firm fixture in the calendar, awareness of it began to rise outside the immediate circle of participating researchers. A negative effect of this is that participation in the Challenge has become a more public affair: poorlyperforming entries no longer go un-noticed but instead start to attract attention. For the research labs in large corporations, this presents a major barrier to participation in the Challenge, since their management/lawyers/marketing department are likely to say “Of course you can enter the Blizzard Challenge, provided that you win.”

It is often said that one learns more from mistakes than successes, and Blizzard is no exception. The organisers of Blizzard are always at pains to point out that it is not a competition, and there are no winners and losers – that is, ‘mistakes’ are encouraged. It is to be hoped that all participants resist the temptation to play it safe with their entries, and that normally risk averse corporations see the benefits to taking part. They can easily mitigate the risks simply by describing their entry as a highly experimental research idea and not as a production system.