![]() We use a general pretraining and fine-tuning strategy for Personal Voice. This results in faster training and inference, as well as reduced memory consumption, making the models shippable on iPhone and iPad. ![]() Further, our team uses dilated convolution layers for decoding instead of transformer-based layers. ![]() However, we add speaker ID as part of the decoder input to learn general voice information during the pretraining stage. The acoustic model follows an architecture similar to FastSpeech2. Modified FastSpeech2-based acoustic model.The next machine learning approaches we will discuss are voice model pretraining and fine-tuning. In addition, fine-tuning can reduce the actual model size enough to achieve real-time speech synthesis for a faster and more satisfying conversation experience. The MOS is 0.43 higher than the universal vocoder version on average. Listening tests showed that fine-tuning both models achieves the best voice quality and similarity to the target speaker’s voice, as measured by mean opinion score (MOS) and voice similarity (VS) score, respectively. The system takes phoneme as the input, then FastSpeech2 model converts phoneme to target speaker Mel spectrum, after that WaveRNN converts Mel spectrum to speech waveform which is the output of the system. Fine-tuning both models, as seen in Figure 2, requires extra training time on device but results in better overall quality.įigure 2: A personal voice text-to-speech system diagram. Unusual prosody, audio glitches, and noise were more prevalent, when tested against unseen speakers. Our team found that fine-tuning only the acoustic model, and using a universal vocoder, often generates poorer voice quality. For the vocoder model, we considered both a universal model and on-device adaptation. To clone the target speaker’s voice, we fine-tuned the acoustic model with on-device training. Both the acoustic model and vocoder model are speaker-dependent in a typical TTS system. Personal Voice must produce speech output that others can recognize as the voice of the target speaker. The cleaned dataset includes 300 hours of 1000 speakers with very different speaking styles or accents. To develop Personal Voice, Apple researchers worked on the Open SLR LibriTTS dataset. Vocoder model: Converts acoustic features to speech waveforms, providing a representation of the audio signal over time.Acoustic model: Converts phonemes to acoustic features (for example, to the Mel spectrum, a frequency representation of sound, engineered to represent the range of human speech).Text processing: Converts graphemes (written text) to phonemes, a written notation that represents a distinct units of sound (such as the h of hat and the c of cat in English).A TTS system includes three major components: The first machine learning approach we will discuss is a typical neural TTS system, which takes in text and provides speech output. Voice model pretraining and fine-tuning.In this research highlight, we discuss the three machine learning approaches behind Personal Voice: Because model training and inference are done entirely on-device, users can take advantage of Personal Voice whenever they want, and keep their information both private and secure. By the next day, the person can type what they want to say using the Live Speech text-to-speech (TTS) feature, as illustrated in Figure 1, and be heard in conversation in a voice that sounds like theirs. This is only for downloading the pre-trained asset. To start, the user reads aloud a randomized set of text prompts to record 150 sentences on the latest iPhone, iPad or Mac.The voice audio is then tuned with machine learning techniques overnight directly on the device while the device is charging, locked and connected to Wi-Fi. First introduced in May 2023 and made available on iOS 17 in September 2023, Personal Voice is a tool that creates a synthesized voice for such users to speak in FaceTime, phone calls, assistive communication apps, and in-person conversations. A voice replicator is a powerful tool for people at risk of losing their ability to speak, including those with a recent diagnosis of amyotrophic lateral sclerosis (ALS) or other conditions that can progressively impact speaking ability.
0 Comments
Leave a Reply. |
Details
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |