Microsoft VALL-E clone anyone’s voice from a 3 second sample
Microsoft researchers have announced a new app that uses artificial intelligence to mimic a person’s voice with just seconds of training. The voice model can then be used for text-to-speech applications.
The app called VALL-E can be used to synthesize high-quality personalized speech with just a three-second registration recording of a speaker as an acoustic prompt, the researchers wrote in a paper published online at arXiv, a free distribution service and an open source. – access archives for scientific articles.
There are now programs that can cut and paste speech into an audio stream, and that speech is converted into a speaker’s voice from the typed text. However, the program must be trained to mimic a person’s voice, which can take an hour or more.
“One of the most remarkable things about this model is that it does it in seconds. It’s very impressive,” said Ross Rubin, principal analyst at Reticle Research, a consumer technology consultancy in New York. York at TechNewsWorld.
According to the researchers, VALL-E significantly outperforms existing state-of-the-art text-to-speech systems (TTS) in both speech naturalness and speaker similarity.
Additionally, VALL-E can preserve a speaker’s emotions and acoustic environment. So if a speech sample was recorded on a phone, for example, text using that voice would sound like it was being read through a phone.
“Super Impressive”
VALL-E is a notable improvement over previous state-of-the-art systems, such as YourTTS, released in early 2022, said Giacomo Miceli, computer scientist and creator of a website with endless AI-generated discussion featuring the synthetic speech by Werner Herzog and Slavoj Žižek.
“What’s interesting about VALL-E is not only the fact that it only needs three seconds of audio to clone a voice, but also how well it can match that voice, at emotional timbre and any background noise,” Miceli told TechNewsWorld. Ritu Jyoti, group vice president for AI and automation at IDC, a global market research firm, called VALL-E “meaningful and super impressive.”
“This is a significant improvement over previous models, which required a much longer training period to generate a new voice,” Jyoti told TechNewsWorld.
“This technology is still in its infancy, and further improvements should make it sound more human,” she added.
Emotional emulation questioned
Unlike OpenAI, the creator of ChatGPT, Microsoft hasn’t opened VALL-E to the public, so questions remain about its performance. For example, are there factors that could lead to speech degradation produced by the application?
“The longer the generated audio snippet, the higher the chance that a human will hear things that sound a little off,” Miceli observed. “Words may be unclear, missed, or duplicated in text-to-speech.”
“It’s also possible that the shift from one emotional register to another doesn’t feel natural,” he added.
The app’s ability to emulate a speaker’s emotions also has skeptics. “It will be interesting to see how robust this capability is,” said Mark N. Vena, president and principal analyst at SmartTech Research in San Jose, Calif.
“The fact that they claim it can do this with just a few seconds of audio is hard to believe,” he continued, “given the current limitations of AI algorithms, which require samples of much longer voices”.
Ethical Concerns
Experts see beneficial applications for VALL-E, as well as less beneficial ones. Jyoti cited speech editing and replacing voice actors. Miceli noted that the technology could be used to create editing tools for podcasters, personalize the voice of smart speakers, as well as be integrated into messaging systems and chat rooms, video games and even navigation systems.
“The flip side is that a malicious user could clone the voice of a politician, for example, and make them say things that sound nonsensical or inflammatory, or in general to spread false information or propaganda,” Miceli added.
Vena sees huge potential for abuse in the technology if it is as good as Microsoft claims. “At the level of financial services and security, it’s not hard to bring up use cases by nefarious actors who could do some really damaging things,” he said.
ADVERTISING
Jyoti, too, sees ethical concerns bubbling around VALL-E. “As technology advances, the voices generated by VALL-E and similar technologies will become more compelling,” she explained. “It would open the door to realistic spamming that reproduces the voices of real people that a potential victim knows.”
“Politicians and other public figures could also be impersonated,” she added.
“There could be potential security issues,” she continued. “For example, some banks allow voice passwords, which raises concerns about misuse. We might expect an escalation in the arms race between AI-generated content and AI detection software to stop abuse.
“It is important to note that VALL-E is not currently available,” added Jyoti. “Overall, regulation of AI is essential. We will have to see what measures Microsoft puts in place to regulate the use of VALL-E.
Enter the lawyers
Legal issues can also arise around the technology. “Unfortunately, there may not be enough current legal tools in place to directly address these issues, and instead a hodgepodge of laws that cover how the technology is misused can be used to curb that abuse,” said Michael L. Teich, a principal at Harness IP, a national intellectual property law firm.
“For example,” he continued, “voice cloning can lead to a deepfake of a real person’s voice that can be used to trick a listener into succumbing to a scam or can even be used to impersonate the voice of an election candidate. While such abuse would likely raise legal issues in the areas of fraud, defamation, or election disinformation laws, there is a lack of specific AI laws that would tackle the use of the technology itself.
ADVERTISING
“In addition, depending on how the initial voice sample was obtained, there may be implications under federal wiretapping law and state wiretapping laws if the voice sample was obtained via, for example, a telephone line,” he added.
“Finally,” Teich noted, “in limited circumstances, there may be First Amendment concerns if such voice cloning were to be used by a government actor to silence, delegitimize, or dilute legitimate voices in the exercise of their rights to freedom of expression”.
“As these technologies mature, it may be necessary to enact specific laws to directly address the technology and prevent its misuse as the technology advances and becomes more accessible,” he said. declared.
Make smart investments
In recent weeks, Microsoft has been making AI headlines. It is expected to integrate ChatGPT technology into its Bing search engine this year and eventually into its Office applications. He also reportedly plans to invest $10 million in OpenAI — and now, VALL-E.
“I think they’re making a lot of smart investments,” said Bob O’Donnell, founder and chief analyst of Technalysis Research, a market research and technology consulting firm in Foster City, Calif.
“They jumped on the OpenAI bandwagon several years ago, so they’ve been behind the scenes for quite a while. Now it’s coming out big,” O’Donnell told TechNewsWorld.
“They had to catch up with Google, which is known for its artificial intelligence, but Microsoft is taking aggressive steps to get to the top,” he continued. “They’re jumping on the popularity and the incredible coverage that all of this stuff has gotten.”
Rubin added, “Microsoft, having been the productivity leader for the past 30 or so years, wants to maintain and expand that lead. AI could hold the key.
Tech
Leave a Reply