阿里云最新开源模型——FunAudioLLM

阿里云最新开源模型——FunAudioLLM 阿里云最新开源模型——FunAudioLLM Modified July 8, 2024 Figure 3. Comparasion of SenseVoice and Whisper on multilingual speech recognition beachmarks. Model Framework Parameters Support Language 3s Audio Latency 5s Audio Latency 10 Audio Latency Whisper Small Autoregressive 244 M 50+ 285ms 367ms 518ms Whisper Large V3 Autoregressive 1550 M 50+ 751ms 1009ms 1281ms Paraformer zh Non Autoregressive 220 M zh 76ms 85ms 100ms SenseVoice Small Non Autoregressive 234 M zh,yue,en,ja,ko 63ms 67ms 70ms SenseVoice Large Autoregressive 1587 M 50+ 738ms 1207ms 1623ms Tabel 1. Comparasion of model architecture, parameter scale, supported languages, and inference efficiency of SenseVoice, Paraformer, and Whisper. SenseVoice small employs a non autoregressive architecture, which offers a significant advantage in inference efficiency compared to Whisper. Speech w/o ITN w ITN 无法获取音频链接 开放时间早上九点至下午五点 开放时间早上9点至下午5点。 无法获取音频链接 呢几个字都表达唔到我想讲嘅意思 呢几个字都表达唔到，我想讲嘅意思。 无法获取音频链接 the tribal chieftain called for the boy and presented him with fifty pieces of gold The tribal chieftain called for the boy and presented him with 50 pieces of gold. 无法获取音频链接 うちの中学は弁当制で持っていけない場合は50円の学校販売のパンを買う うちの中学は弁当制で持っていけない 場合は、50 円の学校販売の パンを買う。 无法获取音频链接 조 금만 생각 을 하 면서 살 면 훨씬 편할 거야 조 금만 생각 을 하 면서 살 면 훨씬 편할 거야. Tabel 2. SenseVoice small can control whether to perform Inverse Text Normalization (ITN) during recognition via the tag prompt. Speech Emotion Recognition SenseVoice can also be used for discrete emotion recognition. Happy, Sad, Angry and Neutral are supported. We evaluate it on 7 popular emotion recognition dataset. The SenseVoice Large can approaching or exceeding the SOTA results on most datasets even without target corpus finetuning. Figure 4. Weighted Average Accuracy (WA(%)) comparison on 7 emotion recognition datasets. EmoBox is a recent speech emotion recognition benchmark based on Self Supervised Models and Whisper. Model on HF stands for the most popular speech emotion recognition model on HuggingFace. Audio SenseVoice Large SenseVoice Small 无法获取音频链接 <|zh| <|Speech| 英国的哲学家曾经说过。<|/Speech| <|HAPPY| <|zh| <|HAPPY| <|Speech| 英国的哲学家曾经说过。 无法获取音频链接 <|zh| <|Speech| 英国的哲学家曾经说过。<|/Speech| <|SAD| <|zh| <|SAD| <|Speech| 英国的哲学家曾经说过。 无法获取音频链接 <|zh| <|Speech| 英国的哲学家曾经说过。<|/Speech| <|ANGRY| <|zh| <|ANGRY| <|Speech| 英国的哲学家曾经说过。 无法获取音频链接 <|zh| <|Speech| 英国的哲学家曾经说过。<|/Speech| <|NEUTRAL| <|zh| <|NEUTRAL| <|Speech| 英国的哲学家曾经说过。 无法获取音频链接 <|en| <|Speech| I did go, and made many prisoners. <|/Speech| <|HAPPY| <|en| <|HAPPY| <|Speech| I did go and made many prisoners. 无法获取音频链接 <|en| <|Speech| I did go, and made many prisoners. <|/Speech| <|SAD| <|en| <|SAD| <|Speech| I did go and made many prisoners. 无法获取音频链接 <|en| <|Speech| I did go, and made many prisoners. <|/Speech| <|ANGRY| <|en| <|ANGRY| <|Speech| I did go and made many prisoners. 无法获取音频链接 <|en| I did go, and made many prisoners. <|/Speech| <|NEUTRAL| <|en| <|NEUTRAL| <|Speech| I did go and made many prisoners. Note SenseVoice will predict a <|Unknown Emo| token when the emotation is weak or unclear, you can ban this token to obtain a result with low confidence level. Audio Events Classification Both SenseVoice Small and SenseVoice Large model can detect the audio event in the speech, including music, applause, laughter. The SenseVoice Large can predict the start and end position of the audio event, while the SenseVoice Small can only predict what happned (only one event) in the audio, however, it can detect more events, such as coughing, sneezing, breathing and crying which could occur during human machine interaction. Audio SenseVoice Large SenseVoice Small 无法获取音频链接 <|en| <|BGM| <|Speech| Winning that ticket rose was the best thing that ever happened to me. It brought me to you, and I'm thankful for that rose. I'm thankful. <|/BGM| <|/Speech| <|en| <|BGM| Wining that ticket rose was the best thing that ever happened to me. It brought me to you. And I'm thankful for that rose. I'm thankful. 无法获取音频链接 <|en| <|Speech| Senior staff, Principal Doris Jackson, Wakefield faculty, and of course, my fellow classmates, <|Applause| I <|/Applause| am honored to have been chosen to speak before my classmates as well as students across America today.<|/Speech| <|en| <|Applause| Senior staff, principalipal Doris Jackson, Wake Food faculty, and, of course, my fellow classmates, I am honored to have been chosen to speak before my classmates as well as students across America today. 无法获取音频链接 <|zh| <|Speech| 啊，那你男朋友能够记得吗？还好吧，这个是你男朋友吗？他喜欢吃什么？<|Laughter| <|/Laughter| <|/Speech| <|zh| <|Laughter| 啊，那你男朋友能够记得吗？还好吧，这个是你男朋友吗？她喜欢吃什么？ 无法获取音频链接 <|zh| <|Applause| <|Speech| 我起飞<|/Applause| 前没有扫二维码，我都在看彭于晏，<|Laughter| 我连<|/Laughter| 安全演示片我都没有看。所以我当时脑。<|/Speech| <|zh| <|Applause| 我起飞前没有扫二维码，我都在看彭于晏，我连安全演示片我都没有看，结以我当时脑。 Although the SenseVoice are trained on speech data, it can still classify the recording with only audio event. We compare the SenseVoice with the audio event detection models BEATS and PANNs on environment sound classification, baby cry/laugh detection, coughing detection and talkshow event detection. The audio events classification ability can prevent the model predicting unexisting word in the environment noisy recordings. Audio Whisper SenseVoice Large SenseVoice Small 无法获取音频链接 <|en| <|transcribe| <|notimestamps| Thank you. <|nospeech| <|Applause| <|/Applause| <|nospeech| <|Unknown Emo| <|Applause| 无法获取音频链接 <|en| <|transcribe| <|notimestamps| laughing <|nospeech| <|Laughter| <|/Laughter| <|nospeech| <|Unknown Emo| <|Laughter| 无法获取音频链接 <|en| <|transcribe| <|notimestamps| Pfft. <|nospeech| <|Unknown Event| <|nospeech| <|Unknown Emo| <|Cough| 无法获取音频链接 <|en| <|transcribe| <|notimestamps| <|en| <|transcribe| <|notimestamps| I love you. <|nospeech| <|Unknown Event| <|nospeech| <|Unknown Emo| <|Cry| 无法获取音频链接 <|ja| <|transcribe| <|notimestamps| ふーふーふー <|nospeech| <|Unknown Event| <|nospeech| <|Unknown Emo| <|Breath| 无法获取音频链接 <|en| <|transcribe| <|notimestamps| I can't! <|nospeech| <|Unknown Event| <|en| <|Unknown Emo| <|Sneeze| 无法获取音频链接 <|en| <|transcribe| <|notimestamps| so <|nospeech| <|Applause| <|/Applause| <|nospeech| <|Unknown Emo| <|Unknown Event| Rich Transcribe Demo Samples Note: Audio examples are segmented into 15 second chunks without using Voice Activity Detection (VAD) and then processed by the ASR model. The ASR results are concatenated, and special tokens are removed. No language prompt is added during decoding. For emotional context, 😊 denotes HAPPY, 😡 denotes ANGRY, and 😔 denotes SAD. For audioevent, 🎼 denotes music, 😀 denotes laugh and 👏 denotes applause. Additional, for Sensevoice Large, 🎵 and 🎶 stands for the begin and the end of the background music, and red font stands for lyric (text without <|Speech| ). Audio Whisper Lagre V3 results SenseVoice Large results SenseVoice Small results 无法获取音频链接 Tangri with his song, Heaven. Heaven. so Yeah! Absolute shock, but in a great way. Wow. That was awesome. That was awesome. What way to open a song. That was awesome. Awesome. I'd love to check out some more Mongolian throat singing stuff. That is correct, right? It is Mongolian. Let me know, I'd love to check out more. I think a lot of you want me to check out The Who. If you guys still want me to, I'd be more than happy to. so晚安的天空清的湖水啊滴滴的草原 That is incredible. That is incredible. For those of you who don't know what I'm saying right now, the way he can make it sound like he's finished a note, you know, he like lowers it so low you can't even hear the note anymore. And then he brings it back and you can see his mouth still open. The way he can like finish a note but not finish it i don't know how to explain that that is an incredible talent that is amazing Hey Tangree with his song, Heaven. 🎼 啊。 🎵 Absolutely shocked but in a great way. That was awesome, 🎶 that was awesome 😊 what way to open a song, that was awesome, awesome, I'd love to check out some more Mongolian throat singing stuff, that is correct right, it is Mongolian. Let me know I'd love to check out more I think a lot of you want to check out the Who if you guys still want me to I'd be more than happy to. 🎼 蓝蓝的天空，清清的湖水啊，绿绿的草原 。That is incredible that is incredible for those of you who don't know what I'm saying right now the way he can make it sound like he's finished a note you know he like lowers it so low you can't even hear the note anymore and then he brings it back and you can see his mouth still open the way it makes the way he can like finish a note but not finish it . 😡 I don't know how to explain that that is an incredible talent that is amazing. 😊 🎼 这是我的家， 哎。 😊👏 Tangry with his song, heaven. 🎼 Absolute shocked but in a great way my. 😊That was awesome, that was awesome what way to open a song that was awesome, awesome, 😡 I'd love to check out some more Moolian fruiting and stuff that is correct right it is Monolian let me know I'd love to check out more I think a lot of you want me to check out the who if you guys still want me to I'd be more than happy to. 🎼 蓝蓝的天空。清清的湖水呀，绿绿的草原。😔 That is incredible, that is incredible that is incredible for those of you who don't know what I'm saying right now, the way he can make it sound like he's finished a note, you know, he like lowers it's so low you you can't even. Hear the note anymore and then he brings it back and you can see his mouth still open the way it makes the way he can like finish a note but not finish it, I don't know how to explain that that is an incredible talent that is amazing. 😡 🎼 这是我的家哎。 EmoBox Model on HF BEATS PANNs Figure 3. Comparasion of SenseVoice and Whisper on multilingual speech recognition beachmarks. Model Framework Parameters Support Language 3s Audio Latency 5s Audio Latency 10 Audio Latency Whisper Small Autoregressive 244 M 50+ 285ms 367ms 518ms Whisper Large V3 Autoregressive 1550 M 50+ 751ms 1009ms 1281ms Paraformer zh Non Autoregressive 220 M zh 76ms 85ms 100ms SenseVoice Small Non Autoregressive 234 M zh,yue,en,ja,ko 63ms 67ms 70ms SenseVoice Large Autoregressive 1587 M 50+ 738ms 1207ms 1623ms Model Model Framework Framework Parameters Parameters Support Language Support Language 3s Audio Latency 3s Audio Latency 5s Audio Latency 5s Audio Latency 10 Audio Latency 10 Audio Latency Whisper Small Whisper Small Autoregressive Autoregressive 244 M 244 M 50+ 50+ 285ms 285ms 367ms 367ms 518ms 518ms Whisper Large V3 Whisper Large V3 Autoregressive Autoregressive 1550 M 1550 M 50+ 50+ 751ms 751ms 1009ms 1009ms 1281ms 1281ms Paraformer zh Paraformer zh Non Autoregressive Non Autoregressive 220 M 220 M zh zh 76ms 76ms 85ms 85ms 100ms 100ms SenseVoice Small SenseVoice Small Non Autoregressive Non Autoregressive 234 M 234 M zh,yue,en,ja,ko zh,yue,en,ja,ko 63ms 63ms 67ms 67ms 70ms 70ms SenseVoice Large SenseVoice Large Autoregressive Autoregressive 1587 M 1587 M 50+ 50+ 738ms 738ms 1207ms 1207ms 1623ms 1623ms Tabel 1. Comparasion of model architecture, parameter scale, supported languages, and inference efficiency of SenseVoice, Paraformer, and Whisper. SenseVoice small employs a non autoregressive architecture, which offers a significant advantage in inference efficiency compared to Whisper. Speech w/o ITN w ITN 无法获取音频链接 开放时间早上九点至下午五点 开放时间早上9点至下午5点。 无法获取音频链接 呢几个字都表达唔到我想讲嘅意思 呢几个字都表达唔到，我想讲嘅意思。 无法获取音频链接 the tribal chieftain called for the boy and presented him with fifty pieces of gold The tribal chieftain called for the boy and presented him with 50 pieces of gold. 无法获取音频链接 うちの中学は弁当制で持っていけない場合は50円の学校販売のパンを買う うちの中学は弁当制で持っていけない 場合は、50 円の学校販売の パンを買う。 无法获取音频链接 조 금만 생각 을 하 면서 살 면 훨씬 편할 거야 조 금만 생각 을 하 면서 살 면 훨씬 편할 거야. Speech Speech w/o ITN w/o ITN w ITN w ITN 无法获取音频链接 无法获取音频链接 开放时间早上九点至下午五点 开放时间早上九点至下午五点 开放时间早上9点至下午5点。 开放时间早上9点至下午5点。 无法获取音频链接 无法获取音频链接 呢几个字都表达唔到我想讲嘅意思 呢几个字都表达唔到我想讲嘅意思 呢几个字都表达唔到，我想讲嘅意思。 呢几个字都表达唔到，我想讲嘅意思。 无法获取音频链接 无法获取音频链接 the tribal chieftain called for the boy and presented him with fifty pieces of gold the tribal chieftain called for the boy and presented him with fifty pieces of gold The tribal chieftain