AllTalk is a web interface, based around the Coqui TTS voice + cloning/speech generation system. To generate TTS, you can use the provided gradio interface or interact with + the server using JSON/CURL commands.
+Note: When loading up a new + character in Text generation webUI it may look like nothing is happening for 20-30 seconds. Its actually + processing the introduction section (greeting message) of the text and once that is completed, it will appear. + You can see the activity occuring in the console window. Refreshing the page multiple times will try force the + TTS engine to keep re-generating the text, so please just wait and check the console if needed.
+Note: Ensure that your RP + character card has asterisks around anything for the narration and double quotes around anything spoken. There + is a complication ONLY with the greeting card so, ensuring it has the correct use of quotes and asterisks will + help make sure that the greeting card sounds correct. I will aim to address this issue in a future update. In + Text-generation-webUI parameters menu > character tab > greeting make + sure that anything in there that is the narrator is in asterisks and anything spoken is in double quotes, then + hit the save (💾) button.
+AllTalk Minor Bug Fixes Changelog & Known + issues
+If I squash any minor bugs or find any issues, I will try to apply an update asap. If + you think something isnt working correctly or you have a problem, check these two links below first.
+ +AllTalk
+
Github link
+ here
Update instructions link here
Help and issues link
+ here
Text generation webUI
+
Web interface link
+ here
Documentation link here
http://{{ params.ip_address }}:{{ params.port_number }}http://{{ params.ip_address }}:{{ params.port_number }}/ready
+ Voice samples are stored in /alltalk_tts/voices/ + and should be named using the following format name.wav
+ +Voice outputs are stored in /alltalk_tts/outputs/
+You can configure automatic maintenence deletion of old wav + files by setting Del WAV's older than in the settings above.
+When Disabled your output + wav files will be left untouched. When set to a setting 1 Day or greater, + your output wav files older than that time period will be automatically deleted on start-up of AllTalk.
+ +This extension will download the 2.0.2 model to /alltalk_tts/models/
+This TTS engine will also download the latest available model + and store it wherever your OS normally stores it (Windows/Linux/Mac).
+ +To create a new voice sample you need to make a wav file that is + 22050Hz, Mono, 16 bit and between 6 to 30 seconds long, though 8 to 10 seconds is usually + good enough. The model can handle up to a 30 second samples, however Ive not noticed any improvement in voice + output from a much longer clips.
+You want to find a nice clear selection of audio, so lets say + you wanted to clone your favourite celebrity. You may go looking for an interview where they are talking. Pay + close attention to the audio you are listening to and trying to sample. Are there noises in the backgroud, hiss + on the soundtrack, a low humm, some quiet music playing or something? The better quality the audio the better + the final TTS result. Dont forget, the AI that processes the sounds can hear everything in your sample and it + will use them in the voice its trying to recreate.
+Try make your clip one of nice flowing speech, like the included + example files. No big pauses, gaps or other sounds. Preferably a sample that the person you are trying to copy + will show a little vocal range and emotion in their voice. Also, try to avoid a clip starting or ending with + breathy sounds (breathing in/out etc).
+ +So youve downloaded your favoutie celebrity interview off + Youtube, from here you need to chop it down to 6 to 30 seconds in length and resample it.
+If you need to clean it up, do audio processing, volume level
+ changes etc, do this before downsampling.
Using the latest version of Audacity select/highlight your 6 to 30 second clip and:
Tracks
+ > Resample to 22050Hz then
Tracks >
+ Mix > Stereo to
+ Mono then
+ File > Export Audio saving it as a WAV of 22050Hz.
Save your generated wav file in the /alltalk_tts/voices/ folder.
+Its worth mentioning that using AI generated audio clips may + introduce unwanted sounds as its already a copy/simulation of a voice.
+ +Maybe you might be interested in trying Finetuning of the model. Otherwise, the reasons can be that you:
+Some samples just never seem to work correctly, so maybe try a + different sample. Always remember though, this is an AI model attempting to re-create a voice, so you will never + get a 100% match.
+ + +This only affects the Narrator function. Most AI models should
+ be using asterisks or double quotes to differentiate between the Narrator or the Character, however, many models
+ sometimes switch between using asterisks and double quotes or sometimes nothing at all for the text it outputs.
+
This leaves a bit of a mess because sometimes un-marked text is narration and sometimes its the
+ character talking, leaving no clear way to know where to split sentences and what voice to use. Whilst there is
+ no 100% solution at the moment many models lean more one way or the other as to what that unmarked text will be
+ (character or narrator).
As such, the "Text not inside" function at least gives you the + choice to set how you want the TTS engine to handle situations of un-marked text.
+The Low VRAM option is a crucial feature designed to enhance + performance under constrained (VRAM) conditions, as the TTS models require 2GB-3GB of VRAM to run effectively. + This feature strategically manages the relocation of the Text-to-Speech (TTS) model between your system's Random + Access Memory (RAM) and VRAM, moving it between the two on the fly. Obviously, this is very useful for people + who have smaller graphics cards and will use all their VRAM to load in their LLM.
+When you dont have enough VRAM free after loading your LLM model + into your VRAM (Normal Mode example below), you can see that with so little working space, your GPU will have to + swap in and out bits of the TTS model, which causes horrible slowdown.
+Note: An Nvidia Graphics + card is required for the LowVRAM option to work, as you will just be using system RAM otherwise.
+ +The Low VRAM mode intelligently orchestrates the relocation of + the entire TTS model and stores the TTS model in your system RAM. When the TTS engine requires VRAM for + processing, the entire model seamlessly moves into VRAM, causing your LLM to unload/displace some layers, + ensuring optimal performance of the TTS engine.
+Post-TTS processing, the model moves back to system RAM, freeing + up VRAM space for your Language Model (LLM) to load back in the missing layers. This adds about 1-2 seconds to + both text generation by the LLM and the TTS engine.
+By transferring the entire model between RAM and VRAM, the Low + VRAM option avoids fragmentation, ensuring the TTS model remains cohesive and has all the working space it needs + in your GPU, without having to just work on small bits of the TTS model at a time (which causes terrible slow + down).
+This creates a TTS generation performance Boost for Low VRAM + Users and is particularly beneficial for users with less than 2GB of free VRAM after loading their LLM, + delivering a substantial 5-10x improvement in TTS generation speed.
+
If you have a voice that the model doesnt quite reproduce correctly, + or indeed you just want to improve the reproduced voice, then finetuning is a way to train your "XTTSv2 + local" model (stored in /alltalk_tts/models/xxxxx/) on a specific voice. + For this you will need:
+Everything has been done to make this as simple as possible. At its + simplest, you can literally just download a large chunk of audio from an interview, and tell the finetuning to + strip through it, find spoken parts and build your dataset. You can literally click 4 buttons, then copy a few + files and you are done. At it's more complicated end you will clean up the audio a little beforehand, but + its still only 4x buttons and copying a few files.
+ +I would suggest that if its in an interview format, you cut out the + interviewer speaking in audacity or your chosen audio editing package. You dont have to worry about being + perfect with your cuts, the finetuning Step 1 will go and find spoken audio and cut it out for you. Is there is + music over the spoken parts, for best quality you would cut out those parts, though its not 100% necessary. As + always, try to avoid bad quality audio with noises in it (humming sounds, hiss etc). You can try something like + Audioenhancer to try clean up noisier audio. There is no need to down-sample any of the audio, all of that is + handled for you. Just give the finetuning some good quality audio to work with.
+ +As mentioned you must have a small portion of the Nvidia CUDA Toolkit 11.8 installed. Not higher or + lower versions. Specifically 11.8. You do not have to uninstall any other versions, change + any graphics drivers, reinstall torch or anything like that. There are instructions within the finetuning + interface on doing this or you can also find them on this link here
+ +Ensure you have followed the instructions on setting up the Nvidia CUDA + Toolkit 11.8 here or + the below procedure will fail.
+The below instructions are also available online here
+ +Close all other applications that are using your GPU/VRAM and copy your audio samples into:
+
+ /alltalk_tts/finetune/put-voice-samples-in-here/
+
In a command prompt/terminal window you need to move into your Text generation webUI folder:
+
+ cd text-generation-webui
+
Start the Text generation webUI Python environment for your OS:
+
+ cmd_windows.bat, ./cmd_linux.sh, cmd_macos.sh or cmd_wsl.bat
+
You can double check your search path environment still works correctly with nvcc --version. It should report back 11.8:
+
+ Cuda compilation tools, release 11.8.
+
Move into your extensions folder:
+
+ cd extensions
+
Move into the alltalk_tts folder:
+
+ cd alltalk_tts
+
Install the finetune requirements file: pip install -r + requirements_finetune.txt
+Type python finetune.py and it should start up.
+Follow the on-screen instructions when the web interface starts up.
+When you have finished finetuning, the final tab will tell you what to do with your files and how to move + your newly trained model to the correct location on disk.
+DeepSpeed provides a 2x-3x speed boost for Text-to-Speech and AI + tasks. It's all about making AI and TTS happen faster and more efficiently.
+
DeepSpeed only works with the XTTSv2 Local model and will deactivate when other models are selected, even + if the checkbox still shows as being selected.
+Note: DeepSpeed/AllTalk may + warn if the Nvidia Cuda Toolkit and CUDA_HOME environment variable isnt set correctly. On Linux you need + CUDA_HOME configured correctly; on Windows, if you use the pre-built wheel its ok without.
+Note: You do not need to set + Text-generation-webUi's --deepspeed setting for AllTalk to be able to use DeepSpeed.
+ + +➡️DeepSpeed requires an Nvidia Graphics card!⬅️
+➡️DeepSpeed requires an Nvidia Graphics card!⬅️
+DeepSpeed v11.2 will work on the current default text-generation-webui Python 3.11 + environment! You have 2x options for how to setup DeepSpeed on Windows. A quick way Option 1 and a long way Option 2.
+Thanks to @S95Sedan They managed to get
+ DeepSpeed 11.2 working on Windows via making some edits to the original Microsoft DeepSpeed v11.2 installation.
+ The original post is here.
+
OPTION 1 - Pre-Compiled Wheel Deepspeed v11.2 (Python + 3.11 and 3.10)
+It's worth noting that all models and methods can and do sound + different from one another. Many people complained about the quality of audio produced by the 2.0.3 model, so + this extension will download the 2.0.2 model to your models folder and give you the choice to use 2.0.2 (API Local and XTTSv2 Local) or use the + most current model 2.0.3 (API TTS). As/When a new model is released by + Coqui it will be downloaded by the TTS service on startup and stored wherever the TTS service keeps new models + for your operating system.
+It is recommended not to modify these settings unless you + fully comprehend their effects. A general overview is provided below for reference.
+Changes to these settings won't take effect until you + restart AllTalk/Text generation webUI.
+These settings only affect API Local and XTTSv2 Local + methods.
+ +In the context of text-to-speech (TTS), the Repetition + Penalty influences how the model handles the repetition of sounds, phonemes, or + intonation patterns. Here's how it works:
+The factory setting for repetition penalty is 10.0
+ +Temperature influences the randomness of the generated + speech. Here's how it affects the output:
+The factory setting for temperature is 0.70
+ +Factory settings should be fine for most people, however if you choose to + experiment, setting extremely high or low values, especially without a good understanding of their effects, + may lead to flat-sounding output or very strange-sounding output. It's advisable to experiment with + adjustments incrementally and observe the impact on the generated speech to find a balance that suits your + desired outcome.
+ + +AllTalk performs a variety of checks on startup and will + warn out messages at the console should you need to do something such as update your TTS version.
+A basic environment check to ensure everything should work + e.g. is the model already downloaded, are the configuration files set correctly etc.
+AllTalk will download the Xtts model (version 2.0.2) into + your models folder. Many people didnt like the quality of the 2.0.3 model, however the latest model will be + accessable on the API TTS setting (2.0.3 at the time of writing) so you have the best of both worlds.
+ + +Its possible to set a custom model for the API Local and XTTSv2 Local methods, + or indeed point it at the same model that API TTS uses (wherever it is + stored on your OS of choice).
+Many people did not like the sound quality of the Coqui + 2.0.3 model, and as such the AllTalk downloads the 2.0.2 model seperately from the 2.0.3 model that TTS service downloads and manages.
+Typically the 2.0.2 + model is stored in your /alltalk_tts/models folder and it is downloaded + on first start-up of the AllTalk_tts extension. However, you may either want to use a custom model + version of your choosing, or re-point AllTalk to a different path on your system, or even point it so + that API Local and XTTSv2 + Local both use the same model that API TTS is using.
+If you do choose to change the location there are a couple + of things to note.
+To change the model path, there are at minimum 2x settings + you need to alter in the modeldownload.json file, base_path and model_path.
+You would edit the settings in the modeldownload.json file as follows (make a backup of your current file in
+ case):
Windows path example: c:\\mystuff\\mydownloads\\myTTSmodel\\{files in here}
+ base_path would be "c:\\mystuff\\mydownloads"
model_path would be: "myTTSmodel"
Note: On Windows systems, you have to + specify a double backslash + \\ for each folder level in the path (as above)
+ Linux path example: /home/myaccount/myTTSmodel/{files in
+ here}
base_path
+ would be "/home/myaccount"
+ model_path would be: "TTSmodel"
Once you restart Alltalk, it will check this
+ path for the files and output any details at the console.
When you are happy it's' working correctly, you are welcome to go delete the + models folder stored at /alltalk_tts/models.
+If you wish to change the files that the modeldownloader is pulling at startup, + you can futher edit the modeldownload.json and change the https + addresses within this files files_to_download section e.g. +
+"files_to_download": {
"LICENSE.txt": "https://huggingface.co/coqui/XTTS-v2/resolve/v2.0.2/LICENSE.txt?download=true",
+ "README.md": "https://huggingface.co/coqui/XTTS-v2/resolve/v2.0.2/README.md?download=true",
+ "config.json": "https://huggingface.co/coqui/XTTS-v2/resolve/v2.0.2/config.json?download=true",
+ "model.pth": "https://huggingface.co/coqui/XTTS-v2/resolve/v2.0.2/model.pth?download=true",
+ "vocab.json": "https://huggingface.co/coqui/XTTS-v2/resolve/v2.0.2/vocab.json?download=true"
}
+
confignew.json file:
+| Key | +Default Value | +Explanation | +
|---|---|---|
| "activate" | +true | +Sets activation state within Text-generation-webUI. | +
| "autoplay" | +true | +Sets autoplay within Text generation webUI. | +
| "branding" | +"AllTalk" | +Used to change the default name. Requires a space e.g. "Mybrand ". | +
| "deepspeed_activate" | +false | +Sets DeepSpeed activation on startup. | +
| "delete_output_wavs" | +"Disabled" | +Sets duration of outputs to delete. | +
| "ip_address" | +"127.0.0.1" | +Sets default IP address. | +
| "language" | +"English" | +Sets default language for Text-generation-webUI TTS. | +
| "low_vram" | +false | +Sets default setting for LowVRAM mode. | +
| "local_temperature" | +"0.70" | +Sets default model temp for API Local and XTTSv2 Local. | +
| "local_repetition_penalty" | +"10.0" | +Sets default model repetition for API Local and XTTSv2 Local. | +
| "tts_model_loaded" | +true | +AllTalk internal use only. Do not change. | +
| "tts_model_name" | +"tts_models/multilingual/multi-dataset/xtts_v2" | +Sets default model that API TTS is looking for through the TTS service (separate to API Local and + XTTSv2 Local). | +
| "narrator_enabled" | +true | +Sets default narrator on/off in Text-generation-webUI TTS. | +
| "narrator_voice" | +"female_02.wav" | +Sets default wav to use for narrator in Text-generation-webUI TTS. | +
| "port_number" | +"7851" | +Sets default port number for AllTalk. | +
| "output_folder_wav" | +"extensions/alltalk_tts/outputs/" | +Sets default output path Text-generation-webUI should use for finding outputs. | +
| "output_folder_wav_standalone" | +"outputs/" | +Sets default output path in standalone mode. | +
| "remove_trailing_dots" | +false | +Sets trailing dot removal pre-generating TTS. | +
| "show_text" | +true | +Sets if text should be displayed below audio in Text-generation-webUI. | +
| "tts_method_api_local" | +false | +Sets API Local as the default model/method for TTS. | +
| "tts_method_api_tts" | +false | +Sets API TTS as the default model/method for TTS. | +
| "tts_method_xtts_local" | +true | +Sets XTTSv2 Local as the default model/method for TTS. | +
| "voice" | +"female_01.wav" | +Sets default voice for TTS. | +
modeldownload.json file:
+| Key | +Value | +Description | +||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| "base_path" | +"models" | +Sets local model base path for API Local and XTTSv2 Local. | +||||||||||||
| "model_path" | +"xttsv2_2.0.2" | +Sets local model folder for API Local and XTTSv2 Local below the base path. | +||||||||||||
| "files_to_download" | +
+
|
+ Sets the model files required to be downloaded into \base_path\model_path\ and where + to download them from. | +
The Text-to-Speech (TTS) Generation API allows you to + generate speech from text input using various configuration options. This API supports both character and + narrator voices, providing flexibility for creating dynamic and engaging audio content.
+ +Standard TTS speech Example (standard text) generating a time-stamped file
+curl -X POST + "http://127.0.0.1:7851/api/tts-generate" -d "text_input=All of this + is text spoken by the character. This is text not inside quotes, though that doesnt matter in the + slightest" -d "text_filtering=standard" -d + "character_voice_gen=female_01.wav" -d "narrator_enabled=false" -d "narrator_voice_gen=male_01.wav" -d "text_not_inside=character" -d + "language=en" -d "output_file_name=myoutputfile" -d "output_file_timestamp=true" -d "autoplay=true" -d + "autoplay_volume=0.8"
+Narrator Example (standard text) generating a time-stamped file
+curl -X POST + "http://127.0.0.1:7851/api/tts-generate" -d "text_input=*This is + text spoken by the narrator* \"This is text spoken by the character\". This is text not inside + quotes." -d "text_filtering=standard" -d + "character_voice_gen=female_01.wav" -d "narrator_enabled=true" -d "narrator_voice_gen=male_01.wav" -d + "text_not_inside=character" -d + "language=en" -d "output_file_name=myoutputfile" -d "output_file_timestamp=true" -d "autoplay=true" -d + "autoplay_volume=0.8"
+Note + that if your text that needs to be generated contains double quotes you will need to escape them + with \" (Please see the narrator example).
+ +text_input: The text you want the TTS engine to + produce. Use escaped double quotes for character speech and asterisks for narrator speech if using + the narrator function. Example:
+-d "text_input=*This is text spoken by the narrator* \"This is + text spoken by the character\". This is text not inside quotes."
+text_filtering: Filter for text. Options:
+-d "text_filtering=none"
-d
+ "text_filtering=standard"
-d
+ "text_filtering=html"
Example:
+character_voice_gen: The WAV file name for the character's voice.
+-d "character_voice_gen=female_01.wav"
+narrator_enabled: Enable or disable the narrator function. If true, minimum text + filtering is set to standard. Anything between double quotes is considered the character's speech, + and anything between asterisks is considered the narrator's speech.
+-d "narrator_enabled=true"
-d "narrator_enabled=false"
+
narrator_voice_gen: The WAV file name for the narrator's voice.
+-d "narrator_voice_gen=male_01.wav"
+text_not_inside: Specify the handling of lines not inside double quotes or + asterisks, for the narrator feature. Options:
+-d "text_not_inside=character"
-d "text_not_inside=narrator"
language: Choose the language for TTS. Options:
+-d "language=en"
+output_file_name: The name of the output file (excluding the .wav extension).
+-d "output_file_name=myoutputfile"
+output_file_timestamp: Add a timestamp to the output file name. If true, each file + will have a unique timestamp; otherwise, the same file name will be overwritten each time you + generate TTS.
+-d "output_file_timestamp=true"
-d "output_file_timestamp=false"
+
autoplay: Feature not yet available. Enable or + disable autoplay. Still needs to be specified in the JSON request.
+-d "autoplay=true"
-d "autoplay=false"
autoplay_volume: Feature not yet available. Set + the autoplay volume. Should be between 0.1 and 1.0. Still needs to be specified in the JSON + request.
+-d "autoplay_volume=0.8"
+The API returns a JSON object with the following properties:
+Example JSON TTS Generation Response:
+{"status": "generate-success", "output_file_path": + "C:\\text-generation-webui\\extensions\\alltalk_tts\\outputs\\myoutputfile_1703149973.wav", + "output_file_url": "http://127.0.0.1:7851/audio/myoutputfile_1703149973.wav"}
+ +curl -X POST
+ "http://127.0.0.1:7851/api/reload?tts_method=API%20Local"
+ curl -X POST "http://127.0.0.1:7851/api/reload?tts_method=API%20TTS"
curl -X POST
+ "http://127.0.0.1:7851/api/reload?tts_method=XTTSv2%20Local"
Switch between the 3 models respectively.
+JSON return {"status": "model-success"}
+ +curl -X POST + "http://127.0.0.1:7851/api/deepspeed?new_deepspeed_value=True"
+Replace True with False to disable DeepSpeed mode.
+JSON return {"status": "deepspeed-success"} +
+ +curl -X POST + "http://127.0.0.1:7851/api/lowvramsetting?new_low_vram_value=True"
+Replace True with False to disable Low VRAM mode.
+JSON return {"status": "lowvram-success"} +
+ +Check if the Text-to-Speech (TTS) service is ready to accept requests.
+curl -X GET + "http://127.0.0.1:7851/api/ready"
+ +Retrieve a list of available voices for generating speech.
+curl -X GET + "http://127.0.0.1:7851/api/voices"
+JSON return: {"voices": ["voice1.wav", + "voice2.wav", "voice3.wav"]}
+ +Generate a preview of a specified voice with hardcoded settings.
+curl -X POST + "http://127.0.0.1:7851/api/previewvoice/" -F "voice=female_01.wav"
+Replace female_01.wav with the name of the + voice sample you want to hear.
+JSON return: {"status": "generate-success", + "output_file_path": "/path/to/outputs/api_preview_voice.wav", "output_file_url": + "http://127.0.0.1:7851/audio/api_preview_voice.wav"}
+ + + +Command line outputs are more verbose to assist in + understanding backend processes and debugging.
+Its possible during startup you can get a warning message
+ such as:
[AllTalk
+ Startup] Warning TTS Subprocess
+ has NOT started up yet, Will keep trying for 80 seconds maximum
This is normal
+ behavior if the subprocess is taking a while to start, however, if there is an issue starting the
+ subprocess, you may see multiples of this message and an it will time out after 80 seconds, resulting in the
+ TTS engine not starting. Its likely that you are not in the correct python environment or one that has a TTS
+ engine inside, if this happens, though the system will output a warning about that ahead of this message
Typically the command line console will output any warning + or error messages. If you need to reset your default configuation, the settings are all listed above in the + configuration details.
+ + +