If you have a voice that the model doesnt quite reproduce correctly, or indeed you just want to improve the reproduced voice, then finetuning is a way to train your "XTTSv2 local" model (stored in /alltalk_tts/models/xxxxx/) on a specific voice. For this you will need:
+ +Everything has been done to make this as simple as possible. At its simplest, you can literally just download a large chunk of audio from an interview, and tell the finetuning to strip through it, find spoken parts and build your dataset. You can literally click 4 buttons, then copy a few files and you are done. At it's more complicated end you will clean up the audio a little beforehand, but its still only 4x buttons and copying a few files.
+ +I would suggest that if its in an interview format, you cut out the interviewer speaking in audacity or your chosen audio editing package. You dont have to worry about being perfect with your cuts, the finetuning Step 1 will go and find spoken audio and cut it out for you. Is there is music over the spoken parts, for best quality you would cut out those parts, though its not 100% necessary. As always, try to avoid bad quality audio with noises in it (humming sounds, hiss etc). You can try something like Audioenhancer to try clean up noisier audio. There is no need to down-sample any of the audio, all of that is handled for you. Just give the finetuning some good quality audio to work with.
+ +As mentioned you must have a small portion of the Nvidia CUDA Toolkit 11.8 installed. Not higher or lower versions. Specifically 11.8. You do not have to uninstall any other versions, change any graphics drivers, reinstall torch or anything like that. There are instructions within the finetuning interface on doing this or you can also find them on this link here
+ +Ensure you have followed the instructions on setting up the Nvidia CUDA Toolkit 11.8 here or the below procedure will fail.
+ +The below instructions are also available online here
+ +Close all other applications that are using your GPU/VRAM and copy your audio samples into:
+
+ /alltalk_tts/finetune/put-voice-samples-in-here/
In a command prompt/terminal window you need to move into your Text generation webUI folder:
+
+ cd text-generation-webui
Start the Text generation webUI Python environment for your OS:
+
+ cmd_windows.bat, ./cmd_linux.sh, cmd_macos.sh or cmd_wsl.bat
You can double check your search path environment still works correctly with nvcc --version. It should report back 11.8:
+
+ Cuda compilation tools, release 11.8.
Move into your extensions folder:
+
+ cd extensions
Move into the alltalk_tts folder:
+
+ cd alltalk_tts
Install the finetune requirements file: pip install -r requirements_finetune.txt
+Type python finetune.py and it should start up.
+Follow the on-screen instructions when the web interface starts up.
+When you have finished finetuning, the final tab will tell you what to do with your files and how to move your newly trained model to the correct location on disk.
+DeepSpeed provides a 2x-3x speed boost for Text-to-Speech and AI tasks. It's all about making AI and TTS happen faster and more efficiently.
curl -X POST -H "Content-Type: application/json" -d '{"text": "This is text to generate as TTS","voice": "female_01.wav", "language": "en", "output_file": "outputfile.wav"}' "http://127.0.0.1:7851/api/generate"
-curl -X POST -H "Content-Type: application/json" -d "{\"text\": \"This is text to generate as TTS\", \"voice\": \"female_01.wav\", \"language\": \"en\", \"output_file\": \"outputfile.wav\"}" http://127.0.0.1:7851/api/generate
-Replace This is text to generate as TTS with whatever you want it to say. female_01.wav with the voice sample you want to use. output_file.wav with the file name you want it to create.
-JSON return {"status":"generate-success","data":{"audio_path":"outputfile.wav"}}
+ +The Text-to-Speech (TTS) Generation API allows you to generate speech from text input using various configuration options. This API supports both character and narrator voices, providing flexibility for creating dynamic and engaging audio content.
+Standard TTS speech Example (standard text) generating a time-stamped file
+curl -X POST "http://127.0.0.1:7851/api/tts-generate" -d "text_input=All of this is text spoken by the character. This is text not inside quotes, though that doesnt matter in the slightest" -d "text_filtering=standard" -d "character_voice_gen=female_01.wav" -d "narrator_enabled=false" -d "narrator_voice_gen=male_01.wav" -d "text_not_inside=character" -d "language=en" -d "output_file_name=myoutputfile" -d "output_file_timestamp=true" -d "autoplay=true" -d "autoplay_volume=0.8"
+Narrator Example (standard text) generating a time-stamped file
+curl -X POST "http://127.0.0.1:7851/api/tts-generate" -d "text_input=*This is text spoken by the narrator* \"This is text spoken by the character\". This is text not inside quotes." -d "text_filtering=standard" -d "character_voice_gen=female_01.wav" -d "narrator_enabled=true" -d "narrator_voice_gen=male_01.wav" -d "text_not_inside=character" -d "language=en" -d "output_file_name=myoutputfile" -d "output_file_timestamp=true" -d "autoplay=true" -d "autoplay_volume=0.8"
+Note that if your text that needs to be generated contains double quotes you will need to escape them with \" (Please see the narrator example).
+text_input: The text you want the TTS engine to produce. Use escaped double quotes for character speech and asterisks for narrator speech if using the narrator function. Example:
+-d "text_input=*This is text spoken by the narrator* \"This is text spoken by the character\". This is text not inside quotes."
+text_filtering: Filter for text. Options:
+-d "text_filtering=none"
-d "text_filtering=standard"
-d "text_filtering=html"
Example:
+character_voice_gen: The WAV file name for the character's voice.
+-d "character_voice_gen=female_01.wav"
+narrator_enabled: Enable or disable the narrator function. If true, minimum text filtering is set to standard. Anything between double quotes is considered the character's speech, and anything between asterisks is considered the narrator's speech.
+-d "narrator_enabled=true"
-d "narrator_enabled=false"
narrator_voice_gen: The WAV file name for the narrator's voice.
+-d "narrator_voice_gen=male_01.wav"
+text_not_inside: Specify the handling of lines not inside double quotes or asterisks, for the narrator feature. Options:
+-d "text_not_inside=character"
-d "text_not_inside=narrator"
language: Choose the language for TTS. Options:
+-d "language=en"
+output_file_name: The name of the output file (excluding the .wav extension).
+-d "output_file_name=myoutputfile"
+output_file_timestamp: Add a timestamp to the output file name. If true, each file will have a unique timestamp; otherwise, the same file name will be overwritten each time you generate TTS.
+-d "output_file_timestamp=true"
-d "output_file_timestamp=false"
autoplay: Feature not yet available. Enable or disable autoplay. Still needs to be specified in the JSON request.
+-d "autoplay=true"
-d "autoplay=false"
autoplay_volume: Feature not yet available. Set the autoplay volume. Should be between 0.1 and 1.0. Still needs to be specified in the JSON request.
+-d "autoplay_volume=0.8"
+The API returns a JSON object with the following properties:
+Example JSON TTS Generation Response:
+{"status": "generate-success", "output_file_path": "C:\\text-generation-webui\\extensions\\alltalk_tts\\outputs\\myoutputfile_1703149973.wav", "output_file_url": "http://127.0.0.1:7851/audio/myoutputfile_1703149973.wav"}
+curl -X POST "http://127.0.0.1:7851/api/reload?tts_method=API%20Local"
curl -X POST "http://127.0.0.1:7851/api/reload?tts_method=API%20TTS"
curl -X POST "http://127.0.0.1:7851/api/reload?tts_method=XTTSv2%20Local"
Switch between the 3 models respectively.
JSON return {"status": "model-success"}
curl -X POST -H "Content-Type: application/json" "http://127.0.0.1:7851/api/deepspeed?new_deepspeed_value=True"
+curl -X POST "http://127.0.0.1:7851/api/deepspeed?new_deepspeed_value=True"
Replace True with False to disable DeepSpeed mode.
JSON return {"status": "deepspeed-success"}
curl -X POST -H "Content-Type: application/json" "http://127.0.0.1:7851/api/lowvramsetting?new_low_vram_value=True"
+curl -X POST "http://127.0.0.1:7851/api/lowvramsetting?new_low_vram_value=True"
Replace True with False to disable Low VRAM mode.
JSON return {"status": "lowvram-success"}
Command line outputs are more verbose to assist in understanding backend processes and debugging.
-Its possible during startup you can get a warning message such as:
[AllTalk Startup] Warning TTS Subprocess has NOT started up yet, Will keep trying for 60 seconds maximum
This is normal behavior if the subprocess is taking a while to start, however, if there is an issue starting the subprocess, you may see multiples of this message and an it will time out after 60 seconds, resulting in the TTS engine not starting. Its likely that you are not in the correct python environment or one that has a TTS engine inside, if this happens, though the system will output a warning about that ahead of this message
Its possible during startup you can get a warning message such as:
[AllTalk Startup] Warning TTS Subprocess has NOT started up yet, Will keep trying for 80 seconds maximum
This is normal behavior if the subprocess is taking a while to start, however, if there is an issue starting the subprocess, you may see multiples of this message and an it will time out after 80 seconds, resulting in the TTS engine not starting. Its likely that you are not in the correct python environment or one that has a TTS engine inside, if this happens, though the system will output a warning about that ahead of this message
Typically the command line console will output any warning or error messages. If you need to reset your default configuation, the settings are all listed above in the configuration details.