Merge pull request #14 from erew123/dev
New Finetuning of your models New API endpoint Full documentation (online and built in) A couple of minor file updates. Small bug fix on file generation in instruct mode with no character name available (thanks to @danielwburch)
This commit is contained in:
9 files changed
+1652
-51
No files matched your search
@@ -2,14 +2,16 @@
|
||||
AllTalk is an updated version of the Coqui_tts extension for Text Generation web UI. Features include:
|
||||
|
||||
- **Custom Start-up Settings:** Adjust your default start-up settings. [Screenshot](https://github.com/erew123/alltalk_tts#screenshots)
|
||||
- **Narrarator:** Use different voices for main character and narration. [Example Narration](https://vocaroo.com/18fYWVxiQpk1)
|
||||
- **Narrarator:** Use different voices for main character and narration. [Example Narration](https://vocaroo.com/18nrv7FR6wuA)
|
||||
- **Low VRAM mode:** Great for people with small GPU memory or if your VRAM is filled by your LLM. [Screenshot](https://github.com/erew123/alltalk_tts#screenshots)
|
||||
- **DeepSpeed:** A 3-4x performance boost generating TTS. [DeepSpeed Windows/Linux Instructions](https://github.com/erew123/alltalk_tts?tab=readme-ov-file#deepspeed-installation-options) [Screenshot](https://github.com/erew123/alltalk_tts/assets/35898566/548619c8-5f1b-47d0-a73d-54d2fee3f3db)
|
||||
- **Local/Custom models:** Use any of the XTTSv2 models (API Local and XTTSv2 Local).
|
||||
- **Optional wav file maintenance:** Configurable deletion of old output wav files. [Screenshot](https://github.com/erew123/alltalk_tts#screenshots)
|
||||
- **Finetuning** Train the model specifically on a voice of your choosing for better reproduction.
|
||||
- **Documentation:** Fully documented with a built in webpage. [Screenshot](https://github.com/erew123/alltalk_tts#screenshots)
|
||||
- **Console output** Clear command line output for any warnings or issues.
|
||||
- **Standalone/3rd Party support via JSON calls** Can be used with 3rd party applications via JSON calls.
|
||||
- **API Suite and 3rd Party support via JSON calls** Can be used with 3rd party applications via JSON calls.
|
||||
- **Can be run as a standalone app** Not just inside of text-generation-webui.
|
||||
|
||||
## Index
|
||||
|
||||
@@ -17,7 +19,9 @@ AllTalk is an updated version of the Coqui_tts extension for Text Generation web
|
||||
- 🟪 [Updating & problems with updating](https://github.com/erew123/alltalk_tts?#-updating)
|
||||
- 🟫 [Screenshots](https://github.com/erew123/alltalk_tts#-screenshots)
|
||||
- 🟨 [Help with problems](https://github.com/erew123/alltalk_tts?#-help-with-problems)
|
||||
- ⚫ [Finetuning a model](https://github.com/erew123/alltalk_tts?#-finetuning-a-model)
|
||||
- 🔵🟢🟡 [DeepSpeed Installation (Windows & Linux)](https://github.com/erew123/alltalk_tts?#-deepspeed-installation-options)
|
||||
- 🟠 [API Suite and JSON-CURL](https://github.com/erew123/alltalk_tts?#-api-suite-and-json-curl)
|
||||
- 🔴 [Future to-do list & Upcoming updates](https://github.com/erew123/alltalk_tts?#-future-to-do-list)
|
||||
|
||||
#### Updates
|
||||
@@ -43,9 +47,9 @@ This has been tested on the current Dec 2023 release of Text generation webUI. I
|
||||
5) Move into the **alltalk_tts** folder:<br><br>
|
||||
`cd alltalk_tts`
|
||||
|
||||
6) Install the requirements:<br><br>
|
||||
*Nvidia graphics card machines* - `pip install -r requirements_nvidia.txt`<br><br>
|
||||
*Other machines (mac, amd etc)* - `pip install -r requirements_other.txt`
|
||||
6) Install the requirements that are correct for your machine:<br><br>
|
||||
**Nvidia graphics card machines** - `pip install -r requirements_nvidia.txt`<br><br>
|
||||
**Other machines (mac, amd etc)** - `pip install -r requirements_other.txt`
|
||||
|
||||
7) **(Optional DeepSpeed)** If you have an Nvidia Graphics card on a system running Linux or Windows and wish to use **DeepSpeed** please follow these instructions [here](https://github.com/erew123/alltalk_tts?#-deepspeed-installation-options). **However**, I would highly reccommend before you install DeepSpeed, you start text-generation-webui up, confirm AllTalk starts correctly and everything is working, as DeepSpeed can add another layer of complications troubleshooting any potential start-up issues. If necessary you can `pip uninstall deepspeed`.
|
||||
|
||||
@@ -81,9 +85,9 @@ This is pretty much a repeat of the installation process.
|
||||
3) At the command prompt/terminal, type:<br><br>
|
||||
`git pull`
|
||||
|
||||
4) Install the requirements:<br><br>
|
||||
*Nvidia graphics card machines* - `pip install -r requirements_nvidia.txt`<br><br>
|
||||
*Other machines (mac, amd etc)* - `pip install -r requirements_other.txt`
|
||||
4) Install the correct requirements for your machine:<br><br>
|
||||
**Nvidia graphics card machines** - `pip install -r requirements_nvidia.txt`<br><br>
|
||||
**Other machines (mac, amd etc)** - `pip install -r requirements_other.txt`
|
||||
|
||||
#### 🟪 Updating "git pull" error
|
||||
|
||||
@@ -123,9 +127,9 @@ This will download a fresh installation.
|
||||
3) Move into the **alltalk_tts** folder:<br><br>
|
||||
`cd alltalk_tts`
|
||||
|
||||
4) Install the requirements:<br><br>
|
||||
*Nvidia graphics card machines* - `pip install -r requirements_nvidia.txt`<br><br>
|
||||
*Other machines (mac, amd etc)* - `pip install -r requirements_other.txt`
|
||||
4) Install the correct requirements for your machine:<br><br>
|
||||
**Nvidia graphics card machines** - `pip install -r requirements_nvidia.txt`<br><br>
|
||||
**Other machines (mac, amd etc)** - `pip install -r requirements_other.txt`
|
||||
|
||||
5) Before starting it up, copy/merge the `models`, `voices` and `outputs` folders over from the `alltalk_tts.old` folder to the newly created `alltalk_tts` folder. This will keep your voices history and also stop it re-downloading the model again.
|
||||
|
||||
@@ -135,9 +139,10 @@ Assuming its all working fine and you are happy, you can delete the old alltalk_
|
||||
</details>
|
||||
|
||||
## 🟫 Screenshots
|
||||
| |  |
|
||||
| |  |
|
||||
|:---:|:---:|
|
||||
| |  |
|
||||
| |  |
|
||||
| |  |
|
||||
|
||||
## 🟨 Help with problems
|
||||
|
||||
@@ -160,7 +165,7 @@ Assuming its all working fine and you are happy, you can delete the old alltalk_
|
||||
<br><br></details>
|
||||
|
||||
|
||||
#### 🟨 [AllTalk Startup] Warning TTS Subprocess has NOT started up yet, Will keep trying for 60 seconds maximum. Please wait. It times out after 60 seconds.
|
||||
#### 🟨 [AllTalk Startup] Warning TTS Subprocess has NOT started up yet, Will keep trying for 120 seconds maximum. Please wait. It times out after 120 seconds.
|
||||
|
||||
<details>
|
||||
<summary>Click to expand</summary><br>
|
||||
@@ -171,7 +176,9 @@ Assuming its all working fine and you are happy, you can delete the old alltalk_
|
||||
|
||||
**Note:** If you need to create a support ticket, please create a `diagnostics.log` report file to submit with a support request. Details on doing this are above.
|
||||
|
||||
Possibilities for this issue are:
|
||||
Before giving other possibilities, some people with **old machines** are finding their startup times are **very** slow 2-3 minutes. Ive extended the allowed time within the script from 1 minute to 2 minutes. **If you have an older machine** and wish to try extending this further, you can do so by editing `script.py` and changing line 251 `timeout = 120` changing the timeout to a larger value e.g `timeout = 240` (4 minutes).
|
||||
|
||||
Other possibilities for this issue are:
|
||||
|
||||
1) You are starting AllTalk in both your `CMD FLAG.txt` and `settings.yaml` file. The `CMD FLAG.txt` you would have manually edited and the `settings.yaml` is the one you change and save in the `session` tab of text-generation-webui and you can `Save UI defaults to settings.yaml`. Please only have one of those two starting up AllTalk.
|
||||
|
||||
@@ -230,6 +237,59 @@ I would suggest following [Problems Updating](https://github.com/erew123/alltalk
|
||||
As far as I am aware, these are to do with the chrome browser the gradio text-generation-webui in some way. I raised an issue about this on the text-generation-webui [here](https://github.com/oobabooga/text-generation-webui/issues/4788) where you can see that AllTalk is not loaded and the messages persist. Either way, this is more a warning than an actual issue, so shouldnt affect any functionality of either AllTalk or text-generation-webui, they are more just an annoyance.
|
||||
</details>
|
||||
|
||||
## ⚫ Finetuning a model
|
||||
If you have a voice that the model doesnt quite reproduce correctly, or indeed you just want to improve the reproduced voice, then finetuning is a way to train your "XTTSv2 local" model **(stored in `/alltalk_tts/models/xxxxx/`)** on a specific voice. For this you will need:
|
||||
|
||||
- An Nvidia graphics card
|
||||
- To install a few portions of the Nvidia CUDA 11.8 Toolkit (this will not impact text-generation-webui's cuda setup.
|
||||
- 18GB of disk space free (most of this is used temporarily)
|
||||
- At least 2 minutes of good quality speech from your chosen speaker in mp3, wav or flacc format, in one or more files (have tested as far as 20 minutes worth of audio).
|
||||
|
||||
#### ⚫ How will this work/How complicated is it?
|
||||
Everything has been done to make this as simple as possible. At its simplest, you can literally just download a large chunk of audio from an interview, and tell the finetuning to strip through it, find spoken parts and build your dataset. You can literally click 4 buttons, then copy a few files and you are done. At it's more complicated end you will clean up the audio a little beforehand, but its still only 4x buttons and copying a few files.
|
||||
|
||||
#### ⚫ The audio you will use
|
||||
I would suggest that if its in an interview format, you cut out the interviewer speaking in audacity or your chosen audio editing package. You dont have to worry about being perfect with your cuts, the finetuning Step 1 will go and find spoken audio and cut it out for you. Is there is music over the spoken parts, for best quality you would cut out those parts, though its not 100% necessary. As always, try to avoid bad quality audio with noises in it (humming sounds, hiss etc). You can try something like [Audioenhancer](https://audioenhancer.ai/) to try clean up noisier audio. There is no need to down-sample any of the audio, all of that is handled for you. Just give the finetuning some good quality audio to work with.
|
||||
|
||||
#### ⚫ Important requirements CUDA 11.8
|
||||
As mentioned you must have a small portion of the Nvidia CUDA Toolkit **11.8** installed. Not higher or lower versions. Specifically **11.8**. You do not have to uninstall any other versions, change any graphics drivers, reinstall torch or anything like that. To keep the download+install as small as possible, you will need to:
|
||||
- Download the **xxx (network)** install of the Nvidia Cuda Toolkit 11.8 from [here](https://developer.nvidia.com/cuda-11-8-0-download-archive)
|
||||
- When you run the installation, select **Custom Advanced** as your installation type. Uncheck `CUDA`, `Other Components` and `Driver Components` at the top then expand `CUDA` > `Development` > `Compiler` > and select `nvcc` then expand `Libraries` and select `CUBLAS`
|
||||
- Back at the top of `CUDA` expand `Runtime` > `Libraries` and select `CUBLAS`. You can now next through the install.
|
||||
- When that has installed, open a terminal/command prompt and type `nvcc --version`. If it reports back `Cuda compilation tools, release 11.8.` you are good to go. **Specifically, 11.8**. If not continue to the next step.
|
||||
- For both Windows an Linux, you will need to ensure that `nvcc` and the 11.8 cuda library files are in your environments search path. You can undo the changes below after finetuning if you prefer.<br><br>
|
||||
**Windows** - Edit the Windows PATH environment variable and add `C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.8\bin`<br><br>
|
||||
**Linux** - The path may be different depending on what flavour of Linux you are running, so you may need to seek out specific instructions on the internet. Generic paths **may** be:<br>
|
||||
`export LD_LIBRARY_PATH=/usr/local/cuda-11.8/lib64:${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}` and<br>
|
||||
`export LD_LIBRARY_PATH=/usr/local/cuda-11.8/bin`<br>
|
||||
Add these to your '~/.bashrc' if you want this to be permanent and not something you have to set each time you open a new terminal.<br><br>
|
||||
- When you have made the changes, open a new terminal/command prompt and `nvcc --version`. It should report back `Cuda compilation tools, release 11.8.` at which point, you are good to go.
|
||||
- If it doesnt report that, check you have correctly set the search environment paths, dont have overlapping other versions of cuda paths etc.
|
||||
|
||||
#### ⚫ Starting Finetuning
|
||||
1) Close all other applications that are using your GPU/VRAM and copy your audio samples into:<br><br>
|
||||
`/alltalk_tts/finetune/put-voice-samples-in-here/`
|
||||
3) In a command prompt/terminal window you need to move into your Text generation webUI folder:<br><br>
|
||||
`cd text-generation-webui`
|
||||
|
||||
4) Start the Text generation webUI Python environment for your OS:<br><br>
|
||||
`cmd_windows.bat`, `./cmd_linux.sh`, `cmd_macos.sh` or `cmd_wsl.bat`
|
||||
|
||||
5) You can double check your search path environment still works correctly with `nvcc --version`. It should report back 11.8:<br><br>
|
||||
`Cuda compilation tools, release 11.8.`
|
||||
|
||||
7) Move into your extensions folder:<br><br>
|
||||
`cd extensions`
|
||||
|
||||
8) Move into the **alltalk_tts** folder:<br><br>
|
||||
`cd alltalk_tts`
|
||||
|
||||
9) Install the finetune requirements file: `pip install -r requirements_finetune.txt`
|
||||
|
||||
10) Type `python finetune.py` and it should start up.
|
||||
11) Follow the on-screen instructions when the web interface starts up.
|
||||
12) When you have finished finetuning, the final tab will tell you what to do with your files and how to move your newly trained model to the correct location on disk.
|
||||
|
||||
## 🔵🟢🟡 DeepSpeed Installation Options
|
||||
#### 🔵 Linux Installation
|
||||
<details>
|
||||
@@ -381,10 +441,142 @@ def get_shm_size():
|
||||
**Removal** - If it became necessary to uninstall DeepSpeed, you can do so with `cmd_windows.bat` and then `pip uninstall deepspeed`<br><br>
|
||||
</details>
|
||||
|
||||
## 🟠 API Suite and JSON-CURL
|
||||
### 🟠Overview
|
||||
The Text-to-Speech (TTS) Generation API allows you to generate speech from text input using various configuration options. This API supports both character and narrator voices, providing flexibility for creating dynamic and engaging audio content.
|
||||
|
||||
- URL: `http://127.0.0.1:7851/api/tts-generate`<br>
|
||||
- Method: `POST`<br>
|
||||
- Content-Type: `application/x-www-form-urlencoded`<br>
|
||||
|
||||
### 🟠 Example command lines
|
||||
Standard TTS speech Example (standard text) generating a time-stamped file<br>
|
||||
|
||||
`curl -X POST "http://127.0.0.1:7851/api/tts-generate" -d "text_input=All of this is text spoken by the character. This is text not inside quotes, though that doesnt matter in the slightest" -d "text_filtering=standard" -d "character_voice_gen=female_01.wav" -d "narrator_enabled=false" -d "narrator_voice_gen=male_01.wav" -d "text_not_inside=character" -d "language=en" -d "output_file_name=myoutputfile" -d "output_file_timestamp=true" -d "autoplay=true" -d "autoplay_volume=0.8"`<br>
|
||||
|
||||
Narrator Example (standard text) generating a time-stamped file
|
||||
|
||||
`curl -X POST "http://127.0.0.1:7851/api/tts-generate" -d "text_input=*This is text spoken by the narrator* \"This is text spoken by the character\". This is text not inside quotes." -d "text_filtering=standard" -d "character_voice_gen=female_01.wav" -d "narrator_enabled=true" -d "narrator_voice_gen=male_01.wav" -d "text_not_inside=character" -d "language=en" -d "output_file_name=myoutputfile" -d "output_file_timestamp=true" -d "autoplay=true" -d "autoplay_volume=0.8"`<br>
|
||||
|
||||
Note that if your text that needs to be generated contains double quotes you will need to escape them with `\"` (Please see the narrator example).
|
||||
|
||||
### 🟠 Request Parameters
|
||||
🟠 **text_input**: The text you want the TTS engine to produce. Use escaped double quotes for character speech and asterisks for narrator speech if using the narrator function. Example:
|
||||
|
||||
`-d "text_input=*This is text spoken by the narrator* \"This is text spoken by the character\". This is text not inside quotes."`
|
||||
|
||||
🟠 **text_filtering**: Filter for text. Options:
|
||||
|
||||
- **none** No filtering. Whatever is sent will go over to the TTS engine as raw text, which may result in some odd sounds with some special characters.<br>
|
||||
- **standard** Human-readable text and a basic level of filtering, just to clean up some special characters.<br>
|
||||
- **html** HTML content. Where you are using HTML entity's like "<br>
|
||||
|
||||
`-d "text_filtering=none"`<br>
|
||||
`-d "text_filtering=standard"`<br>
|
||||
`-d "text_filtering=html"`<br>
|
||||
|
||||
Example:
|
||||
|
||||
- **Standard Example**: `*This is text spoken by the narrator* "This is text spoken by the character" This is text not inside quotes.`<br>
|
||||
- **HTML Example**: `*This is text spoken by the narrator* "This is text spoken by the character" This is text not inside quotes.`<br>
|
||||
- **None**: `Will just pass whatever characters/text you send at it.`<br>
|
||||
|
||||
🟠 **character_voice_gen**: The WAV file name for the character's voice.<br>
|
||||
|
||||
`-d "character_voice_gen=female_01.wav"`
|
||||
|
||||
🟠 **narrator_enabled**: Enable or disable the narrator function. If true, minimum text filtering is set to standard. Anything between double quotes is considered the character's speech, and anything between asterisks is considered the narrator's speech.
|
||||
|
||||
`-d "narrator_enabled=true"`<br>
|
||||
`-d "narrator_enabled=false"`
|
||||
|
||||
🟠 **narrator_voice_gen**: The WAV file name for the narrator's voice.
|
||||
|
||||
`-d "narrator_voice_gen=male_01.wav"`
|
||||
|
||||
🟠 **text_not_inside**: Specify the handling of lines not inside double quotes or asterisks, for the narrator feature. Options:
|
||||
|
||||
- **character**: Treat as character speech.<br>
|
||||
- **narrator**: Treat as narrator speech.<br>
|
||||
|
||||
`-d "text_not_inside=character"`<br>
|
||||
`-d "text_not_inside=narrator"`
|
||||
|
||||
🟠 **language**: Choose the language for TTS. Options:
|
||||
|
||||
`ar Arabic`<br>
|
||||
`zh-cn Chinese (Simplified)`<br>
|
||||
`cs Czech`<br>
|
||||
`nl Dutch`<br>
|
||||
`en English`<br>
|
||||
`fr French`<br>
|
||||
`de German`<br>
|
||||
`hu Hungarian`<br>
|
||||
`it Italian`<br>
|
||||
`ja Japanese`<br>
|
||||
`ko Korean`<br>
|
||||
`pl Polish`<br>
|
||||
`pt Portuguese`<br>
|
||||
`ru Russian`<br>
|
||||
`es Spanish`<br>
|
||||
`tr Turkish`<br>
|
||||
|
||||
`-d "language=en"`<br>
|
||||
|
||||
🟠 **output_file_name**: The name of the output file (excluding the .wav extension).
|
||||
|
||||
`-d "output_file_name=myoutputfile"`<br>
|
||||
|
||||
🟠 **output_file_timestamp**: Add a timestamp to the output file name. If true, each file will have a unique timestamp; otherwise, the same file name will be overwritten each time you generate TTS.
|
||||
|
||||
`-d "output_file_timestamp=true"`<br>
|
||||
`-d "output_file_timestamp=false"`
|
||||
|
||||
🟠 **autoplay**: Feature not yet available. Enable or disable autoplay. Still needs to be specified in the JSON request.
|
||||
|
||||
`-d "autoplay=true"`<br>
|
||||
`-d "autoplay=false"`
|
||||
|
||||
🟠 **autoplay_volume**: Feature not yet available. Set the autoplay volume. Should be between 0.1 and 1.0. Still needs to be specified in the JSON request.
|
||||
|
||||
`-d "autoplay_volume=0.8"`
|
||||
|
||||
### 🟠 TTS Generation Response
|
||||
The API returns a JSON object with the following properties:
|
||||
|
||||
- **status** Indicates whether the generation was successful (generate-success) or failed (generate-failure).<br>
|
||||
- **output_file_path** The on-disk location of the generated WAV file.<br>
|
||||
- **output_file_url** The HTTP location for accessing the generated WAV file.<br>
|
||||
|
||||
Example JSON TTS Generation Response:
|
||||
|
||||
`{"status": "generate-success", "output_file_path": "C:\text-generation-webui\extensions\alltalk_tts\outputs\myoutputfile_1703149973.wav", "output_file_url": "http://127.0.0.1:7851/audio/myoutputfile_1703149973.wav"}`
|
||||
|
||||
🟠 **Switching Model**<br><br>
|
||||
`curl -X POST "http://127.0.0.1:7851/api/reload?tts_method=API%20Local"`<br>
|
||||
`curl -X POST "http://127.0.0.1:7851/api/reload?tts_method=API%20TTS"`<br>
|
||||
`curl -X POST "http://127.0.0.1:7851/api/reload?tts_method=XTTSv2%20Local"`<br>
|
||||
|
||||
Switch between the 3 models respectively.
|
||||
|
||||
JSON return `{"status": "model-success"}`
|
||||
|
||||
🟠 **Switch DeepSpeed**<br><br>
|
||||
`curl -X POST "http://127.0.0.1:7851/api/deepspeed?new_deepspeed_value=True"`
|
||||
|
||||
Replace True with False to disable DeepSpeed mode.
|
||||
|
||||
JSON return `{"status": "deepspeed-success"}`
|
||||
|
||||
🟠 **Switching Low VRAM**<br><br>
|
||||
`curl -X POST "http://127.0.0.1:7851/api/lowvramsetting?new_low_vram_value=True"`
|
||||
|
||||
Replace True with False to disable Low VRAM mode.
|
||||
|
||||
JSON return `{"status": "lowvram-success"}`
|
||||
|
||||
### 🔴 Future to-do list
|
||||
- Complete & document the new/full standalone mode API.
|
||||
- Voice output within the command prompt/terminal (TBD).
|
||||
- Correct an issue on incorrect output folder path when running as a standalone app.
|
||||
- Correct a few spelling mistakes in the documnentation.
|
||||
- Correct a few spelling mistakes in the documentation.
|
||||
- Possibly add some additional TTS engines (TBD).
|
||||
- Have a break!
|
||||
+1017
File diff suppressed because it is too large.
Load diff
Whitespace-only changes.
@@ -6,6 +6,8 @@
|
||||
"README.md": "https://huggingface.co/coqui/XTTS-v2/resolve/v2.0.2/README.md?download=true",
|
||||
"config.json": "https://huggingface.co/coqui/XTTS-v2/resolve/v2.0.2/config.json?download=true",
|
||||
"model.pth": "https://huggingface.co/coqui/XTTS-v2/resolve/v2.0.2/model.pth?download=true",
|
||||
"dvae.pth": "https://huggingface.co/coqui/XTTS-v2/resolve/v2.0.2/dvae.pth?download=true",
|
||||
"mel_stats.pth": "https://huggingface.co/coqui/XTTS-v2/resolve/v2.0.2/mel_stats.pth?download=true",
|
||||
"vocab.json": "https://huggingface.co/coqui/XTTS-v2/resolve/v2.0.2/vocab.json?download=true"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,7 @@
|
||||
gradio>=3.50.2
|
||||
torch>=2.1.0+cu118
|
||||
torchaudio>=2.1.0+cu118
|
||||
TTS==0.21.3
|
||||
faster-whisper>=0.10.0
|
||||
tqdm>=4.66.0
|
||||
pandas>=1.5.0
|
||||
@@ -1,7 +1,7 @@
|
||||
numpy>=1.22.0
|
||||
soundfile>=0.12.1
|
||||
uvicorn>=0.24.0.post1
|
||||
TTS>=0.22.0
|
||||
TTS>=0.21.3
|
||||
torch>=2.1.0+cu118
|
||||
torchaudio>=2.1.0+cu118
|
||||
fastapi>=0.104.1
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
numpy>=1.22.0
|
||||
soundfile>=0.12.1
|
||||
uvicorn>=0.24.0.post1
|
||||
TTS>=0.22.0
|
||||
TTS>=0.21.3
|
||||
torch>=2.1.0
|
||||
torchaudio>=2.1.0
|
||||
fastapi>=0.104.1
|
||||
|
||||
@@ -39,8 +39,8 @@ try:
|
||||
except ModuleNotFoundError:
|
||||
logger.error(
|
||||
f"[{params['branding']}]\033[91mWarning\033[0m Could not find the TTS module. Make sure to install the requirements for the {params['branding']} extension."
|
||||
f"[{params['branding']}]\033[91mWarning\033[0m Linux / Mac:\npip install -r extensions/alltalk_tts/requirements.txt\n"
|
||||
f"[{params['branding']}]\033[91mWarning\033[0m Windows:\npip install -r extensions\\alltalk_tts\\requirements.txt\n"
|
||||
f"[{params['branding']}]\033[91mWarning\033[0m Linux / Mac:\npip install -r /alltalk_tts/requirements.txt\n"
|
||||
f"[{params['branding']}]\033[91mWarning\033[0m Windows:\npip install -r \\alltalk_tts\\requirements.txt\n"
|
||||
f"[{params['branding']}]\033[91mWarning\033[0m If you used the one-click installer, paste the command above in the terminal window launched after running the cmd_ script. On Windows, that's cmd_windows.bat."
|
||||
)
|
||||
raise
|
||||
@@ -62,7 +62,9 @@ try:
|
||||
from modules.utils import gradio
|
||||
# This is set to check if the script is being run within text generation webui or as a standalone script. False is running as part of text gen web ui or a gradio interface
|
||||
running_in_standalone = False
|
||||
output_folder_wav = params["output_folder_wav"]
|
||||
except ModuleNotFoundError:
|
||||
output_folder_wav = params["output_folder_wav_standalone"]
|
||||
print(f"[{params['branding']}Startup] Running script.py in standalone mode")
|
||||
# This is set to check if the script is being run within text generation webui or as a standalone script. true means standalone
|
||||
running_in_standalone = True
|
||||
@@ -108,7 +110,6 @@ def delete_old_files(folder_path, days_to_keep):
|
||||
|
||||
# Extract settings using params dictionary
|
||||
delete_output_wavs_setting = params["delete_output_wavs"]
|
||||
output_folder_wav = params["output_folder_wav"]
|
||||
output_folder_wav = os.path.normpath(output_folder_wav)
|
||||
|
||||
# Check and perform file deletion
|
||||
@@ -551,15 +552,10 @@ def output_modifier(string, state):
|
||||
else:
|
||||
cleaned_part = html.unescape(part.replace('< ', '').replace('< ', '').replace('< ', ''))
|
||||
voice_to_use = params["narrator_voice"]
|
||||
# Check if character name exists and if not, just call it TTSOUT_
|
||||
if "character_menu" in state:
|
||||
output_file = Path(f'{params["output_folder_wav"]}/{state["character_menu"]}_{int(time.time())}_{i}.wav')
|
||||
else:
|
||||
output_file = Path(f'{params["output_folder_wav"]}/TTSOUT_{int(time.time())}_{i}.wav')
|
||||
# Generate that TTS and output to a file
|
||||
output_file_str = output_file.as_posix()
|
||||
output_filename = get_output_filename(state)
|
||||
generate_response = send_generate_request(
|
||||
cleaned_part, voice_to_use, language_code, output_file_str
|
||||
cleaned_part, voice_to_use, language_code, output_filename
|
||||
)
|
||||
audio_path = generate_response.get("data", {}).get("audio_path")
|
||||
audio_files_paragraph.append(audio_path)
|
||||
@@ -580,14 +576,9 @@ def output_modifier(string, state):
|
||||
)
|
||||
cleaned_part = html.unescape(processed_string)
|
||||
# Process the part and give it a non-character name if being used vai API or standalone.
|
||||
if "character_menu" in state:
|
||||
output_file = Path(f'{params["output_folder_wav"]}/{state["character_menu"]}_{int(time.time())}.wav')
|
||||
else:
|
||||
output_file = Path(f'{params["output_folder_wav"]}/TTSOUT_{int(time.time())}.wav')
|
||||
output_file_str = output_file.as_posix()
|
||||
output_file = get_output_filename(state)
|
||||
output_filename = get_output_filename(state)
|
||||
generate_response = send_generate_request(
|
||||
cleaned_part, params["voice"], language_code, output_file_str
|
||||
cleaned_part, params["voice"], language_code, output_filename
|
||||
)
|
||||
audio_path = generate_response.get("data", {}).get("audio_path")
|
||||
final_output_file = audio_path
|
||||
@@ -627,9 +618,11 @@ def output_modifier(string, state):
|
||||
|
||||
|
||||
def get_output_filename(state):
|
||||
return Path(
|
||||
f'{params["output_folder_wav"]}/{state["character_menu"]}_{str(uuid.uuid4())[:8]}.wav'
|
||||
).as_posix()
|
||||
# Check if character name exists and if not, just call it TTSOUT_
|
||||
if "character_menu" in state:
|
||||
return Path(f'{params["output_folder_wav"]}/{state["character_menu"]}_{int(time.time())}.wav').as_posix()
|
||||
else:
|
||||
return Path(f'{params["output_folder_wav"]}/TTSOUT_{int(time.time())}.wav').as_posix()
|
||||
|
||||
|
||||
###############################################
|
||||
|
||||
+401
-11
@@ -22,6 +22,7 @@ from fastapi.templating import Jinja2Templates
|
||||
from fastapi.staticfiles import StaticFiles
|
||||
from jinja2 import Template
|
||||
from contextlib import asynccontextmanager
|
||||
from pydantic import field_validator
|
||||
|
||||
###########################
|
||||
#### STARTUP VARIABLES ####
|
||||
@@ -161,7 +162,9 @@ async def setup():
|
||||
# Set "tts_model_loaded" to true
|
||||
params["tts_model_loaded"] = True
|
||||
# Set the output path for wav files
|
||||
Path(f'{params["output_folder_wav"]}').mkdir(parents=True, exist_ok=True)
|
||||
output_directory = this_dir / params["output_folder_wav_standalone"]
|
||||
output_directory.mkdir(parents=True, exist_ok=True)
|
||||
#Path(f'this_folder/outputs/').mkdir(parents=True, exist_ok=True)
|
||||
|
||||
|
||||
# MODEL LOADER For "API TTS"
|
||||
@@ -617,6 +620,218 @@ async def get_audio(filename: str):
|
||||
audio_path = this_dir / "outputs" / filename
|
||||
return FileResponse(audio_path)
|
||||
|
||||
########################
|
||||
#### GENERATION API ####
|
||||
########################
|
||||
import html
|
||||
import re
|
||||
import numpy as np
|
||||
import soundfile as sf
|
||||
from typing import Union, Dict
|
||||
from pydantic import BaseModel, ValidationError, Field
|
||||
|
||||
class Request(BaseModel):
|
||||
# Define the structure of the 'Request' class if needed
|
||||
pass
|
||||
|
||||
class JSONInput(BaseModel):
|
||||
text_input: str = Field(..., max_length=1000, description="text_input needs to be 1000 characters or less.")
|
||||
text_filtering: str = Field(..., pattern="^(none|standard|html)$", description="text_filtering needs to be 'none', 'standard' or 'html'.")
|
||||
character_voice_gen: str = Field(..., pattern="^.*\.wav$", description="character_voice_gen needs to be the name of a wav file e.g. mysample.wav.")
|
||||
narrator_enabled: bool = Field(..., description="narrator_enabled needs to be true or false.")
|
||||
narrator_voice_gen: str = Field(..., pattern="^.*\.wav$", description="narrator_voice_gen needs to be the name of a wav file e.g. mysample.wav.")
|
||||
text_not_inside: str = Field(..., pattern="^(character|narrator)$", description="text_not_inside needs to be 'character' or 'narrator'.")
|
||||
language: str = Field(..., pattern="^(ar|zh-cn|cs|nl|en|fr|de|hu|it|ja|ko|pl|pt|ru|es|tr)$", description="language needs to be one of the following ar|zh-cn|cs|nl|en|fr|de|hu|it|ja|ko|pl|pt|ru|es|tr.")
|
||||
output_file_name: str = Field(..., pattern="^[a-zA-Z0-9_]+$", description="output_file_name needs to be the name without any special characters or file extension e.g. 'filename'")
|
||||
output_file_timestamp: bool = Field(..., description="output_file_timestamp needs to be true or false.")
|
||||
autoplay: bool = Field(..., description="autoplay needs to be a true or false value.")
|
||||
autoplay_volume: float = Field(..., ge=0.1, le=1.0, description="autoplay_volume needs to be from 0.1 to 1.0")
|
||||
|
||||
@field_validator("autoplay_volume")
|
||||
@classmethod
|
||||
def validate_autoplay_volume(cls, value):
|
||||
if not (0.1 <= value <= 1.0):
|
||||
raise ValidationError("Autoplay volume must be between 0.1 and 1.0")
|
||||
return value
|
||||
|
||||
|
||||
class TTSGenerator:
|
||||
@staticmethod
|
||||
def validate_json_input(json_data: Union[Dict, str]) -> Union[None, str]:
|
||||
try:
|
||||
if isinstance(json_data, str):
|
||||
json_data = json.loads(json_data)
|
||||
JSONInput(**json_data)
|
||||
return None # JSON is valid
|
||||
except ValidationError as e:
|
||||
return str(e)
|
||||
|
||||
def standard_filtering(text_input):
|
||||
text_output = (text_input
|
||||
.replace("***", "")
|
||||
.replace("**", "")
|
||||
.replace("*", "")
|
||||
.replace("\n\n", "\n")
|
||||
.replace("'", "'")
|
||||
)
|
||||
return text_output
|
||||
|
||||
def narrator_filtering(narrator_text_input):
|
||||
processed_string = re.sub(r'\.\*\n\*', '. ', narrator_text_input)
|
||||
processed_string = (
|
||||
processed_string
|
||||
.replace("\n", " ")
|
||||
.replace('"', '"<')
|
||||
.replace('"', '"<')
|
||||
)
|
||||
processed_string = processed_string.replace('"<. *', '"< *"')
|
||||
processed_string = processed_string.replace('< *"', '< *')
|
||||
processed_string = processed_string.replace('. *', '< *')
|
||||
text_output = html.unescape(processed_string)
|
||||
return text_output
|
||||
|
||||
def combine(output_file_timestamp, output_file_name, audio_files):
|
||||
audio = np.array([])
|
||||
sample_rate = None
|
||||
try:
|
||||
for audio_file in audio_files:
|
||||
audio_data, current_sample_rate = sf.read(audio_file)
|
||||
if audio.size == 0:
|
||||
audio = audio_data
|
||||
sample_rate = current_sample_rate
|
||||
elif sample_rate == current_sample_rate:
|
||||
audio = np.concatenate((audio, audio_data))
|
||||
else:
|
||||
raise ValueError("Sample rates of input files are not consistent.")
|
||||
except Exception as e:
|
||||
# Handle exceptions (e.g., file not found, invalid audio format)
|
||||
return None, None
|
||||
if output_file_timestamp:
|
||||
timestamp = int(time.time())
|
||||
output_file_path = os.path.join(this_dir / "outputs" / f'{output_file_name}_{timestamp}_combined.wav')
|
||||
output_file_url = f'http://{params["ip_address"]}:{params["port_number"]}/audio/{output_file_name}_{timestamp}_combined.wav'
|
||||
else:
|
||||
output_file_path = os.path.join(this_dir / "outputs" / f'{output_file_name}_combined.wav')
|
||||
output_file_url = f'http://{params["ip_address"]}:{params["port_number"]}/audio/{output_file_name}.wav'
|
||||
try:
|
||||
sf.write(output_file_path, audio, samplerate=sample_rate)
|
||||
# Clean up unnecessary files
|
||||
for audio_file in audio_files:
|
||||
os.remove(audio_file)
|
||||
except Exception as e:
|
||||
# Handle exceptions (e.g., failed to write output file)
|
||||
return None, None
|
||||
return output_file_path, output_file_url
|
||||
|
||||
# Generation API (separate from text-generation-webui)
|
||||
@app.post("/api/tts-generate", response_class=JSONResponse)
|
||||
async def tts_generate(
|
||||
text_input: str = Form(...),
|
||||
text_filtering: str = Form(...),
|
||||
character_voice_gen: str = Form(...),
|
||||
narrator_enabled: bool = Form(...),
|
||||
narrator_voice_gen: str = Form(...),
|
||||
text_not_inside: str = Form(...),
|
||||
language: str = Form(...),
|
||||
output_file_name: str = Form(...),
|
||||
output_file_timestamp: bool = Form(...),
|
||||
autoplay: bool = Form(...),
|
||||
autoplay_volume: float = Form(...),
|
||||
):
|
||||
try:
|
||||
#print(f"text_filtering: {text_filtering}")
|
||||
#print(f"narrator_enabled: {narrator_enabled}")
|
||||
#print(f"text_not_inside: {text_not_inside}")
|
||||
#print(f"output_file_timestamp: {output_file_timestamp}")
|
||||
json_input_data = {
|
||||
"text_input": text_input,
|
||||
"text_filtering": text_filtering,
|
||||
"character_voice_gen": character_voice_gen,
|
||||
"narrator_enabled": narrator_enabled,
|
||||
"narrator_voice_gen": narrator_voice_gen,
|
||||
"text_not_inside": text_not_inside,
|
||||
"language": language,
|
||||
"output_file_name": output_file_name,
|
||||
"output_file_timestamp": output_file_timestamp,
|
||||
"autoplay": autoplay,
|
||||
"autoplay_volume": autoplay_volume,
|
||||
}
|
||||
JSONresult = TTSGenerator.validate_json_input(json_input_data)
|
||||
if JSONresult is None:
|
||||
pass
|
||||
else:
|
||||
return JSONResponse(content={"error": JSONresult}, status_code=400)
|
||||
if narrator_enabled:
|
||||
print("ORIGINAL UNTOUCHED STRING IS:", text_input,"\n")
|
||||
if text_filtering in ["standard", "none"]:
|
||||
cleaned_string = (
|
||||
text_input
|
||||
.replace('"', '"<')
|
||||
.replace('*', '<*')
|
||||
)
|
||||
print("STANDARD FILTERING IS NOW:", cleaned_string,"\n")
|
||||
#parts = re.split(r'\.(?<!<[*"])', cleaned_string)
|
||||
parts = re.split(r'(?<=<\*\s)|(?<=\.\s)|(?<=\."\<)|(?<=\."<)', cleaned_string)
|
||||
parts = list(filter(lambda x: x.strip(), parts))
|
||||
elif text_filtering == "html":
|
||||
cleaned_string = standard_filtering(text_input)
|
||||
cleaned_string = narrator_filtering(cleaned_string)
|
||||
print("HTML FILTERING IS NOW:", cleaned_string,"\n")
|
||||
parts = re.split(r'"|\.\*', cleaned_string)
|
||||
audio_files_all_paragraphs = []
|
||||
audio_files_paragraph = []
|
||||
for i, part in enumerate(parts):
|
||||
if len(part.strip()) <= 1:
|
||||
continue
|
||||
print("THIS IS A PART", part)
|
||||
# Figure out which type of line it is, then replace characters as necessary to avoid TTS trying to pronunce them, htmlunescape after.
|
||||
# Character will always be a < with a letter immediately after it
|
||||
if '<' in part and '<*' not in part and '< *' not in part and '< *' not in part and '< ' not in part and '< ' not in part:
|
||||
print("IF - Character\n")
|
||||
cleaned_part = html.unescape(part.replace('<', ''))
|
||||
voice_to_use = character_voice_gen
|
||||
#Narrator will always be an * or < with an * a position or two after it.
|
||||
elif '<*' in part or '< *' in part or '< *' in part or '*' in part:
|
||||
print("IF - Narrator\n")
|
||||
cleaned_part = html.unescape(part.replace('<*', '').replace('< *', '').replace('< *', '').replace('*', '').replace('<. ', ''))
|
||||
voice_to_use = narrator_voice_gen
|
||||
#If the other two dont capture it, aka, the AI gave no * or " on the line, use non_quoted_text_is aka user interface, user can choose Char or Narrator
|
||||
elif text_not_inside == "character":
|
||||
print("ELSE - CHARACTER\n")
|
||||
cleaned_part = html.unescape(part.replace('< ', '').replace('< ', '').replace('< ', ''))
|
||||
voice_to_use = character_voice_gen
|
||||
elif text_not_inside == "narrator":
|
||||
print("ELSE - NARRATOR\n")
|
||||
cleaned_part = html.unescape(part.replace('< ', '').replace('< ', '').replace('< ', ''))
|
||||
voice_to_use = narrator_voice_gen
|
||||
output_file = this_dir / "outputs" / f"{output_file_name}_{int(time.time())}_{i}.wav"
|
||||
output_file_str = output_file.as_posix()
|
||||
await generate_audio(cleaned_part, voice_to_use, language, output_file_str)
|
||||
audio_path = output_file_str
|
||||
audio_files_paragraph.append(audio_path)
|
||||
# Accumulate audio files within the paragraph
|
||||
audio_files_all_paragraphs.extend(audio_files_paragraph)
|
||||
# Combine audio files across paragraphs
|
||||
output_file_path, output_file_url = combine(output_file_timestamp, output_file_name, audio_files_all_paragraphs)
|
||||
else:
|
||||
if output_file_timestamp:
|
||||
timestamp = int(time.time())
|
||||
output_file_path = this_dir / "outputs" / f"{output_file_name}_{timestamp}.wav"
|
||||
output_file_url = f'http://{params["ip_address"]}:{params["port_number"]}/audio/{output_file_name}_{timestamp}.wav'
|
||||
else:
|
||||
output_file_path = this_dir / "outputs" / f"{output_file_name}.wav"
|
||||
output_file_url = f'http://{params["ip_address"]}:{params["port_number"]}/audio/{output_file_name}.wav'
|
||||
if text_filtering == "html":
|
||||
cleaned_string = html.unescape(standard_filtering(text_input))
|
||||
elif text_filtering == "standard":
|
||||
cleaned_string = standard_filtering(text_input)
|
||||
else:
|
||||
cleaned_string = text_input
|
||||
await generate_audio(cleaned_string, character_voice_gen, language, output_file_path)
|
||||
return JSONResponse(content={"status": "generate-success", "output_file_path": str(output_file_path), "output_file_url": str(output_file_url)}, status_code=200)
|
||||
except Exception as e:
|
||||
return JSONResponse(content={"status": "generate-failure", "error": "An error occurred"}, status_code=500)
|
||||
|
||||
#############################################################
|
||||
#### DOCUMENTATION - README ETC - PRESENTED AS A WEBPAGE ####
|
||||
#############################################################
|
||||
@@ -818,6 +1033,7 @@ simple_webpage = """
|
||||
<li><a href="#using-voice-samples">Using Voice Samples</a></li>
|
||||
<li><a href="#text-not-inside">Text Not inside</a></li>
|
||||
<li><a href="#low-vram">Low VRAM</a></li>
|
||||
<li><a href="#finetune">Finetuning (Training the model)</a></li>
|
||||
<li><a href="#deepspeed">DeepSpeed</a>
|
||||
<ul>
|
||||
<li><a href="#deepspeed-linux">DeepSpeed Setup Linux</a></li>
|
||||
@@ -967,6 +1183,82 @@ simple_webpage = """
|
||||
<p style="padding-left: 30px;"><a href="#toc">Back to top of page</a></p>
|
||||
|
||||
|
||||
<h2 id="finetune"><strong>Finetuning (Training the model)</strong></h2>
|
||||
|
||||
<p style="margin-left:40px; text-align:justify">If you have a voice that the model doesnt quite reproduce correctly, or indeed you just want to improve the reproduced voice, then finetuning is a way to train your "XTTSv2 local" model <span style="color:#3366ff">(stored in /alltalk_tts/models/xxxxx/)</span> on a specific voice. For this you will need:</p>
|
||||
|
||||
<ul style="margin-left:40px">
|
||||
<li>An Nvidia graphics card To install a few portions of the Nvidia CUDA <strong>11.8</strong> Toolkit (this will not impact text-generation-webui's cuda setup.</li>
|
||||
<li>18GB of disk space free (most of this is used temporarily)</li>
|
||||
<li>At least 2 minutes of good quality speech from your chosen speaker in mp3, wav or flacc format, in one or more files (have tested as far as 20 minutes worth of audio).</li>
|
||||
</ul>
|
||||
|
||||
<h4>How will this work/How complicated is it?</h4>
|
||||
|
||||
<p style="margin-left:40px; text-align:justify">Everything has been done to make this as simple as possible. At its simplest, you can literally just download a large chunk of audio from an interview, and tell the finetuning to strip through it, find spoken parts and build your dataset. You can literally click 4 buttons, then copy a few files and you are done. At it's more complicated end you will clean up the audio a little beforehand, but its still only 4x buttons and copying a few files.</p>
|
||||
|
||||
<h4>The audio you will use:</h4>
|
||||
|
||||
<p style="margin-left:40px; text-align:justify">I would suggest that if its in an interview format, you cut out the interviewer speaking in audacity or your chosen audio editing package. You dont have to worry about being perfect with your cuts, the finetuning Step 1 will go and find spoken audio and cut it out for you. Is there is music over the spoken parts, for best quality you would cut out those parts, though its not 100% necessary. As always, try to avoid bad quality audio with noises in it (humming sounds, hiss etc). You can try something like Audioenhancer to try clean up noisier audio. There is no need to down-sample any of the audio, all of that is handled for you. Just give the finetuning some good quality audio to work with.</p>
|
||||
|
||||
<h4>Important requirements CUDA 11.8:</h4>
|
||||
|
||||
<p style="margin-left:40px; text-align:justify">As mentioned you must have a small portion of the <span style="color:#3366ff">Nvidia CUDA Toolkit <strong>11.8</strong></span> installed. Not higher or lower versions. Specifically <strong>11.8</strong>. You do not have to uninstall any other versions, change any graphics drivers, reinstall torch or anything like that. There are instructions within the finetuning interface on doing this or you can also find them on this link <a href="https://github.com/erew123/alltalk_tts#-important-requirements-cuda-118">here</a></p>
|
||||
|
||||
<h4>Starting Finetuning:</h4>
|
||||
|
||||
<p style="margin-left:40px"><strong>Ensure</strong> you have followed the instructions on setting up the Nvidia CUDA Toolkit 11.8 <a href="https://github.com/erew123/alltalk_tts#-important-requirements-cuda-118">here</a> or the below procedure will fail.</p>
|
||||
|
||||
<p style="margin-left:40px">The below instructions are also available online <a href="https://github.com/erew123/alltalk_tts#-finetuning-a-model">here</a></p>
|
||||
|
||||
<ol>
|
||||
<li>
|
||||
<p>Close all other applications that are using your GPU/VRAM and copy your audio samples into:<br />
|
||||
<br />
|
||||
<span style="color:#3366ff">/alltalk_tts/finetune/put-voice-samples-in-here/</span></p>
|
||||
</li>
|
||||
<li>
|
||||
<p>In a command prompt/terminal window you need to move into your Text generation webUI folder:<br />
|
||||
<br />
|
||||
<span style="color:#3366ff">cd text-generation-webui</span></p>
|
||||
</li>
|
||||
<li>
|
||||
<p>Start the Text generation webUI Python environment for your OS:<br />
|
||||
<br />
|
||||
<span style="color:#3366ff">cmd_windows.bat, ./cmd_linux.sh, cmd_macos.sh or cmd_wsl.bat</span></p>
|
||||
</li>
|
||||
<li>
|
||||
<p>You can double check your search path environment still works correctly with <span style="color:#3366ff">nvcc --version</span>. It should report back <span style="color:#3366ff">11.8</span>:<br />
|
||||
<br />
|
||||
<span style="color:#3366ff">Cuda compilation tools, release 11.8.</span></p>
|
||||
</li>
|
||||
<li>
|
||||
<p>Move into your extensions folder:<br />
|
||||
<br />
|
||||
<span style="color:#3366ff">cd extensions</span></p>
|
||||
</li>
|
||||
<li>
|
||||
<p>Move into the <span style="color:#3366ff">alltalk_tts</span> folder:<br />
|
||||
<br />
|
||||
<span style="color:#3366ff">cd alltalk_tts</span></p>
|
||||
</li>
|
||||
<li>
|
||||
<p>Install the finetune requirements file: <span style="color:#3366ff">pip install -r requirements_finetune.txt</span></p>
|
||||
</li>
|
||||
<li>
|
||||
<p>Type<span style="color:#3366ff"> python finetune.py</span> and it should start up.</p>
|
||||
</li>
|
||||
<li>
|
||||
<p>Follow the on-screen instructions when the web interface starts up.</p>
|
||||
</li>
|
||||
<li>
|
||||
<p>When you have finished finetuning, the final tab will tell you what to do with your files and how to move your newly trained model to the correct location on disk.</p>
|
||||
</li>
|
||||
</ol>
|
||||
|
||||
<p><a href="#toc">Back to top of page</a></p>
|
||||
|
||||
|
||||
<h2 id="deepspeed"><strong>DeepSpeed</strong></h2>
|
||||
<p style="padding-left: 30px; text-align: justify;">DeepSpeed provides a 2x-3x speed boost for Text-to-Speech and AI tasks. It's all about making AI and TTS happen faster and more efficiently.</p>
|
||||
<ul>
|
||||
@@ -1258,31 +1550,129 @@ simple_webpage = """
|
||||
</table>
|
||||
|
||||
<h2 id="curl-commands"><strong>JSON calls & CURL Commands</strong></h2>
|
||||
<h4>Generating TTS - Linux</h4>
|
||||
<p style="padding-left: 30px;"><span style="color: #3366ff;">curl -X POST -H "Content-Type: application/json" -d '{"text": "<span style="color: #ff9900;">This is text to generate as TTS</span>","voice": "<span style="color: #ff9900;">female_01.wav</span>", "language": "<span style="color: #ff9900;">en</span>", "output_file": "<span style="color: #ff9900;">outputfile.wav</span>"}' "http://127.0.0.1:7851/api/generate"</span></p>
|
||||
<h4>Generating TTS - Windows</h4>
|
||||
<p style="padding-left: 30px;"><span style="color: #3366ff;">curl -X POST -H "Content-Type: application/json" -d "{\"text\": \"<span style="color: #ff9900;">This is text to generate as TTS</span>\", \"voice\": \"<span style="color: #ff9900;">female_01.wav</span>\", \"language\": \"<span style="color: #ff9900;">en</span>\", \"output_file\": \"<span style="color: #ff9900;">outputfile.wav</span>\"}" http://127.0.0.1:7851/api/generate</span></p>
|
||||
<h4>Generating TTS - Notes</h4>
|
||||
<p style="padding-left: 30px;">Replace <span style="color: #ff9900;">This is text to generate as TTS</span> with whatever you want it to say. <span style="color: #ff9900;">female_01.wav</span> with the voice sample you want to use. <span style="color: #ff9900;">output_file.wav</span> with the file name you want it to create.</p>
|
||||
<p style="padding-left: 30px;">JSON return <span style="color: #339966;">{"status":"generate-success","data":{"audio_path":"outputfile.wav"}}</span></p>
|
||||
|
||||
<h3>Overview</h3>
|
||||
<p style="margin-left: 40px; text-align: justify;">The Text-to-Speech (TTS) Generation API allows you to generate speech from text input using various configuration options. This API supports both character and narrator voices, providing flexibility for creating dynamic and engaging audio content.</p>
|
||||
<h3>TTS Generation Endpoint</h3>
|
||||
<ul>
|
||||
<li><strong>URL</strong>: <span style="color: #3366ff;">http://127.0.0.1:7851/api/tts-generate</span></li>
|
||||
<li><strong>Method</strong>: <span style="color: #3366ff;">POST</span></li>
|
||||
<li><strong>Content-Type</strong>: <span style="color: #3366ff;">application/x-www-form-urlencoded</span></li>
|
||||
</ul>
|
||||
<h3>Example command line</h3>
|
||||
<p style="padding-left: 30px;">Standard TTS speech Example (standard text) generating a time-stamped file</p>
|
||||
<p style="padding-left: 30px;"><span style="color: #3366ff;">curl -X POST "http://127.0.0.1:7851/api/tts-generate" -d "text_input=</span><span style="color: #ff9900;">All of this is text spoken by the character. This is text not inside quotes, though that doesnt matter in the slightest</span><span style="color: #3366ff;">" -d "text_filtering=</span><span style="color: #339966;">standard</span><span style="color: #3366ff;">" -d "character_voice_gen=</span><span style="color: #ff9900;">female_01.wav</span><span style="color: #3366ff;">" -d "narrator_enabled=</span><span style="color: #339966;">false</span><span style="color: #3366ff;">" -d "narrator_voice_gen=male_01.wav" -d "text_not_inside=character" -d "language=en" -d "output_file_name=</span><span style="color: #ff9900;">myoutputfile</span><span style="color: #3366ff;">" -d "output_file_timestamp=</span><span style="color: #ff9900;">true</span><span style="color: #3366ff;">" -d "autoplay=true" -d "autoplay_volume=0.8"</span></p>
|
||||
<p style="padding-left: 30px;">Narrator Example (standard text) generating a time-stamped file</p>
|
||||
<p style="padding-left: 30px;"><span style="color: #3366ff;">curl -X POST "http://127.0.0.1:7851/api/tts-generate" -d "text_input=</span><span style="color: #ff9900;">*This is text spoken by the narrator* \"This is text spoken by the character\". This is text not inside quotes.</span><span style="color: #3366ff;">" -d "text_filtering=</span><span style="color: #339966;">standard</span><span style="color: #3366ff;">" -d "character_voice_gen=</span><span style="color: #ff9900;">female_01.wav</span><span style="color: #3366ff;">" -d "narrator_enabled=</span><span style="color: #339966;">true</span><span style="color: #3366ff;">" -d "narrator_voice_gen=</span><span style="color: #339966;">male_01.wav</span><span style="color: #3366ff;">" -d "text_not_inside=</span><span style="color: #339966;">character</span><span style="color: #3366ff;">" -d "language=en" -d "output_file_name=</span><span style="color: #ff9900;">myoutputfile</span><span style="color: #3366ff;">" -d "output_file_timestamp=</span><span style="color: #ff9900;">true</span><span style="color: #3366ff;">" -d "autoplay=true" -d "autoplay_volume=0.8"</span></p>
|
||||
<p style="padding-left: 30px; text-align: justify;"><strong><span style="color: #ff0000;">Note</span></strong> that if your text that needs to be generated contains double quotes you will need to escape them with \" (Please see the narrator example).</p>
|
||||
<h3> Request Parameters</h3>
|
||||
<ul>
|
||||
<li>
|
||||
<p style="text-align: justify;"><strong>text_input</strong>: The text you want the TTS engine to produce. Use escaped double quotes for character speech and asterisks for narrator speech if using the narrator function. Example:</p>
|
||||
<p>-<span style="color: #3366ff;">d "text_input=*This is text spoken by the narrator* \"This is text spoken by the character\". This is text not inside quotes." </span></p>
|
||||
</li>
|
||||
<li>
|
||||
<p><strong>text_filtering</strong>: Filter for text. Options:</p>
|
||||
<ul>
|
||||
<li style="text-align: justify;"><span style="color: #3366ff;">none</span> No filtering. Whatever is sent will go over to the TTS engine as raw text, which may result in some odd sounds with some special characters.</li>
|
||||
<li><span style="color: #3366ff;">standard</span> Human-readable text and a basic level of filtering, just to clean up some special characters.</li>
|
||||
<li><span style="color: #3366ff;">html</span> HTML content. Where you are using HTML entity's like &quot;</li>
|
||||
</ul>
|
||||
<p><span style="color: #3366ff;">-d "text_filtering=none" </span><br /><span style="color: #3366ff;">-d "text_filtering=standard" </span><br /><span style="color: #3366ff;">-d "text_filtering=html"</span></p>
|
||||
<p><strong>Example:</strong></p>
|
||||
<ul>
|
||||
<li>Standard Example: <span style="color: #3366ff;">*This is text spoken by the narrator* "This is text spoken by the character" This is text not inside quotes.</span></li>
|
||||
<li>HTML Example:<span style="color: #3366ff;"> *This is text spoken by the narrator* &quot;This is text spoken by the character&quot; This is text not inside quotes.</span></li>
|
||||
<li>None will just pass whatever characters/text you send at it.</li>
|
||||
</ul>
|
||||
</li>
|
||||
<li>
|
||||
<p><strong>character_voice_gen</strong>: The WAV file name for the character's voice.</p>
|
||||
<p><span style="color: #3366ff;">-d "character_voice_gen=female_01.wav" </span></p>
|
||||
</li>
|
||||
<li>
|
||||
<p><strong>narrator_enabled</strong>: Enable or disable the narrator function. If true, minimum text filtering is set to standard. Anything between double quotes is considered the character's speech, and anything between asterisks is considered the narrator's speech.</p>
|
||||
<p><span style="color: #3366ff;">-d "narrator_enabled=true"<br />-d "narrator_enabled=false" </span><span style="font-family: Verdana, Arial, Helvetica, sans-serif;"> </span></p>
|
||||
</li>
|
||||
<li>
|
||||
<p><strong>narrator_voice_gen</strong>: The WAV file name for the narrator's voice.</p>
|
||||
<p><span style="color: #3366ff;">-d "narrator_voice_gen=male_01.wav"</span></p>
|
||||
</li>
|
||||
<li>
|
||||
<p><strong>text_not_inside</strong>: Specify the handling of lines not inside double quotes or asterisks, for the narrator feature. Options:</p>
|
||||
<ul>
|
||||
<li><span style="color: #3366ff;">character</span>: Treat as character speech.</li>
|
||||
<li><span style="color: #3366ff;">narrator</span>: Treat as narrator speech.</li>
|
||||
</ul>
|
||||
<p><span style="color: #3366ff;">-d "text_not_inside=character" </span><br /><span style="color: #3366ff;">-d "text_not_inside=narrator"</span></p>
|
||||
</li>
|
||||
<li>
|
||||
<p><strong>language</strong>: Choose the language for TTS. Options:</p>
|
||||
<ul>
|
||||
<li><span style="color: #3366ff;">ar</span> Arabic</li>
|
||||
<li><span style="color: #3366ff;">zh-cn</span> Chinese (Simplified)</li>
|
||||
<li><span style="color: #3366ff;">cs</span> Czech</li>
|
||||
<li><span style="color: #3366ff;">nl</span> Dutch</li>
|
||||
<li><span style="color: #3366ff;">en</span> English</li>
|
||||
<li><span style="color: #3366ff;">fr</span> French</li>
|
||||
<li><span style="color: #3366ff;">de</span> German</li>
|
||||
<li><span style="color: #3366ff;">hu</span> Hungarian</li>
|
||||
<li><span style="color: #3366ff;">it</span> Italian</li>
|
||||
<li><span style="color: #3366ff;">ja</span> Japanese</li>
|
||||
<li><span style="color: #3366ff;">ko</span> Korean</li>
|
||||
<li><span style="color: #3366ff;">pl</span> Polish</li>
|
||||
<li><span style="color: #3366ff;">pt</span> Portuguese</li>
|
||||
<li><span style="color: #3366ff;">ru </span>Russian</li>
|
||||
<li><span style="color: #3366ff;">es</span> Spanish</li>
|
||||
<li><span style="color: #3366ff;">tr</span> Turkish</li>
|
||||
</ul>
|
||||
<p><span style="color: #3366ff;">-d "language=en"</span></p>
|
||||
</li>
|
||||
<li>
|
||||
<p><strong>output_file_name</strong>: The name of the output file (excluding the .wav extension).</p>
|
||||
<p><span style="color: #3366ff;">-d "output_file_name=myoutputfile" </span></p>
|
||||
</li>
|
||||
<li>
|
||||
<p><strong>output_file_timestamp</strong>: Add a timestamp to the output file name. If true, each file will have a unique timestamp; otherwise, the same file name will be overwritten each time you generate TTS.</p>
|
||||
<p><span style="color: #3366ff;">-d "output_file_timestamp=true" <br />-d "output_file_timestamp=false" </span></p>
|
||||
</li>
|
||||
<li>
|
||||
<p><strong>autoplay</strong>: <span style="color: #ff0000;">Feature not yet available</span>. Enable or disable autoplay. Still needs to be specified in the JSON request.</p>
|
||||
<p><span style="color: #3366ff;">-d "autoplay=true" <br />-d "autoplay=false"</span></p>
|
||||
</li>
|
||||
<li>
|
||||
<p><strong>autoplay_volume</strong>: <span style="color: #ff0000;">Feature not yet available</span>. Set the autoplay volume. Should be between 0.1 and 1.0. Still needs to be specified in the JSON request.</p>
|
||||
<p><span style="color: #3366ff;">-d "autoplay_volume=0.8" </span></p>
|
||||
</li>
|
||||
</ul>
|
||||
<h3>TTS Generation Response</h3>
|
||||
<p style="padding-left: 30px;">The API returns a JSON object with the following properties:</p>
|
||||
<ul>
|
||||
<li><span style="color: #3366ff;"><strong>status</strong></span> Indicates whether the generation was successful (<span style="color: #3366ff;">generate-success</span>) or failed (<span style="color: #3366ff;">generate-failure</span>).</li>
|
||||
<li><span style="color: #3366ff;"><strong>output_file_path</strong></span> The on-disk location of the generated WAV file.</li>
|
||||
<li><span style="color: #3366ff;"><strong>output_file_url</strong></span> The HTTP location for accessing the generated WAV file.</li>
|
||||
</ul>
|
||||
<p><strong>Example JSON TTS Generation Response:</strong></p>
|
||||
<p style="padding-left: 30px;"><span style="color: #339966;">{"status": "generate-success", "output_file_path": "C:\\text-generation-webui\\extensions\\alltalk_tts\\outputs\\myoutputfile_1703149973.wav", "output_file_url": "http://127.0.0.1:7851/audio/myoutputfile_1703149973.wav"}</span></p>
|
||||
|
||||
|
||||
<h4>Switching Model</h4>
|
||||
<p style="padding-left: 30px;"><span style="color: #3366ff;">curl -X POST "http://127.0.0.1:7851/api/reload?tts_method=API%20Local"</span><br /><span style="color: #3366ff;"> curl -X POST "http://127.0.0.1:7851/api/reload?tts_method=API%20TTS"</span><br /><span style="color: #3366ff;"> curl -X POST "http://127.0.0.1:7851/api/reload?tts_method=XTTSv2%20Local"</span></p>
|
||||
<p style="padding-left: 30px;">Switch between the 3 models respectively.</p>
|
||||
<p style="padding-left: 30px;">JSON return <span style="color: #339966;">{"status": "model-success"}</span></p>
|
||||
<h4>Switch DeepSpeed</h4>
|
||||
<p style="padding-left: 30px;"><span style="color: #3366ff;">curl -X POST -H "Content-Type: application/json" "http://127.0.0.1:7851/api/deepspeed?new_deepspeed_value=True"</span></p>
|
||||
<p style="padding-left: 30px;"><span style="color: #3366ff;">curl -X POST "http://127.0.0.1:7851/api/deepspeed?new_deepspeed_value=True"</span></p>
|
||||
<p style="padding-left: 30px;">Replace True with False to disable DeepSpeed mode.</p>
|
||||
<p style="padding-left: 30px;">JSON return <span style="color: #339966;">{"status": "deepspeed-success"}</span></p>
|
||||
<h4>Switching Low VRAM</h4>
|
||||
<p style="padding-left: 30px;"><span style="color: #3366ff;">curl -X POST -H "Content-Type: application/json" "http://127.0.0.1:7851/api/lowvramsetting?new_low_vram_value=True"</span></p>
|
||||
<p style="padding-left: 30px;"><span style="color: #3366ff;">curl -X POST "http://127.0.0.1:7851/api/lowvramsetting?new_low_vram_value=True"</span></p>
|
||||
<p style="padding-left: 30px;">Replace True with False to disable Low VRAM mode.</p>
|
||||
<p style="padding-left: 30px;">JSON return <span style="color: #339966;">{"status": "lowvram-success"}</span></p>
|
||||
<p><a href="#toc">Back to top of page<br /></a></p>
|
||||
|
||||
<h2 id="debugging-and-tts-generation-information"><strong>Debugging and TTS Generation Information</strong></h2>
|
||||
<p style="padding-left: 30px; text-align: justify;">Command line outputs are more verbose to assist in understanding backend processes and debugging.</p>
|
||||
<p style="padding-left: 30px; text-align: justify;">Its possible during startup you can get a warning message such as: <br /><br /><span style="color: black;"> [AllTalk Startup]</span> <span style="color: #ff0000;">Warning</span> <span style="color: black;">TTS Subprocess has NOT started up yet, Will keep trying for 60 seconds maximum</span> <br /><br />This is normal behavior if the subprocess is taking a while to start, however, if there is an issue starting the subprocess, you may see multiples of this message and an it will time out after 60 seconds, resulting in the TTS engine not starting. Its likely that you are not in the correct python environment or one that has a TTS engine inside, if this happens, though the system will output a warning about that ahead of this message</p>
|
||||
<p style="padding-left: 30px; text-align: justify;">Its possible during startup you can get a warning message such as: <br /><br /><span style="color: black;"> [AllTalk Startup]</span> <span style="color: #ff0000;">Warning</span> <span style="color: black;">TTS Subprocess has NOT started up yet, Will keep trying for 80 seconds maximum</span> <br /><br />This is normal behavior if the subprocess is taking a while to start, however, if there is an issue starting the subprocess, you may see multiples of this message and an it will time out after 80 seconds, resulting in the TTS engine not starting. Its likely that you are not in the correct python environment or one that has a TTS engine inside, if this happens, though the system will output a warning about that ahead of this message</p>
|
||||
<p style="padding-left: 30px; text-align: justify;">Typically the command line console will output any warning or error messages. If you need to reset your default configuation, the settings are all listed above in the configuration details.</p>
|
||||
<p style="padding-left: 30px;"><a href="#toc">Back to top of page</a></p>
|
||||
<h2 id="references"><strong>Thanks & References</strong></h2>
|
||||
|
||||
Reference in new issue
Block a user