Compare commits

...
41 Commits
Author SHA1 Message Date
erew123 97f0d6441a Improved links in Documentation
Added links to the minor bugs/known issues on gthub
2023-12-26 07:30:30 +00:00
erew123 2ea6cb6304 Update README.md 2023-12-26 06:23:26 +00:00
erew123 5bed68a963 minor update to standard voice naming. 2023-12-25 21:53:40 +00:00
erew123 be9862d0b0 Update README.md 2023-12-25 21:15:54 +00:00
erew123 f5a811f0f8 Update README.md 2023-12-25 21:11:02 +00:00
erew123 dc9d343115 Update uuid to avoid possible race condition
file naming on short sentences could possibly have a race condition. set uuid as opposed to timestamp to avoid this outcome.
2023-12-25 21:09:39 +00:00
erew123 c92aa435e5 Update README.md 2023-12-25 09:37:39 +00:00
erew123 1f425e6bac Update README.md 2023-12-25 06:04:35 +00:00
erew123 0854544a47 Update README.md 2023-12-25 05:45:05 +00:00
erew123 52a7922e98 Add files via upload 2023-12-25 02:05:05 +00:00
erew123 558583b16f Update README.md 2023-12-24 20:37:59 +00:00
erew123 993d065368 Few minor updates
Couple of documentation spelling mistakes
Set environment variable
2023-12-24 18:01:27 +00:00
erew123 d7376fa8ef Update README.md 2023-12-24 17:01:33 +00:00
erew123 2b175b13c4 update to find mp3 and flac
Forgot it was only looking for wav's!
2023-12-24 15:56:44 +00:00
erew123 da0445489e Update requirements_other.txt 2023-12-24 14:17:07 +00:00
erew123 e4eaa760b9 Update requirements_nvidia.txt 2023-12-24 14:16:46 +00:00
erew123 de4017f385 Update README.md 2023-12-24 14:07:22 +00:00
erew123 f2119aa0f7 Merge pull request #14 from erew123/dev
New Finetuning of your models
New API endpoint
Full documentation (online and built in)
A couple of minor file updates.
Small bug fix on file generation in instruct mode with no character name available (thanks to @danielwburch)
2023-12-24 13:10:18 +00:00
erew123 bb81f3f874 Update README.md 2023-12-24 13:03:26 +00:00
erew123 fae15eecb1 minor spelling mistake
updating minor spelling mistake.
2023-12-24 12:54:08 +00:00
erew123 6edcc2125b create finetune directory 2023-12-24 12:35:02 +00:00
erew123 d26909a1d6 All the updates for Finetuning
Additional files to download
Updated documentation
Updated requirements files
2023-12-24 12:32:10 +00:00
erew123 3be6b842c4 Update README.md 2023-12-24 10:24:03 +00:00
erew123 e10dc700c3 Update README.md 2023-12-24 10:20:03 +00:00
erew123 e8e21f949c Update README.md
Detail API Suite
2023-12-24 10:00:08 +00:00
erew123 e68a7d72f2 Update README.md 2023-12-24 08:49:14 +00:00
erew123 fd26db8e8a Updated model + added finetuning requirements file
Updated modeldownload.json with finetuning requirements inside. 
New requirements_finetune.txt file.
2023-12-24 08:35:33 +00:00
erew123 396f1290e7 Update README.md 2023-12-24 08:30:32 +00:00
erew123 5bc7dd8692 Update README.md
added finetuning details
2023-12-24 08:29:06 +00:00
erew123 b5b68ad51f Update script.py
updated timeout
2023-12-23 00:16:38 +00:00
erew123 e49ceed40c Merge pull request #13 from danielwburch/refactor_output_filename_2
Refactor output file naming in script.py
2023-12-21 18:03:32 +00:00
Daniel Burch 598aec7f52 Refactor output file naming in script.py 2023-12-21 10:17:41 -06:00
erew123 f25d2627cd updated timeout to 80 seconds
Much older systems with low system ram are taking 70 seconds to start up (so Im told). Ive added an extra 20 seconds onto the 60 seconds for subprocess start-up.

documentation updated too.
2023-12-21 12:47:53 +00:00
erew123 3e6f1bace3 Update README.md
updated docs based on 80 second timeout for older systems
2023-12-21 12:45:58 +00:00
erew123 fb851ff3f0 Update README.md 2023-12-21 12:32:52 +00:00
erew123 92bf8f5de8 Update README.md 2023-12-21 12:31:56 +00:00
erew123 8f318892c2 New API endpoint
adds the /api/tts-generation endpoint & documentation.
2023-12-21 11:32:59 +00:00
erew123 28a62ed39a Merge pull request #12 from erew123/revert-9-refactor_output_filename_setting
Revert "Refactor output file naming in script.py"
2023-12-21 06:27:23 +00:00
erew123 4c08927ed9 Revert "Refactor output file naming in script.py" 2023-12-21 06:26:32 +00:00
erew123 a93da39711 Merge pull request #9 from danielwburch/refactor_output_filename_setting
Refactor output file naming in script.py
2023-12-21 06:14:03 +00:00
Daniel Burch 557f4b80b6 Refactor output file naming in script.py 2023-12-20 15:07:28 -06:00
9 changed files with 1687 additions and 52 deletions

No files matched your search

+232 -18
View File
@@ -2,14 +2,16 @@
AllTalk is an updated version of the Coqui_tts extension for Text Generation web UI. Features include:
- **Custom Start-up Settings:** Adjust your default start-up settings. [Screenshot](https://github.com/erew123/alltalk_tts#screenshots)
- **Narrarator:** Use different voices for main character and narration. [Example Narration](https://vocaroo.com/18fYWVxiQpk1)
- **Narrarator:** Use different voices for main character and narration. [Example Narration](https://vocaroo.com/18nrv7FR6wuA)
- **Low VRAM mode:** Great for people with small GPU memory or if your VRAM is filled by your LLM. [Screenshot](https://github.com/erew123/alltalk_tts#screenshots)
- **DeepSpeed:** A 3-4x performance boost generating TTS. [DeepSpeed Windows/Linux Instructions](https://github.com/erew123/alltalk_tts?tab=readme-ov-file#deepspeed-installation-options) [Screenshot](https://github.com/erew123/alltalk_tts/assets/35898566/548619c8-5f1b-47d0-a73d-54d2fee3f3db)
- **Local/Custom models:** Use any of the XTTSv2 models (API Local and XTTSv2 Local).
- **Optional wav file maintenance:** Configurable deletion of old output wav files. [Screenshot](https://github.com/erew123/alltalk_tts#screenshots)
- **Finetuning** Train the model specifically on a voice of your choosing for better reproduction.
- **Documentation:** Fully documented with a built in webpage. [Screenshot](https://github.com/erew123/alltalk_tts#screenshots)
- **Console output** Clear command line output for any warnings or issues.
- **Standalone/3rd Party support via JSON calls** Can be used with 3rd party applications via JSON calls.
- **API Suite and 3rd Party support via JSON calls** Can be used with 3rd party applications via JSON calls.
- **Can be run as a standalone app** Not just inside of text-generation-webui.
## Index
@@ -17,10 +19,16 @@ AllTalk is an updated version of the Coqui_tts extension for Text Generation web
- 🟪 [Updating & problems with updating](https://github.com/erew123/alltalk_tts?#-updating)
- 🟫 [Screenshots](https://github.com/erew123/alltalk_tts#-screenshots)
- 🟨 [Help with problems](https://github.com/erew123/alltalk_tts?#-help-with-problems)
- ⚫ [Finetuning a model](https://github.com/erew123/alltalk_tts?#-finetuning-a-model)
- 🔵🟢🟡 [DeepSpeed Installation (Windows & Linux)](https://github.com/erew123/alltalk_tts?#-deepspeed-installation-options)
- 🟠 [API Suite and JSON-CURL](https://github.com/erew123/alltalk_tts?#-api-suite-and-json-curl)
- 🔴 [Future to-do list & Upcoming updates](https://github.com/erew123/alltalk_tts?#-future-to-do-list)
#### Updates
Dec 25th - Applied a small update to avoid a possible race condition on file naming with small sentences when generating narrator/character speech. Please update if you are experiencing missing small bits of missing audio on narrator/character.
####     🔄 **Minor updates/bug fixes list** can be found [here](https://github.com/erew123/alltalk_tts/issues/25)
The latest build (13 Dec 2023) has had the entire text filtering engine and narration engine rebuilt from scratch. It's highly complicated how its actually working, but the end result it a much clearer TTS output and much better control over the narrator option and how to handle text that isnt within quotes or asterisks. It does however mean you need to ensure your character card is set up correctly if using the narrator function. Details are below in the installation notes.
DeepSpeed **v11.2** can be installed within the **default text-generation-webui Python 3.11 environment**. Installs in custom Python environments are possible, but can be more complicated. Instructions [here](https://github.com/erew123/alltalk_tts##deepspeed-installation-options) (or scroll down).
@@ -43,9 +51,9 @@ This has been tested on the current Dec 2023 release of Text generation webUI. I
5) Move into the **alltalk_tts** folder:<br><br>
`cd alltalk_tts`
6) Install the requirements:<br><br>
*Nvidia graphics card machines* - `pip install -r requirements_nvidia.txt`<br><br>
*Other machines (mac, amd etc)* - `pip install -r requirements_other.txt`
6) Install the requirements that are correct for your machine:<br><br>
**Nvidia graphics card machines** - `pip install -r requirements_nvidia.txt`<br><br>
**Other machines (mac, amd etc)** - `pip install -r requirements_other.txt`
7) **(Optional DeepSpeed)** If you have an Nvidia Graphics card on a system running Linux or Windows and wish to use **DeepSpeed** please follow these instructions [here](https://github.com/erew123/alltalk_tts?#-deepspeed-installation-options). **However**, I would highly reccommend before you install DeepSpeed, you start text-generation-webui up, confirm AllTalk starts correctly and everything is working, as DeepSpeed can add another layer of complications troubleshooting any potential start-up issues. If necessary you can `pip uninstall deepspeed`.
@@ -81,9 +89,9 @@ This is pretty much a repeat of the installation process.
3) At the command prompt/terminal, type:<br><br>
`git pull`
4) Install the requirements:<br><br>
*Nvidia graphics card machines* - `pip install -r requirements_nvidia.txt`<br><br>
*Other machines (mac, amd etc)* - `pip install -r requirements_other.txt`
4) Install the correct requirements for your machine:<br><br>
**Nvidia graphics card machines** - `pip install -r requirements_nvidia.txt`<br><br>
**Other machines (mac, amd etc)** - `pip install -r requirements_other.txt`
#### 🟪 Updating "git pull" error
@@ -123,9 +131,9 @@ This will download a fresh installation.
3) Move into the **alltalk_tts** folder:<br><br>
`cd alltalk_tts`
4) Install the requirements:<br><br>
*Nvidia graphics card machines* - `pip install -r requirements_nvidia.txt`<br><br>
*Other machines (mac, amd etc)* - `pip install -r requirements_other.txt`
4) Install the correct requirements for your machine:<br><br>
**Nvidia graphics card machines** - `pip install -r requirements_nvidia.txt`<br><br>
**Other machines (mac, amd etc)** - `pip install -r requirements_other.txt`
5) Before starting it up, copy/merge the `models`, `voices` and `outputs` folders over from the `alltalk_tts.old` folder to the newly created `alltalk_tts` folder. This will keep your voices history and also stop it re-downloading the model again.
@@ -135,11 +143,13 @@ Assuming its all working fine and you are happy, you can delete the old alltalk_
</details>
## 🟫 Screenshots
|![image](https://github.com/erew123/alltalk_tts/assets/35898566/4ca9b4c7-60ce-4ac6-82e5-fd1989b84644) | ![image](https://github.com/erew123/alltalk_tts/assets/35898566/b0e13dba-c6b1-4ab7-845d-244ac1158330) |
|![image](https://github.com/erew123/alltalk_tts/assets/35898566/4ca9b4c7-60ce-4ac6-82e5-fd1989b84644) | ![image](https://github.com/erew123/alltalk_tts/assets/35898566/548619c8-5f1b-47d0-a73d-54d2fee3f3db) |
|:---:|:---:|
|![image](https://github.com/erew123/alltalk_tts/assets/35898566/548619c8-5f1b-47d0-a73d-54d2fee3f3db) | ![image](https://github.com/erew123/alltalk_tts/assets/35898566/e35e987c-543a-486b-b4fb-ee6ebe6f59c6) |
|![image](https://github.com/erew123/alltalk_tts/assets/35898566/b0e13dba-c6b1-4ab7-845d-244ac1158330) | ![image](https://github.com/erew123/alltalk_tts/assets/35898566/e35e987c-543a-486b-b4fb-ee6ebe6f59c6) |
|![image](https://github.com/erew123/alltalk_tts/assets/35898566/68edc968-bdd2-4f98-9d56-b9ca40decf26) | ![image](https://github.com/erew123/alltalk_tts/assets/35898566/eb5e617b-2345-418d-8267-a9d30f705a10) |
## 🟨 Help with problems
#### &nbsp;&nbsp;&nbsp;&nbsp; 🔄 **Minor updates/bug fixes list** can be found [here](https://github.com/erew123/alltalk_tts/issues/25)
#### 🟨 How to make a diagnostics report file
<details>
@@ -160,7 +170,7 @@ Assuming its all working fine and you are happy, you can delete the old alltalk_
<br><br></details>
#### 🟨 [AllTalk Startup] Warning TTS Subprocess has NOT started up yet, Will keep trying for 60 seconds maximum. Please wait. It times out after 60 seconds.
#### 🟨 [AllTalk Startup] Warning TTS Subprocess has NOT started up yet, Will keep trying for 120 seconds maximum. Please wait. It times out after 120 seconds.
<details>
<summary>Click to expand</summary><br>
@@ -169,9 +179,11 @@ Assuming its all working fine and you are happy, you can delete the old alltalk_
**A)** Its trying to load the voice model into your graphics card VRAM (assuming you have a Nvidia Graphics card, otherwise its your system RAM)<br>
**B)** Its trying to start up the mini-webserver and send the "ready" signal back to the main process.
Before giving other possibilities a go, some people with **old machines** are finding their startup times are **very** slow 2-3 minutes. Ive extended the allowed time within the script from 1 minute to 2 minutes. **If you have an older machine** and wish to try extending this further, you can do so by editing `script.py` and changing line 251 `timeout = 120` changing the timeout to a larger value e.g `timeout = 240` (4 minutes).
**Note:** If you need to create a support ticket, please create a `diagnostics.log` report file to submit with a support request. Details on doing this are above.
Possibilities for this issue are:
Other possibilities for this issue are:
1) You are starting AllTalk in both your `CMD FLAG.txt` and `settings.yaml` file. The `CMD FLAG.txt` you would have manually edited and the `settings.yaml` is the one you change and save in the `session` tab of text-generation-webui and you can `Save UI defaults to settings.yaml`. Please only have one of those two starting up AllTalk.
@@ -196,6 +208,19 @@ Possibilities for this issue are:
11) You have built yourself a custom Python environment and something is funky with it. This is very hard to diagnose as its not a standard environment. You may want to updating text-generation-webui and re installing its requirements file (whichever one you use that comes down with text-generation-webui).
</details>
#### 🟨 I think AllTalks requirements file has installed something another extension doesn't like
<details>
<summary>Click to expand</summary><br>
Ive paid very close attention to **not** impact what Text-generation-webui is requesting on a factory install. This is one of the requirements of submitting an extension to Text-generation-webui. If you want to look at a comparison of a factory fresh text-generation-webui installed packages (with cuda 12.1, though AllTalk's requirements were set on cuda 11.8) you can find that comparison [here](https://github.com/erew123/alltalk_tts/issues/23). This comparison shows that AllTalk is requesting the same package version numbers as Text-generation-webui or even lower version numbers (meaning AllTalk will not update them to a later version). What other extensions do, I cant really account for that.
I will note that the TTS engine downgrades Pandas data validator to 1.5.3 though its unlikely to cause any issues. You can upgrade it back to text-generation-webui default (december 2023) with `pip install pandas==2.1.4` when inside of the python environment. I have noticed no ill effects from it being a lower or higher version, as far as AllTalk goes. This is also the same behaviour as the Coqui_tts extension that comes with Text-generation-webui.
Other people are reporting issues with extensions not starting with errors about Pydantic e.g. ```pydantic.errors.PydanticImportError: BaseSettings` has been moved to the pydantic-settings package. See https://docs.pydantic.dev/2.5/migration/#basesettings-has-moved-to-pydantic-settings for more details.```
Im not sure if the Pydantic version has been recently updated by the Text-generation-webui installer, but this is nothing to do with AllTalk. The other extension you are having an issue with, need to be updated to work with Pydantic 2.5.x. AllTalk was updated in mid december to work with 2.5.x. I am not specifically condoning doing this, as it may have other knock on effects, but within the text-gen Python environment, you can use `pip install pydantic==2.5.0` or `pip install pydantic==1.10.13` to change the version of Pydantic installed.
</details>
#### 🟨 I activated DeepSpeed in the settings page, but I didnt install DeepSpeed yet and now I have issues starting up
<details>
<summary>Click to expand</summary><br>
@@ -230,7 +255,64 @@ I would suggest following [Problems Updating](https://github.com/erew123/alltalk
As far as I am aware, these are to do with the chrome browser the gradio text-generation-webui in some way. I raised an issue about this on the text-generation-webui [here](https://github.com/oobabooga/text-generation-webui/issues/4788) where you can see that AllTalk is not loaded and the messages persist. Either way, this is more a warning than an actual issue, so shouldnt affect any functionality of either AllTalk or text-generation-webui, they are more just an annoyance.
</details>
## ⚫ Finetuning a model
If you have a voice that the model doesnt quite reproduce correctly, or indeed you just want to improve the reproduced voice, then finetuning is a way to train your "XTTSv2 local" model **(stored in `/alltalk_tts/models/xxxxx/`)** on a specific voice. For this you will need:
- An Nvidia graphics card
- To install a few portions of the Nvidia CUDA 11.8 Toolkit (this will not impact text-generation-webui's cuda setup.
- 18GB of disk space free (most of this is used temporarily)
- At least 2 minutes of good quality speech from your chosen speaker in mp3, wav or flacc format, in one or more files (have tested as far as 20 minutes worth of audio).
#### ⚫ How will this work/How complicated is it?
Everything has been done to make this as simple as possible. At its simplest, you can literally just download a large chunk of audio from an interview, and tell the finetuning to strip through it, find spoken parts and build your dataset. You can literally click 4 buttons, then copy a few files and you are done. At it's more complicated end you will clean up the audio a little beforehand, but its still only 4x buttons and copying a few files.
#### ⚫ The audio you will use
I would suggest that if its in an interview format, you cut out the interviewer speaking in audacity or your chosen audio editing package. You dont have to worry about being perfect with your cuts, the finetuning Step 1 will go and find spoken audio and cut it out for you. Is there is music over the spoken parts, for best quality you would cut out those parts, though its not 100% necessary. As always, try to avoid bad quality audio with noises in it (humming sounds, hiss etc). You can try something like [Audioenhancer](https://audioenhancer.ai/) to try clean up noisier audio. There is no need to down-sample any of the audio, all of that is handled for you. Just give the finetuning some good quality audio to work with.
#### ⚫ Important requirements CUDA 11.8
As mentioned you must have a small portion of the Nvidia CUDA Toolkit **11.8** installed. Not higher or lower versions. Specifically **11.8**. You do not have to uninstall any other versions, change any graphics drivers, reinstall torch or anything like that. To keep the download+install as small as possible, you will need to:
- Download the **xxx (network)** install of the Nvidia Cuda Toolkit 11.8 from [here](https://developer.nvidia.com/cuda-11-8-0-download-archive)
- When you run the installation, select **Custom Advanced** as your installation type. Uncheck `CUDA`, `Other Components` and `Driver Components` at the top then expand `CUDA` > `Development` > `Compiler` > and select `nvcc` then expand `Libraries` and select `CUBLAS`
- Back at the top of `CUDA` expand `Runtime` > `Libraries` and select `CUBLAS`. You can now next through the install.
- When that has installed, open a terminal/command prompt and type `nvcc --version`. If it reports back `Cuda compilation tools, release 11.8.` you are good to go. **Specifically, 11.8**. If not continue to the next step.
- For both Windows an Linux, you will need to ensure that `nvcc` and the 11.8 cuda library files are in your environments search path. You can undo the changes below after finetuning if you prefer.<br><br>
**Windows** - Edit the Windows PATH environment variable and add `C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.8\bin`<br><br>
**Linux** - The path may be different depending on what flavour of Linux you are running, so you may need to seek out specific instructions on the internet. Generic paths **may** be:<br>
`export LD_LIBRARY_PATH=/usr/local/cuda-11.8/lib64:${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}` and<br>
`export LD_LIBRARY_PATH=/usr/local/cuda-11.8/bin`<br>
Add these to your '~/.bashrc' if you want this to be permanent and not something you have to set each time you open a new terminal.<br><br>
- When you have made the changes, open a new terminal/command prompt and `nvcc --version`. It should report back `Cuda compilation tools, release 11.8.` at which point, you are good to go.
- If it doesnt report that, check you have correctly set the search environment paths, dont have overlapping other versions of cuda paths etc.
#### ⚫ Starting Finetuning
**NOTE:** Please make sure you have started AllTalk at least once after updating, so that it downloads the additional files needed for finetuning.
1) Close all other applications that are using your GPU/VRAM and copy your audio samples into:<br><br>
`/alltalk_tts/finetune/put-voice-samples-in-here/`
3) In a command prompt/terminal window you need to move into your Text generation webUI folder:<br><br>
`cd text-generation-webui`
4) Start the Text generation webUI Python environment for your OS:<br><br>
`cmd_windows.bat`, `./cmd_linux.sh`, `cmd_macos.sh` or `cmd_wsl.bat`
5) You can double check your search path environment still works correctly with `nvcc --version`. It should report back 11.8:<br><br>
`Cuda compilation tools, release 11.8.`
7) Move into your extensions folder:<br><br>
`cd extensions`
8) Move into the **alltalk_tts** folder:<br><br>
`cd alltalk_tts`
9) Install the finetune requirements file: `pip install -r requirements_finetune.txt`
10) Type `python finetune.py` and it should start up.
11) Follow the on-screen instructions when the web interface starts up.
12) When you have finished finetuning, the final tab will tell you what to do with your files and how to move your newly trained model to the correct location on disk.
## 🔵🟢🟡 DeepSpeed Installation Options
**NOTE**: You **DO NOT** need to set Text-generation-webUI's **--deepspeed** setting for AllTalk to be able to use DeepSpeed. These are two completely separate things and incorrectly setting that on Text-generation-webUI may cause other complications.
#### 🔵 Linux Installation
<details>
<summary>Click to expand: Linux DeepSpeed installation</summary>
@@ -381,10 +463,142 @@ def get_shm_size():
**Removal** - If it became necessary to uninstall DeepSpeed, you can do so with `cmd_windows.bat` and then `pip uninstall deepspeed`<br><br>
</details>
## 🟠 API Suite and JSON-CURL
### 🟠Overview
The Text-to-Speech (TTS) Generation API allows you to generate speech from text input using various configuration options. This API supports both character and narrator voices, providing flexibility for creating dynamic and engaging audio content.
- URL: `http://127.0.0.1:7851/api/tts-generate`<br>
- Method: `POST`<br>
- Content-Type: `application/x-www-form-urlencoded`<br>
### 🟠 Example command lines
Standard TTS speech Example (standard text) generating a time-stamped file<br>
`curl -X POST "http://127.0.0.1:7851/api/tts-generate" -d "text_input=All of this is text spoken by the character. This is text not inside quotes, though that doesnt matter in the slightest" -d "text_filtering=standard" -d "character_voice_gen=female_01.wav" -d "narrator_enabled=false" -d "narrator_voice_gen=male_01.wav" -d "text_not_inside=character" -d "language=en" -d "output_file_name=myoutputfile" -d "output_file_timestamp=true" -d "autoplay=true" -d "autoplay_volume=0.8"`<br>
Narrator Example (standard text) generating a time-stamped file
`curl -X POST "http://127.0.0.1:7851/api/tts-generate" -d "text_input=*This is text spoken by the narrator* \"This is text spoken by the character\". This is text not inside quotes." -d "text_filtering=standard" -d "character_voice_gen=female_01.wav" -d "narrator_enabled=true" -d "narrator_voice_gen=male_01.wav" -d "text_not_inside=character" -d "language=en" -d "output_file_name=myoutputfile" -d "output_file_timestamp=true" -d "autoplay=true" -d "autoplay_volume=0.8"`<br>
Note that if your text that needs to be generated contains double quotes you will need to escape them with `\"` (Please see the narrator example).
### 🟠 Request Parameters
🟠 **text_input**: The text you want the TTS engine to produce. Use escaped double quotes for character speech and asterisks for narrator speech if using the narrator function. Example:
`-d "text_input=*This is text spoken by the narrator* \"This is text spoken by the character\". This is text not inside quotes."`
🟠 **text_filtering**: Filter for text. Options:
- **none** No filtering. Whatever is sent will go over to the TTS engine as raw text, which may result in some odd sounds with some special characters.<br>
- **standard** Human-readable text and a basic level of filtering, just to clean up some special characters.<br>
- **html** HTML content. Where you are using HTML entity's like &quot;<br>
`-d "text_filtering=none"`<br>
`-d "text_filtering=standard"`<br>
`-d "text_filtering=html"`<br>
Example:
- **Standard Example**: `*This is text spoken by the narrator* "This is text spoken by the character" This is text not inside quotes.`<br>
- **HTML Example**: `&ast;This is text spoken by the narrator&ast; &quot;This is text spoken by the character&quot; This is text not inside quotes.`<br>
- **None**: `Will just pass whatever characters/text you send at it.`<br>
🟠 **character_voice_gen**: The WAV file name for the character's voice.<br>
`-d "character_voice_gen=female_01.wav"`
🟠 **narrator_enabled**: Enable or disable the narrator function. If true, minimum text filtering is set to standard. Anything between double quotes is considered the character's speech, and anything between asterisks is considered the narrator's speech.
`-d "narrator_enabled=true"`<br>
`-d "narrator_enabled=false"`
🟠 **narrator_voice_gen**: The WAV file name for the narrator's voice.
`-d "narrator_voice_gen=male_01.wav"`
🟠 **text_not_inside**: Specify the handling of lines not inside double quotes or asterisks, for the narrator feature. Options:
- **character**: Treat as character speech.<br>
- **narrator**: Treat as narrator speech.<br>
`-d "text_not_inside=character"`<br>
`-d "text_not_inside=narrator"`
🟠 **language**: Choose the language for TTS. Options:
`ar Arabic`<br>
`zh-cn Chinese (Simplified)`<br>
`cs Czech`<br>
`nl Dutch`<br>
`en English`<br>
`fr French`<br>
`de German`<br>
`hu Hungarian`<br>
`it Italian`<br>
`ja Japanese`<br>
`ko Korean`<br>
`pl Polish`<br>
`pt Portuguese`<br>
`ru Russian`<br>
`es Spanish`<br>
`tr Turkish`<br>
`-d "language=en"`<br>
🟠 **output_file_name**: The name of the output file (excluding the .wav extension).
`-d "output_file_name=myoutputfile"`<br>
🟠 **output_file_timestamp**: Add a timestamp to the output file name. If true, each file will have a unique timestamp; otherwise, the same file name will be overwritten each time you generate TTS.
`-d "output_file_timestamp=true"`<br>
`-d "output_file_timestamp=false"`
🟠 **autoplay**: Feature not yet available. Enable or disable autoplay. Still needs to be specified in the JSON request.
`-d "autoplay=true"`<br>
`-d "autoplay=false"`
🟠 **autoplay_volume**: Feature not yet available. Set the autoplay volume. Should be between 0.1 and 1.0. Still needs to be specified in the JSON request.
`-d "autoplay_volume=0.8"`
### 🟠 TTS Generation Response
The API returns a JSON object with the following properties:
- **status** Indicates whether the generation was successful (generate-success) or failed (generate-failure).<br>
- **output_file_path** The on-disk location of the generated WAV file.<br>
- **output_file_url** The HTTP location for accessing the generated WAV file.<br>
Example JSON TTS Generation Response:
`{"status": "generate-success", "output_file_path": "C:\text-generation-webui\extensions\alltalk_tts\outputs\myoutputfile_1703149973.wav", "output_file_url": "http://127.0.0.1:7851/audio/myoutputfile_1703149973.wav"}`
🟠 **Switching Model**<br><br>
`curl -X POST "http://127.0.0.1:7851/api/reload?tts_method=API%20Local"`<br>
`curl -X POST "http://127.0.0.1:7851/api/reload?tts_method=API%20TTS"`<br>
`curl -X POST "http://127.0.0.1:7851/api/reload?tts_method=XTTSv2%20Local"`<br>
Switch between the 3 models respectively.
JSON return `{"status": "model-success"}`
🟠 **Switch DeepSpeed**<br><br>
`curl -X POST "http://127.0.0.1:7851/api/deepspeed?new_deepspeed_value=True"`
Replace True with False to disable DeepSpeed mode.
JSON return `{"status": "deepspeed-success"}`
🟠 **Switching Low VRAM**<br><br>
`curl -X POST "http://127.0.0.1:7851/api/lowvramsetting?new_low_vram_value=True"`
Replace True with False to disable Low VRAM mode.
JSON return `{"status": "lowvram-success"}`
### 🔴 Future to-do list
- Complete & document the new/full standalone mode API.
- Voice output within the command prompt/terminal (TBD).
- Correct an issue on incorrect output folder path when running as a standalone app.
- Correct a few spelling mistakes in the documnentation.
- Correct a few spelling mistakes in the documentation.
- Possibly add some additional TTS engines (TBD).
- Have a break!
+1019
View File
File diff suppressed because it is too large. Load diff
Whitespace-only changes.
+2
View File
@@ -6,6 +6,8 @@
"README.md": "https://huggingface.co/coqui/XTTS-v2/resolve/v2.0.2/README.md?download=true",
"config.json": "https://huggingface.co/coqui/XTTS-v2/resolve/v2.0.2/config.json?download=true",
"model.pth": "https://huggingface.co/coqui/XTTS-v2/resolve/v2.0.2/model.pth?download=true",
"dvae.pth": "https://huggingface.co/coqui/XTTS-v2/resolve/v2.0.2/dvae.pth?download=true",
"mel_stats.pth": "https://huggingface.co/coqui/XTTS-v2/resolve/v2.0.2/mel_stats.pth?download=true",
"vocab.json": "https://huggingface.co/coqui/XTTS-v2/resolve/v2.0.2/vocab.json?download=true"
}
}
+7
View File
@@ -0,0 +1,7 @@
gradio>=3.50.2
torch>=2.1.0+cu118
torchaudio>=2.1.0+cu118
TTS==0.21.3
faster-whisper>=0.10.0
tqdm>=4.66.0
pandas>=1.5.0
+2 -1
View File
@@ -1,7 +1,7 @@
numpy>=1.22.0
soundfile>=0.12.1
uvicorn>=0.24.0.post1
TTS>=0.22.0
TTS>=0.21.3
torch>=2.1.0+cu118
torchaudio>=2.1.0+cu118
fastapi>=0.104.1
@@ -10,3 +10,4 @@ requests>=2.31.0
tqdm>=4.66.1
importlib-metadata>=4.8.1
packaging>=23.2
pydantic>=1.10.13
+2 -1
View File
@@ -1,7 +1,7 @@
numpy>=1.22.0
soundfile>=0.12.1
uvicorn>=0.24.0.post1
TTS>=0.22.0
TTS>=0.21.3
torch>=2.1.0
torchaudio>=2.1.0
fastapi>=0.104.1
@@ -10,3 +10,4 @@ requests>=2.31.0
tqdm>=4.66.1
importlib-metadata>=4.8.1
packaging>=23.2
pydantic>=1.10.13
+18 -21
View File
@@ -39,8 +39,8 @@ try:
except ModuleNotFoundError:
logger.error(
f"[{params['branding']}]\033[91mWarning\033[0m Could not find the TTS module. Make sure to install the requirements for the {params['branding']} extension."
f"[{params['branding']}]\033[91mWarning\033[0m Linux / Mac:\npip install -r extensions/alltalk_tts/requirements.txt\n"
f"[{params['branding']}]\033[91mWarning\033[0m Windows:\npip install -r extensions\\alltalk_tts\\requirements.txt\n"
f"[{params['branding']}]\033[91mWarning\033[0m Linux / Mac:\npip install -r /alltalk_tts/requirements.txt\n"
f"[{params['branding']}]\033[91mWarning\033[0m Windows:\npip install -r \\alltalk_tts\\requirements.txt\n"
f"[{params['branding']}]\033[91mWarning\033[0m If you used the one-click installer, paste the command above in the terminal window launched after running the cmd_ script. On Windows, that's cmd_windows.bat."
)
raise
@@ -62,7 +62,9 @@ try:
from modules.utils import gradio
# This is set to check if the script is being run within text generation webui or as a standalone script. False is running as part of text gen web ui or a gradio interface
running_in_standalone = False
output_folder_wav = params["output_folder_wav"]
except ModuleNotFoundError:
output_folder_wav = params["output_folder_wav_standalone"]
print(f"[{params['branding']}Startup] Running script.py in standalone mode")
# This is set to check if the script is being run within text generation webui or as a standalone script. true means standalone
running_in_standalone = True
@@ -108,7 +110,6 @@ def delete_old_files(folder_path, days_to_keep):
# Extract settings using params dictionary
delete_output_wavs_setting = params["delete_output_wavs"]
output_folder_wav = params["output_folder_wav"]
output_folder_wav = os.path.normpath(output_folder_wav)
# Check and perform file deletion
@@ -248,7 +249,7 @@ else:
# Cleanly kill off this script, but allow text-generation-webui to keep running, albeit without this alltalk_tts
sys.exit(1)
timeout = 60 # Adjust the timeout as needed
timeout = 120 # Adjust the timeout as needed
# Introduce a delay before starting the check loop
time.sleep(25) # Wait 25 secs before checking if the tts_server.py has started up.
@@ -261,7 +262,7 @@ while time.time() - start_time < timeout:
except requests.RequestException as e:
# Print the exception for debugging purposes
print(
f"[{params['branding']}Startup] \033[91mWarning\033[0m TTS Subprocess has NOT started up yet, Will keep trying for 60 seconds maximum. Please wait."
f"[{params['branding']}Startup] \033[91mWarning\033[0m TTS Subprocess has NOT started up yet, Will keep trying for 120 seconds maximum. Please wait."
)
time.sleep(1)
else:
@@ -500,7 +501,7 @@ def output_modifier(string, state):
if process_lock.acquire(blocking=False):
try:
if params["narrator_enabled"]:
# print(original_string)
#print(original_string)
# Do Some basic stripping and remove any double CR's
processed_string = (
original_string
@@ -521,11 +522,11 @@ def output_modifier(string, state):
.replace('"', '&quot;<')
)
#capturing another outlier in inital character paragraph
# print("processed string 1 is:", processed_string)
#print("processed string 1 is:", processed_string)
processed_string = processed_string.replace('&quot;<. *', '&quot;< *"')
processed_string = processed_string.replace('< *"', '< *')
processed_string = processed_string.replace('. *', '< *')
# print("processed string 2 is:", processed_string)
#print("processed string 2 is:", processed_string)
# Set up a tracking of the individual wav files.
audio_files_all_paragraphs = []
# Split the line using &quot; and ".* " (so end of sentences, leaving special characters added to the start of all OTHER sentences, bar possibly the first one if its starting with a *
@@ -540,26 +541,21 @@ def output_modifier(string, state):
if '<' in part and '< *' not in part and '<*' not in part and '< *' not in part and '< ' not in part and '< ' not in part:
cleaned_part = html.unescape(part.replace('<', ''))
voice_to_use = params["voice"]
#Narrator will always be an * or < with an * a position or two after it.
#Narrator will always be an * or < with an * a position or two after it.
elif '<*' in part or '< *' in part or '< *' in part or '*' in part:
cleaned_part = html.unescape(part.replace('<*', '').replace('< *', '').replace('< *', '').replace('*', ''))
voice_to_use = params["narrator_voice"]
#If the other two dont capture it, aka, the AI gave no * or &quot; on the line, use non_quoted_text_is aka user interface, user can choose Char or Narrator
#If the other two dont capture it, aka, the AI gave no * or &quot; on the line, use non_quoted_text_is aka user interface, user can choose Char or Narrator
elif non_quoted_text_is:
cleaned_part = html.unescape(part.replace('< ', '').replace('< ', '').replace('< ', ''))
voice_to_use = params["voice"]
else:
cleaned_part = html.unescape(part.replace('< ', '').replace('< ', '').replace('< ', ''))
voice_to_use = params["narrator_voice"]
# Check if character name exists and if not, just call it TTSOUT_
if "character_menu" in state:
output_file = Path(f'{params["output_folder_wav"]}/{state["character_menu"]}_{int(time.time())}_{i}.wav')
else:
output_file = Path(f'{params["output_folder_wav"]}/TTSOUT_{int(time.time())}_{i}.wav')
# Generate that TTS and output to a file
output_file_str = output_file.as_posix()
output_filename = get_output_filename(state)
generate_response = send_generate_request(
cleaned_part, voice_to_use, language_code, output_file_str
cleaned_part, voice_to_use, language_code, output_filename
)
audio_path = generate_response.get("data", {}).get("audio_path")
audio_files_paragraph.append(audio_path)
@@ -627,9 +623,10 @@ def output_modifier(string, state):
def get_output_filename(state):
return Path(
f'{params["output_folder_wav"]}/{state["character_menu"]}_{str(uuid.uuid4())[:8]}.wav'
).as_posix()
if "character_menu" in state:
return Path(f'{params["output_folder_wav"]}/{state["character_menu"]}_{str(uuid.uuid4())[:8]}.wav').as_posix()
else:
return Path(f'{params["output_folder_wav"]}/TTSOUT_{str(uuid.uuid4())[:8]}.wav').as_posix()
###############################################
@@ -888,4 +885,4 @@ if running_in_standalone:
try:
time.sleep(1) # Add a small delay to avoid high CPU usage
except KeyboardInterrupt:
break # Allow graceful exit on Ctrl+C
break # Allow graceful exit on Ctrl+C
+405 -11
View File
@@ -161,7 +161,9 @@ async def setup():
# Set "tts_model_loaded" to true
params["tts_model_loaded"] = True
# Set the output path for wav files
Path(f'{params["output_folder_wav"]}').mkdir(parents=True, exist_ok=True)
output_directory = this_dir / params["output_folder_wav_standalone"]
output_directory.mkdir(parents=True, exist_ok=True)
#Path(f'this_folder/outputs/').mkdir(parents=True, exist_ok=True)
# MODEL LOADER For "API TTS"
@@ -617,6 +619,217 @@ async def get_audio(filename: str):
audio_path = this_dir / "outputs" / filename
return FileResponse(audio_path)
########################
#### GENERATION API ####
########################
import html
import re
import numpy as np
import soundfile as sf
from typing import Union, Dict
from pydantic import BaseModel, ValidationError, Field
class Request(BaseModel):
# Define the structure of the 'Request' class if needed
pass
class JSONInput(BaseModel):
text_input: str = Field(..., max_length=1000, description="text_input needs to be 1000 characters or less.")
text_filtering: str = Field(..., pattern="^(none|standard|html)$", description="text_filtering needs to be 'none', 'standard' or 'html'.")
character_voice_gen: str = Field(..., pattern="^.*\.wav$", description="character_voice_gen needs to be the name of a wav file e.g. mysample.wav.")
narrator_enabled: bool = Field(..., description="narrator_enabled needs to be true or false.")
narrator_voice_gen: str = Field(..., pattern="^.*\.wav$", description="narrator_voice_gen needs to be the name of a wav file e.g. mysample.wav.")
text_not_inside: str = Field(..., pattern="^(character|narrator)$", description="text_not_inside needs to be 'character' or 'narrator'.")
language: str = Field(..., pattern="^(ar|zh-cn|cs|nl|en|fr|de|hu|it|ja|ko|pl|pt|ru|es|tr)$", description="language needs to be one of the following ar|zh-cn|cs|nl|en|fr|de|hu|it|ja|ko|pl|pt|ru|es|tr.")
output_file_name: str = Field(..., pattern="^[a-zA-Z0-9_]+$", description="output_file_name needs to be the name without any special characters or file extension e.g. 'filename'")
output_file_timestamp: bool = Field(..., description="output_file_timestamp needs to be true or false.")
autoplay: bool = Field(..., description="autoplay needs to be a true or false value.")
autoplay_volume: float = Field(..., ge=0.1, le=1.0, description="autoplay_volume needs to be from 0.1 to 1.0")
@classmethod
def validate_autoplay_volume(cls, value):
if not (0.1 <= value <= 1.0):
raise ValueError("Autoplay volume must be between 0.1 and 1.0")
return value
class TTSGenerator:
@staticmethod
def validate_json_input(json_data: Union[Dict, str]) -> Union[None, str]:
try:
if isinstance(json_data, str):
json_data = json.loads(json_data)
JSONInput(**json_data)
return None # JSON is valid
except ValidationError as e:
return str(e)
def standard_filtering(text_input):
text_output = (text_input
.replace("***", "")
.replace("**", "")
.replace("*", "")
.replace("\n\n", "\n")
.replace("&#x27;", "'")
)
return text_output
def narrator_filtering(narrator_text_input):
processed_string = re.sub(r'\.\*\n\*', '. ', narrator_text_input)
processed_string = (
processed_string
.replace("\n", " ")
.replace('&quot;', '&quot;<')
.replace('"', '&quot;<')
)
processed_string = processed_string.replace('&quot;<. *', '&quot;< *"')
processed_string = processed_string.replace('< *"', '< *')
processed_string = processed_string.replace('. *', '< *')
text_output = html.unescape(processed_string)
return text_output
def combine(output_file_timestamp, output_file_name, audio_files):
audio = np.array([])
sample_rate = None
try:
for audio_file in audio_files:
audio_data, current_sample_rate = sf.read(audio_file)
if audio.size == 0:
audio = audio_data
sample_rate = current_sample_rate
elif sample_rate == current_sample_rate:
audio = np.concatenate((audio, audio_data))
else:
raise ValueError("Sample rates of input files are not consistent.")
except Exception as e:
# Handle exceptions (e.g., file not found, invalid audio format)
return None, None
if output_file_timestamp:
timestamp = int(time.time())
output_file_path = os.path.join(this_dir / "outputs" / f'{output_file_name}_{timestamp}_combined.wav')
output_file_url = f'http://{params["ip_address"]}:{params["port_number"]}/audio/{output_file_name}_{timestamp}_combined.wav'
else:
output_file_path = os.path.join(this_dir / "outputs" / f'{output_file_name}_combined.wav')
output_file_url = f'http://{params["ip_address"]}:{params["port_number"]}/audio/{output_file_name}.wav'
try:
sf.write(output_file_path, audio, samplerate=sample_rate)
# Clean up unnecessary files
for audio_file in audio_files:
os.remove(audio_file)
except Exception as e:
# Handle exceptions (e.g., failed to write output file)
return None, None
return output_file_path, output_file_url
# Generation API (separate from text-generation-webui)
@app.post("/api/tts-generate", response_class=JSONResponse)
async def tts_generate(
text_input: str = Form(...),
text_filtering: str = Form(...),
character_voice_gen: str = Form(...),
narrator_enabled: bool = Form(...),
narrator_voice_gen: str = Form(...),
text_not_inside: str = Form(...),
language: str = Form(...),
output_file_name: str = Form(...),
output_file_timestamp: bool = Form(...),
autoplay: bool = Form(...),
autoplay_volume: float = Form(...),
):
try:
#print(f"text_filtering: {text_filtering}")
#print(f"narrator_enabled: {narrator_enabled}")
#print(f"text_not_inside: {text_not_inside}")
#print(f"output_file_timestamp: {output_file_timestamp}")
json_input_data = {
"text_input": text_input,
"text_filtering": text_filtering,
"character_voice_gen": character_voice_gen,
"narrator_enabled": narrator_enabled,
"narrator_voice_gen": narrator_voice_gen,
"text_not_inside": text_not_inside,
"language": language,
"output_file_name": output_file_name,
"output_file_timestamp": output_file_timestamp,
"autoplay": autoplay,
"autoplay_volume": autoplay_volume,
}
JSONresult = TTSGenerator.validate_json_input(json_input_data)
if JSONresult is None:
pass
else:
return JSONResponse(content={"error": JSONresult}, status_code=400)
if narrator_enabled:
print("ORIGINAL UNTOUCHED STRING IS:", text_input,"\n")
if text_filtering in ["standard", "none"]:
cleaned_string = (
text_input
.replace('"', '"<')
.replace('*', '<*')
)
print("STANDARD FILTERING IS NOW:", cleaned_string,"\n")
#parts = re.split(r'\.(?<!<[*"])', cleaned_string)
parts = re.split(r'(?<=<\*\s)|(?<=\.\s)|(?<=\."\<)|(?<=\."<)', cleaned_string)
parts = list(filter(lambda x: x.strip(), parts))
elif text_filtering == "html":
cleaned_string = standard_filtering(text_input)
cleaned_string = narrator_filtering(cleaned_string)
print("HTML FILTERING IS NOW:", cleaned_string,"\n")
parts = re.split(r'&quot;|\.\*', cleaned_string)
audio_files_all_paragraphs = []
audio_files_paragraph = []
for i, part in enumerate(parts):
if len(part.strip()) <= 1:
continue
print("THIS IS A PART", part)
# Figure out which type of line it is, then replace characters as necessary to avoid TTS trying to pronunce them, htmlunescape after.
# Character will always be a < with a letter immediately after it
if '<' in part and '<*' not in part and '< *' not in part and '< *' not in part and '< ' not in part and '< ' not in part:
print("IF - Character\n")
cleaned_part = html.unescape(part.replace('<', ''))
voice_to_use = character_voice_gen
#Narrator will always be an * or < with an * a position or two after it.
elif '<*' in part or '< *' in part or '< *' in part or '*' in part:
print("IF - Narrator\n")
cleaned_part = html.unescape(part.replace('<*', '').replace('< *', '').replace('< *', '').replace('*', '').replace('<. ', ''))
voice_to_use = narrator_voice_gen
#If the other two dont capture it, aka, the AI gave no * or &quot; on the line, use non_quoted_text_is aka user interface, user can choose Char or Narrator
elif text_not_inside == "character":
print("ELSE - CHARACTER\n")
cleaned_part = html.unescape(part.replace('< ', '').replace('< ', '').replace('< ', ''))
voice_to_use = character_voice_gen
elif text_not_inside == "narrator":
print("ELSE - NARRATOR\n")
cleaned_part = html.unescape(part.replace('< ', '').replace('< ', '').replace('< ', ''))
voice_to_use = narrator_voice_gen
output_file = this_dir / "outputs" / f"{output_file_name}_{int(time.time())}_{i}.wav"
output_file_str = output_file.as_posix()
await generate_audio(cleaned_part, voice_to_use, language, output_file_str)
audio_path = output_file_str
audio_files_paragraph.append(audio_path)
# Accumulate audio files within the paragraph
audio_files_all_paragraphs.extend(audio_files_paragraph)
# Combine audio files across paragraphs
output_file_path, output_file_url = combine(output_file_timestamp, output_file_name, audio_files_all_paragraphs)
else:
if output_file_timestamp:
timestamp = int(time.time())
output_file_path = this_dir / "outputs" / f"{output_file_name}_{timestamp}.wav"
output_file_url = f'http://{params["ip_address"]}:{params["port_number"]}/audio/{output_file_name}_{timestamp}.wav'
else:
output_file_path = this_dir / "outputs" / f"{output_file_name}.wav"
output_file_url = f'http://{params["ip_address"]}:{params["port_number"]}/audio/{output_file_name}.wav'
if text_filtering == "html":
cleaned_string = html.unescape(standard_filtering(text_input))
elif text_filtering == "standard":
cleaned_string = standard_filtering(text_input)
else:
cleaned_string = text_input
await generate_audio(cleaned_string, character_voice_gen, language, output_file_path)
return JSONResponse(content={"status": "generate-success", "output_file_path": str(output_file_path), "output_file_url": str(output_file_url)}, status_code=200)
except Exception as e:
return JSONResponse(content={"status": "generate-failure", "error": "An error occurred"}, status_code=500)
#############################################################
#### DOCUMENTATION - README ETC - PRESENTED AS A WEBPAGE ####
#############################################################
@@ -818,6 +1031,7 @@ simple_webpage = """
<li><a href="#using-voice-samples">Using Voice Samples</a></li>
<li><a href="#text-not-inside">Text Not inside</a></li>
<li><a href="#low-vram">Low VRAM</a></li>
<li><a href="#finetune">Finetuning (Training the model)</a></li>
<li><a href="#deepspeed">DeepSpeed</a>
<ul>
<li><a href="#deepspeed-linux">DeepSpeed Setup Linux</a></li>
@@ -838,6 +1052,12 @@ simple_webpage = """
<p style="padding-left: 30px; text-align: justify;">AllTalk is a web interface, based around the Coqui TTS voice cloning/speech generation system. To generate TTS, you can use the provided gradio interface or interact with the server using JSON/CURL commands.&nbsp;</p>
<p style="padding-left: 30px; text-align: justify;"><span style="color: #ff0000;">Note:</span> When loading up a new character in Text generation webUI it may look like nothing is happening for 20-30 seconds. Its actually processing the introduction section (greeting message) of the text and once that is completed, it will appear. You can see the activity occuring in the console window. Refreshing the page multiple times will try force the TTS engine to keep re-generating the text, so please just wait and check the console if needed.</p>
<p style="padding-left: 30px; text-align: justify;"><span style="color: #ff0000;">Note: </span>Ensure that your RP character card has asterisks around anything for the narration and double quotes around anything spoken. There is a complication ONLY with the greeting card so, ensuring it has the correct use of quotes and asterisks will help make sure that the greeting card sounds correct. I will aim to address this issue in a future update. In Text-generation-webUI <span style="color: #3366ff;">parameters menu</span> &gt; <span style="color: #3366ff;">character tab</span> &gt; <span style="color: #3366ff;">greeting</span>&nbsp;make sure that anything in there that is the narrator is in asterisks and anything spoken is in double quotes, then hit the <span style="color: #3366ff;">save</span> (💾) button.</p>
<p style="padding-left: 30px;"><span style="text-decoration: underline;"><strong>AllTalk <span style="color: #ff0000; text-decoration: underline;">Minor Bug Fixes Changelog &amp; Known issues</span></strong></span></p>
<p style="padding-left: 30px;">If I squash any minor bugs or find any issues, I will try to apply an update asap. If you think something isnt working correctly or you have a problem, check these two links below first.</p>
<ul>
<li>Minor bug fixes changelog&nbsp;<a href="https://github.com/erew123/alltalk_tts/issues/25" target="_blank" rel="noopener">link here</a>.</li>
<li>Help and known issues <a href="https://github.com/erew123/alltalk_tts?#-help-with-problems" target="_blank" rel="noopener">link here</a>.</li>
</ul>
<p style="padding-left: 30px;"><span style="text-decoration: underline;"><strong>AllTalk</strong></span> <br />Github <a href="https://github.com/erew123/alltalk_tts" target="_blank" rel="noopener">link here</a><br />Update instructions <a href="https://github.com/erew123/alltalk_tts?#-updating" target="_blank" rel="noopener">link here</a><br />Help and issues <a href="https://github.com/erew123/alltalk_tts?#-help-with-problems" target="_blank" rel="noopener">link here<br /><br /></a><span style="text-decoration: underline;"><strong>Text generation webUI</strong></span> <br />Web interface&nbsp;<a href="http://{{ params["ip_address"] }}:7860" target="_blank" rel="noopener">link here</a> <br />Documentation <a href="https://github.com/oobabooga/text-generation-webui/wiki" target="_blank" rel="noopener">link here</a></p>
<p style="padding-left: 30px;"><a href="#toc">Back to top of page</a></p>
<h2 id="server-information">Server Information</h2>
@@ -967,6 +1187,82 @@ simple_webpage = """
<p style="padding-left: 30px;"><a href="#toc">Back to top of page</a></p>
<h2 id="finetune"><strong>Finetuning (Training the model)</strong></h2>
<p style="margin-left:40px; text-align:justify">If you have a voice that the model doesnt quite reproduce correctly, or indeed you just want to improve the reproduced voice, then finetuning is a way to train your &quot;XTTSv2 local&quot; model <span style="color:#3366ff">(stored in /alltalk_tts/models/xxxxx/)</span> on a specific voice. For this you will need:</p>
<ul style="margin-left:40px">
<li>An Nvidia graphics card To install a few portions of the Nvidia CUDA <strong>11.8</strong> Toolkit (this will not impact text-generation-webui&#39;s cuda setup.</li>
<li>18GB of disk space free (most of this is used temporarily)</li>
<li>At least 2 minutes of good quality speech from your chosen speaker in mp3, wav or flacc format, in one or more files (have tested as far as 20 minutes worth of audio).</li>
</ul>
<h4>How will this work/How complicated is it?</h4>
<p style="margin-left:40px; text-align:justify">Everything has been done to make this as simple as possible. At its simplest, you can literally just download a large chunk of audio from an interview, and tell the finetuning to strip through it, find spoken parts and build your dataset. You can literally click 4 buttons, then copy a few files and you are done. At it&#39;s more complicated end you will clean up the audio a little beforehand, but its still only 4x buttons and copying a few files.</p>
<h4>The audio you will use:</h4>
<p style="margin-left:40px; text-align:justify">I would suggest that if its in an interview format, you cut out the interviewer speaking in audacity or your chosen audio editing package. You dont have to worry about being perfect with your cuts, the finetuning Step 1 will go and find spoken audio and cut it out for you. Is there is music over the spoken parts, for best quality you would cut out those parts, though its not 100% necessary. As always, try to avoid bad quality audio with noises in it (humming sounds, hiss etc). You can try something like Audioenhancer to try clean up noisier audio. There is no need to down-sample any of the audio, all of that is handled for you. Just give the finetuning some good quality audio to work with.</p>
<h4>Important requirements CUDA 11.8:</h4>
<p style="margin-left:40px; text-align:justify">As mentioned you must have a small portion of the <span style="color:#3366ff">Nvidia CUDA Toolkit&nbsp;<strong>11.8</strong></span>&nbsp;installed. Not higher or lower versions. Specifically&nbsp;<strong>11.8</strong>. You do not have to uninstall any other versions, change any graphics drivers, reinstall torch or anything like that. There are instructions within the finetuning interface on doing this or you can also find them on this link&nbsp;<a href="https://github.com/erew123/alltalk_tts#-important-requirements-cuda-118">here</a></p>
<h4>Starting Finetuning:</h4>
<p style="margin-left:40px"><strong>Ensure</strong> you have followed the instructions on setting up the Nvidia CUDA Toolkit 11.8 <a href="https://github.com/erew123/alltalk_tts#-important-requirements-cuda-118">here</a>&nbsp;or the below procedure will fail.</p>
<p style="margin-left:40px">The below instructions are also available online&nbsp;<a href="https://github.com/erew123/alltalk_tts#-finetuning-a-model">here</a></p>
<ol>
<li>
<p>Close all other applications that are using your GPU/VRAM and copy your audio samples into:<br />
<br />
<span style="color:#3366ff">/alltalk_tts/finetune/put-voice-samples-in-here/</span></p>
</li>
<li>
<p>In a command prompt/terminal window you need to move into your Text generation webUI folder:<br />
<br />
<span style="color:#3366ff">cd text-generation-webui</span></p>
</li>
<li>
<p>Start the Text generation webUI Python environment for your OS:<br />
<br />
<span style="color:#3366ff">cmd_windows.bat,&nbsp;./cmd_linux.sh,&nbsp;cmd_macos.sh&nbsp;or&nbsp;cmd_wsl.bat</span></p>
</li>
<li>
<p>You can double check your search path environment still works correctly with&nbsp;<span style="color:#3366ff">nvcc --version</span>. It should report back <span style="color:#3366ff">11.8</span>:<br />
<br />
<span style="color:#3366ff">Cuda compilation tools, release 11.8.</span></p>
</li>
<li>
<p>Move into your extensions folder:<br />
<br />
<span style="color:#3366ff">cd extensions</span></p>
</li>
<li>
<p>Move into the&nbsp;<span style="color:#3366ff">alltalk_tts</span>&nbsp;folder:<br />
<br />
<span style="color:#3366ff">cd alltalk_tts</span></p>
</li>
<li>
<p>Install the finetune requirements file:&nbsp;<span style="color:#3366ff">pip install -r requirements_finetune.txt</span></p>
</li>
<li>
<p>Type<span style="color:#3366ff">&nbsp;python finetune.py</span>&nbsp;and it should start up.</p>
</li>
<li>
<p>Follow the on-screen instructions when the web interface starts up.</p>
</li>
<li>
<p>When you have finished finetuning, the final tab will tell you what to do with your files and how to move your newly trained model to the correct location on disk.</p>
</li>
</ol>
<p><a href="#toc">Back to top of page</a></p>
<h2 id="deepspeed"><strong>DeepSpeed</strong></h2>
<p style="padding-left: 30px; text-align: justify;">DeepSpeed provides a 2x-3x speed boost for Text-to-Speech and AI tasks. It's all about making AI and TTS happen faster and more efficiently.</p>
<ul>
@@ -1258,31 +1554,129 @@ simple_webpage = """
</table>
<h2 id="curl-commands"><strong>JSON calls &amp; CURL Commands</strong></h2>
<h4>Generating TTS - Linux</h4>
<p style="padding-left: 30px;"><span style="color: #3366ff;">curl -X POST -H "Content-Type: application/json" -d '{"text": "<span style="color: #ff9900;">This is text to generate as TTS</span>","voice": "<span style="color: #ff9900;">female_01.wav</span>", "language": "<span style="color: #ff9900;">en</span>", "output_file": "<span style="color: #ff9900;">outputfile.wav</span>"}' "http://127.0.0.1:7851/api/generate"</span></p>
<h4>Generating TTS - Windows</h4>
<p style="padding-left: 30px;"><span style="color: #3366ff;">curl -X POST -H "Content-Type: application/json" -d "{&bsol;"text&bsol;": &bsol;"<span style="color: #ff9900;">This is text to generate as TTS</span>&bsol;", &bsol;"voice&bsol;": &bsol;"<span style="color: #ff9900;">female_01.wav</span>&bsol;", &bsol;"language&bsol;": &bsol;"<span style="color: #ff9900;">en</span>&bsol;", &bsol;"output_file&bsol;": &bsol;"<span style="color: #ff9900;">outputfile.wav</span>&bsol;"}" http://127.0.0.1:7851/api/generate</span></p>
<h4>Generating TTS - Notes</h4>
<p style="padding-left: 30px;">Replace <span style="color: #ff9900;">This is text to generate as TTS</span> with whatever you want it to say. <span style="color: #ff9900;">female_01.wav</span> with the voice sample you want to use. <span style="color: #ff9900;">output_file.wav</span> with the file name you want it to create.</p>
<p style="padding-left: 30px;">JSON return <span style="color: #339966;">{"status":"generate-success","data":{"audio_path":"outputfile.wav"}}</span></p>
<h3>Overview</h3>
<p style="margin-left: 40px; text-align: justify;">The Text-to-Speech (TTS) Generation API allows you to generate speech from text input using various configuration options. This API supports both character and narrator voices, providing flexibility for creating dynamic and engaging audio content.</p>
<h3>TTS Generation Endpoint</h3>
<ul>
<li><strong>URL</strong>: <span style="color: #3366ff;">http://127.0.0.1:7851/api/tts-generate</span></li>
<li><strong>Method</strong>: <span style="color: #3366ff;">POST</span></li>
<li><strong>Content-Type</strong>: <span style="color: #3366ff;">application/x-www-form-urlencoded</span></li>
</ul>
<h3>Example command line</h3>
<p style="padding-left: 30px;">Standard TTS speech Example (standard text) generating a time-stamped file</p>
<p style="padding-left: 30px;"><span style="color: #3366ff;">curl -X POST "http://127.0.0.1:7851/api/tts-generate" -d "text_input=</span><span style="color: #ff9900;">All of this is text spoken by the character. This is text not inside quotes, though that doesnt matter in the slightest</span><span style="color: #3366ff;">" -d "text_filtering=</span><span style="color: #339966;">standard</span><span style="color: #3366ff;">" -d "character_voice_gen=</span><span style="color: #ff9900;">female_01.wav</span><span style="color: #3366ff;">" -d "narrator_enabled=</span><span style="color: #339966;">false</span><span style="color: #3366ff;">" -d "narrator_voice_gen=male_01.wav" -d "text_not_inside=character" -d "language=en" -d "output_file_name=</span><span style="color: #ff9900;">myoutputfile</span><span style="color: #3366ff;">" -d "output_file_timestamp=</span><span style="color: #ff9900;">true</span><span style="color: #3366ff;">" -d "autoplay=true" -d "autoplay_volume=0.8"</span></p>
<p style="padding-left: 30px;">Narrator Example (standard text)&nbsp;generating a time-stamped file</p>
<p style="padding-left: 30px;"><span style="color: #3366ff;">curl -X POST "http://127.0.0.1:7851/api/tts-generate" -d "text_input=</span><span style="color: #ff9900;">*This is text spoken by the narrator* &bsol;"This is text spoken by the character&bsol;". This is text not inside quotes.</span><span style="color: #3366ff;">" -d "text_filtering=</span><span style="color: #339966;">standard</span><span style="color: #3366ff;">" -d "character_voice_gen=</span><span style="color: #ff9900;">female_01.wav</span><span style="color: #3366ff;">" -d "narrator_enabled=</span><span style="color: #339966;">true</span><span style="color: #3366ff;">" -d "narrator_voice_gen=</span><span style="color: #339966;">male_01.wav</span><span style="color: #3366ff;">" -d "text_not_inside=</span><span style="color: #339966;">character</span><span style="color: #3366ff;">" -d "language=en" -d "output_file_name=</span><span style="color: #ff9900;">myoutputfile</span><span style="color: #3366ff;">" -d "output_file_timestamp=</span><span style="color: #ff9900;">true</span><span style="color: #3366ff;">" -d "autoplay=true" -d "autoplay_volume=0.8"</span></p>
<p style="padding-left: 30px; text-align: justify;"><strong><span style="color: #ff0000;">Note</span></strong> that if your text that needs to be generated contains double quotes you will need to escape them with&nbsp;&bsol;" (Please see the narrator example).</p>
<h3>&nbsp;Request Parameters</h3>
<ul>
<li>
<p style="text-align: justify;"><strong>text_input</strong>: The text you want the TTS engine to produce. Use escaped double quotes for character speech and asterisks for narrator speech if using the narrator function. Example:</p>
<p>-<span style="color: #3366ff;">d "text_input=*This is text spoken by the narrator* &bsol;"This is text spoken by the character&bsol;". This is text not inside quotes." </span></p>
</li>
<li>
<p><strong>text_filtering</strong>: Filter for text. Options:</p>
<ul>
<li style="text-align: justify;"><span style="color: #3366ff;">none</span>&nbsp;No filtering. Whatever is sent will go over to the TTS engine as raw text, which may result in some odd sounds with some special characters.</li>
<li><span style="color: #3366ff;">standard</span>&nbsp;Human-readable text and a basic level of filtering, just to clean up some special characters.</li>
<li><span style="color: #3366ff;">html</span>&nbsp;HTML content. Where you are using HTML entity's like &amp;quot;</li>
</ul>
<p><span style="color: #3366ff;">-d "text_filtering=none" </span><br /><span style="color: #3366ff;">-d "text_filtering=standard" </span><br /><span style="color: #3366ff;">-d "text_filtering=html"</span></p>
<p><strong>Example:</strong></p>
<ul>
<li>Standard Example: <span style="color: #3366ff;">*This is text spoken by the narrator* "This is text spoken by the character" This is text not inside quotes.</span></li>
<li>HTML Example:<span style="color: #3366ff;"> *This is text spoken by the narrator*&nbsp;&amp;quot;This is text spoken by the character&amp;quot;&nbsp;This is text not inside quotes.</span></li>
<li>None will just pass whatever characters/text you send at it.</li>
</ul>
</li>
<li>
<p><strong>character_voice_gen</strong>: The WAV file name for the character's voice.</p>
<p><span style="color: #3366ff;">-d "character_voice_gen=female_01.wav" </span></p>
</li>
<li>
<p><strong>narrator_enabled</strong>: Enable or disable the narrator function. If true, minimum text filtering is set to standard. Anything between double quotes is considered the character's speech, and anything between asterisks is considered the narrator's speech.</p>
<p><span style="color: #3366ff;">-d "narrator_enabled=true"<br />-d "narrator_enabled=false" </span><span style="font-family: Verdana, Arial, Helvetica, sans-serif;">&nbsp;</span></p>
</li>
<li>
<p><strong>narrator_voice_gen</strong>: The WAV file name for the narrator's voice.</p>
<p><span style="color: #3366ff;">-d "narrator_voice_gen=male_01.wav"</span></p>
</li>
<li>
<p><strong>text_not_inside</strong>: Specify the handling of lines not inside double quotes or asterisks, for the narrator feature. Options:</p>
<ul>
<li><span style="color: #3366ff;">character</span>: Treat as character speech.</li>
<li><span style="color: #3366ff;">narrator</span>: Treat as narrator speech.</li>
</ul>
<p><span style="color: #3366ff;">-d "text_not_inside=character" </span><br /><span style="color: #3366ff;">-d "text_not_inside=narrator"</span></p>
</li>
<li>
<p><strong>language</strong>: Choose the language for TTS. Options:</p>
<ul>
<li><span style="color: #3366ff;">ar</span>&nbsp;Arabic</li>
<li><span style="color: #3366ff;">zh-cn</span>&nbsp;Chinese (Simplified)</li>
<li><span style="color: #3366ff;">cs</span>&nbsp;Czech</li>
<li><span style="color: #3366ff;">nl</span> Dutch</li>
<li><span style="color: #3366ff;">en</span> English</li>
<li><span style="color: #3366ff;">fr</span> French</li>
<li><span style="color: #3366ff;">de</span> German</li>
<li><span style="color: #3366ff;">hu</span> Hungarian</li>
<li><span style="color: #3366ff;">it</span> Italian</li>
<li><span style="color: #3366ff;">ja</span> Japanese</li>
<li><span style="color: #3366ff;">ko</span> Korean</li>
<li><span style="color: #3366ff;">pl</span> Polish</li>
<li><span style="color: #3366ff;">pt</span> Portuguese</li>
<li><span style="color: #3366ff;">ru </span>Russian</li>
<li><span style="color: #3366ff;">es</span> Spanish</li>
<li><span style="color: #3366ff;">tr</span> Turkish</li>
</ul>
<p><span style="color: #3366ff;">-d "language=en"</span></p>
</li>
<li>
<p><strong>output_file_name</strong>: The name of the output file (excluding the .wav extension).</p>
<p><span style="color: #3366ff;">-d "output_file_name=myoutputfile" </span></p>
</li>
<li>
<p><strong>output_file_timestamp</strong>: Add a timestamp to the output file name. If true, each file will have a unique timestamp; otherwise, the same file name will be overwritten each time you generate TTS.</p>
<p><span style="color: #3366ff;">-d "output_file_timestamp=true" <br />-d "output_file_timestamp=false" </span></p>
</li>
<li>
<p><strong>autoplay</strong>: <span style="color: #ff0000;">Feature not yet available</span>. Enable or disable autoplay. Still needs to be specified in the JSON request.</p>
<p><span style="color: #3366ff;">-d "autoplay=true" <br />-d "autoplay=false"</span></p>
</li>
<li>
<p><strong>autoplay_volume</strong>: <span style="color: #ff0000;">Feature not yet available</span>. Set the autoplay volume. Should be between 0.1 and 1.0.&nbsp;Still needs to be specified in the JSON request.</p>
<p><span style="color: #3366ff;">-d "autoplay_volume=0.8" </span></p>
</li>
</ul>
<h3>TTS Generation Response</h3>
<p style="padding-left: 30px;">The API returns a JSON object with the following properties:</p>
<ul>
<li><span style="color: #3366ff;"><strong>status</strong></span>&nbsp;Indicates whether the generation was successful (<span style="color: #3366ff;">generate-success</span>) or failed (<span style="color: #3366ff;">generate-failure</span>).</li>
<li><span style="color: #3366ff;"><strong>output_file_path</strong></span>&nbsp;The on-disk location of the generated WAV file.</li>
<li><span style="color: #3366ff;"><strong>output_file_url</strong></span>&nbsp;The HTTP location for accessing the generated WAV file.</li>
</ul>
<p><strong>Example JSON TTS Generation Response:</strong></p>
<p style="padding-left: 30px;"><span style="color: #339966;">{"status": "generate-success", "output_file_path": "C:\\text-generation-webui\\extensions\\alltalk_tts\\outputs\\myoutputfile_1703149973.wav", "output_file_url": "http://127.0.0.1:7851/audio/myoutputfile_1703149973.wav"}</span></p>
<h4>Switching Model</h4>
<p style="padding-left: 30px;"><span style="color: #3366ff;">curl -X POST "http://127.0.0.1:7851/api/reload?tts_method=API%20Local"</span><br /><span style="color: #3366ff;"> curl -X POST "http://127.0.0.1:7851/api/reload?tts_method=API%20TTS"</span><br /><span style="color: #3366ff;"> curl -X POST "http://127.0.0.1:7851/api/reload?tts_method=XTTSv2%20Local"</span></p>
<p style="padding-left: 30px;">Switch between the 3 models respectively.</p>
<p style="padding-left: 30px;">JSON return <span style="color: #339966;">{"status": "model-success"}</span></p>
<h4>Switch DeepSpeed</h4>
<p style="padding-left: 30px;"><span style="color: #3366ff;">curl -X POST -H "Content-Type: application/json" "http://127.0.0.1:7851/api/deepspeed?new_deepspeed_value=True"</span></p>
<p style="padding-left: 30px;"><span style="color: #3366ff;">curl -X POST "http://127.0.0.1:7851/api/deepspeed?new_deepspeed_value=True"</span></p>
<p style="padding-left: 30px;">Replace True with False to disable DeepSpeed mode.</p>
<p style="padding-left: 30px;">JSON return <span style="color: #339966;">{"status": "deepspeed-success"}</span></p>
<h4>Switching Low VRAM</h4>
<p style="padding-left: 30px;"><span style="color: #3366ff;">curl -X POST -H "Content-Type: application/json" "http://127.0.0.1:7851/api/lowvramsetting?new_low_vram_value=True"</span></p>
<p style="padding-left: 30px;"><span style="color: #3366ff;">curl -X POST "http://127.0.0.1:7851/api/lowvramsetting?new_low_vram_value=True"</span></p>
<p style="padding-left: 30px;">Replace True with False to disable Low VRAM mode.</p>
<p style="padding-left: 30px;">JSON return <span style="color: #339966;">{"status": "lowvram-success"}</span></p>
<p><a href="#toc">Back to top of page<br /></a></p>
<h2 id="debugging-and-tts-generation-information"><strong>Debugging and TTS Generation Information</strong></h2>
<p style="padding-left: 30px; text-align: justify;">Command line outputs are more verbose to assist in understanding backend processes and debugging.</p>
<p style="padding-left: 30px; text-align: justify;">Its possible during startup you can get a warning message such as: <br /><br /><span style="color: black;">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[AllTalk Startup]</span> <span style="color: #ff0000;">Warning</span> <span style="color: black;">TTS Subprocess has NOT started up yet, Will keep trying for 60 seconds maximum</span> <br /><br />This is normal behavior if the subprocess is taking a while to start, however, if there is an issue starting the subprocess, you may see multiples of this message and an it will time out after 60 seconds, resulting in the TTS engine not starting. Its likely that you are not in the correct python environment or one that has a TTS engine inside, if this happens, though the system will output a warning about that ahead of this message</p>
<p style="padding-left: 30px; text-align: justify;">Its possible during startup you can get a warning message such as: <br /><br /><span style="color: black;">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[AllTalk Startup]</span> <span style="color: #ff0000;">Warning</span> <span style="color: black;">TTS Subprocess has NOT started up yet, Will keep trying for 80 seconds maximum</span> <br /><br />This is normal behavior if the subprocess is taking a while to start, however, if there is an issue starting the subprocess, you may see multiples of this message and an it will time out after 80 seconds, resulting in the TTS engine not starting. Its likely that you are not in the correct python environment or one that has a TTS engine inside, if this happens, though the system will output a warning about that ahead of this message</p>
<p style="padding-left: 30px; text-align: justify;">Typically the command line console will output any warning or error messages. If you need to reset your default configuation, the settings are all listed above in the configuration details.</p>
<p style="padding-left: 30px;"><a href="#toc">Back to top of page</a></p>
<h2 id="references"><strong>Thanks &amp; References</strong></h2>