diff --git a/README.md b/README.md index 72baa97..aa312d7 100644 --- a/README.md +++ b/README.md @@ -1,40 +1,82 @@ # AllTalk TTS AllTalk is an updated version of the Coqui_tts extension for Text Generation web UI. Features include: -- **Can be run as a** [standalone](https://github.com/erew123/alltalk_tts?#-running-alltalk-as-a-standalone-app) **or part of** [Text-generation-webui](https://github.com/erew123/alltalk_tts?#-installation-on-text-generation-web-ui) +- **Can be run as a** [standalone](https://github.com/erew123/alltalk_tts/#-quick-setup-text-generation-webui--standalone-installation) **or part of** [Text-generation-webui](https://github.com/erew123/alltalk_tts/#-quick-setup-text-generation-webui--standalone-installation) **using the a quick setup utility** - **Model Finetuning:** Train the model specifically on a voice of your choosing for better reproduction. - **Local/Custom models:** Use any of the XTTSv2 models (API Local and XTTSv2 Local). - **Bulk TTS Generator/Editor:** Generate hours of TTS into one big file or have something read back to you [demo](https://www.youtube.com/watch?v=hunvXn0mLzc). -- **DeepSpeed:** A 3-4x performance boost generating TTS. [DeepSpeed Windows/Linux Instructions](https://github.com/erew123/alltalk_tts?tab=readme-ov-file#deepspeed-installation-options) [Screenshot](https://github.com/erew123/alltalk_tts/assets/35898566/548619c8-5f1b-47d0-a73d-54d2fee3f3db) -- **Low VRAM mode:** Great for people with small GPU memory or if your VRAM is filled by your LLM. [Screenshot](https://github.com/erew123/alltalk_tts#screenshots) -- **Custom Start-up Settings:** Adjust your default start-up settings. [Screenshot](https://github.com/erew123/alltalk_tts#screenshots) +- **DeepSpeed:** A 2-3x performance boost generating TTS. [Screenshot](https://github.com/erew123/screenshots/raw/main/deepspeed.jpg) +- **Low VRAM mode:** Great for people with small GPU memory or if your VRAM is filled by your LLM. +- **Custom Start-up Settings:** Adjust your default start-up settings. [Screenshot](https://github.com/erew123/screenshots/raw/main/settingsanddocs.jpg) - **Narrarator:** Use different voices for main character and narration. [Example Narration](https://vocaroo.com/18nrv7FR6wuA) -- **Optional wav file maintenance:** Configurable deletion of old output wav files. [Screenshot](https://github.com/erew123/alltalk_tts#screenshots) -- **Documentation:** Fully documented with a built in webpage. [Screenshot](https://github.com/erew123/alltalk_tts#screenshots) -- **Console output:** Clear command line output for any warnings or issues. +- **Optional wav file maintenance:** Configurable deletion of old output wav files. [Screenshot](https://github.com/erew123/screenshots/raw/main/settingsanddocs.jpg) +- **Documentation:** Fully documented with a built in webpage. [Screenshot](https://github.com/erew123/screenshots/raw/main/settingsanddocs.jpg) +- **Clear Console output:** Clear command line output for any warnings or issues. - **API Suite and 3rd Party support via JSON calls:** Can be used with 3rd party applications via JSON calls. +- **SillyTavern integration:** Full integration with SillyTavern. [Screenshot](https://github.com/erew123/screenshots/raw/main/sillytavern.jpg) ## Index -- 🟩 [Installation](https://github.com/erew123/alltalk_tts?#-installation-on-text-generation-web-ui) +- 🟩 [Installation](https://github.com/erew123/alltalk_tts/#-quick-setup-text-generation-webui--standalone-installation) - 🟪 [Updating & problems with updating](https://github.com/erew123/alltalk_tts?#-updating) - 🟫 [Screenshots](https://github.com/erew123/alltalk_tts#-screenshots) - 🟨 [Help with problems](https://github.com/erew123/alltalk_tts?#-help-with-problems) - ⚫ [Finetuning a model](https://github.com/erew123/alltalk_tts?#-finetuning-a-model) - 🔵🟢🟡 [DeepSpeed Installation (Windows & Linux)](https://github.com/erew123/alltalk_tts?#-deepspeed-installation-options) -- 🟦 [Running AllTalk as a standalone app](https://github.com/erew123/alltalk_tts?#-running-alltalk-as-a-standalone-app) - ⬜ [AllTalk TTS Generator](https://github.com/erew123/alltalk_tts?#-alltalk-tts-generator) - 🟠 [API Suite and JSON-CURL](https://github.com/erew123/alltalk_tts?#-api-suite-and-json-curl) - 🔴 [Future to-do list & Upcoming updates](https://github.com/erew123/alltalk_tts?#-future-to-do-list) -## 🔄 Updates +## 🔄 Updates & Bug fixes Please check the below link to find a list of all recent updates and changes. ####     🔄 **Updates list & bug fixes list** can be found [here](https://github.com/erew123/alltalk_tts/issues/25) -## 🟩 Installation on Text generation web UI +## 🟩 Quick Setup (Text-generation-webui & Standalone Installation) +For Windows 10/11 and Linux machines there is a quick setup script. Click to expand the correct section below: +
+ QUICK SETUP - Text-Generation-webui
+ + If you wish to see this as a video, please go [here](https://www.youtube.com/watch?v=icn2XS5rUH8) +1) To download the AllTalk setup you can either: + - A) On this page select **CODE** > **Download ZIP** then extract it to the text-generation-webui extensions folder
e.g. `\text-generation-webui\extensions\alltalk_tts\`

+ - B) Go to a terminal/console, move into the `\text-generation-webui\extensions\` folder
and `git clone https://github.com/erew123/alltalk_tts`

+3) In a terminal/command prompt, in the text-generation-webui folder you will start its Python environment with either `cmd_windows.bat` or `./cmd_linux.sh` +4) Move into the AllTalk folder e.g. `cd extensions` then `cd alltalk_tts` +5) Start the AllTalk setup script `atsetup.bat` or `./atsetup.sh` +6) Follow the on-screen prompts and install the correct requirements files that you need. It's recommended to test AllTalk works before installing DeepSpeed. + + Any time you need to make changes to AllTalk, or use Finetuning etc, always start the Text-generation-webui Python environment first. + + Please read the `🟩 Other installation notes` (also additional voices are available there). + + Finetuning & DeepSpeed have other installation requirements (depending on your OS) so please read any instructions in the setup utility and refer back here to this page for detailed instructions (as needed).

+
+
+ QUICK SETUP - Standalone Installation
+ + If you wish to see this as a video, please go [here](https://www.youtube.com/watch?v=AQYCccDRbaY) +1) To download the AllTalk setup you can either: + - A) On this page select **CODE** > **Download ZIP** then extract it to the folder of your choice
e.g. `C:\myfiles\alltalk_tts\`

+ - B) Go to a terminal/console, move into the folder of your choice e.g `C:\myfiles\` folder
and `git clone https://github.com/erew123/alltalk_tts`

+4) In a terminal/command prompt, move into the AllTalk folder e.g. `cd alltalk_tts` +5) Start the AllTalk setup script `atsetup.bat` or `./atsetup.sh` +6) Follow the on-screen prompts and install the correct requirements files that you need. It's recommended to test AllTalk works before installing DeepSpeed. + + DeepSpeed on Windows machines will be installed as standard. Linux machines have other requirements which are detailed within the setup utility and on this page. + + Please read the `🟩 Other installation notes` (also additional voices are available there). + + Finetuning has other installation requirements so please read any instructions in the setup utility and refer back here to this page for detailed instructions.

+
+ +## 🟩 Manual Installation - As part of Text generation web UI +On Mac's or if you wish to perform a manual installation. Click to expand the correct section below: +
+ MANUAL INSTALLATION - Text-Generation-webui
+ This has been tested on the current Dec 2023 release of Text generation webUI. If you have not updated it for a while, you may wish to update Text generation webUI, [instructions here](https://github.com/oobabooga/text-generation-webui?tab=readme-ov-file#how-to-install) -If you want to watch a video of how to do the below [link here](https://youtu.be/9BPKuwaav5w) +- If you want to watch a video of how to do the below [link here](https://youtu.be/9BPKuwaav5w)
1) In a command prompt/terminal window you need to move into your Text generation webUI folder:

`cd text-generation-webui` @@ -58,10 +100,64 @@ If you want to watch a video of how to do the below [link here](https://youtu.be 7) **(Optional DeepSpeed)** If you have an Nvidia Graphics card on a system running Linux or Windows and wish to use **DeepSpeed** please follow these instructions [here](https://github.com/erew123/alltalk_tts?#-deepspeed-installation-options). **However**, I would highly reccommend before you install DeepSpeed, you start text-generation-webui up, confirm AllTalk starts correctly and everything is working, as DeepSpeed can add another layer of complications troubleshooting any potential start-up issues. If necessary you can `pip uninstall deepspeed`. 8) You can now start move back to the main Text generation webUI folder `cd ..` (a few times), start Text generation webUI with whichever **one** of the startup scripts is correct for your OS (`start_windows.bat`,`./start_linux.sh`, `start_macos.sh` or `start_wsl.bat`) and load the AllTalk extension in the Text generation webUI **session** tab.

Starting Text-generation-webui with its correct start-up script is $\textcolor{red}{\textsf{VERY IMPORTANT}}$. - -10) Please read the note below about start-up times and also the note about ensuring your character cards are set up [correctly](https://github.com/erew123/alltalk_tts#the-one-thing-i-cant-easily-work-around) -11) Some extra voices downloadable [here](https://drive.google.com/file/d/1bYdZdr3L69kmzUN3vSiqZmLRD7-A3M47/view?usp=drive_link) + Any time you need to make changes to AllTalk, or use Finetuning etc, always start the Text-generation-webui Python environment first. + + Please read the `🟩 Other installation notes` (also additional voices are available there). + + Finetuning & DeepSpeed have other installation requirements (depending on your OS) so please read any instructions in the setup utility and refer back here to this page for detailed instructions (as needed).

+ +
+ +## 🟩 Manual Installation - As a Standalone Application +On Mac's or if you wish to perform a manual installation. Click to expand the correct section below: +
+ MANUAL INSTALLATION - I want to run AllTalk as a standalone when installed with Text-generation-webui
+ +If you already have AllTalk as a extension of Text-generation-webui, and wish to run it as standalone, load Text-generation-webui's Python environment `cmd_windows.bat`, `./cmd_linux.sh`, `cmd_macos.sh` or `cmd_wsl.bat`, move into the AllTalk folder `cd extensions` > `cd alltalk_tts` and start AllTalk with `python script.py`. There is nothing beyond this you would need to do. +
+
+ MANUAL INSTALLATION - I wish to do a custom install of AllTalk
+ +AllTalk will run as a standalone app, as long as you install its requirements files into whatever Python environment you are using. You can follow the steps to install the AllTalk's requirements into whatever Python environment you wish. Because I dont know what Python environment you are wanting to use, I can only give you a loose set of installation instructions. + +Please note, at time of writing, the TTS engine requires Python **3.9.x** to **3.11.x** [TTS Engine details here](https://pypi.org/project/TTS/). AllTalk and its requirements are tested on Python **3.11.x**. + +#### 🟩 A very quick understanding of Python Environments. +Different Python applications have different requirements, some of those requirement’s conflict with other Python applications requirements. To work around this problem, you can create different Python environments that remain separated from one another. A simple way of looking at Python environments, is just like how your house has different rooms for specific purposes (Kitchen, Bathroom, Bedroom etc). You can create a Python environment that is built/customised specifically for your current applications needs/purposes and will **not** interfere with any other Python applications environments/installations.

If you are adept at managing Python environments, have an existing Python environment and know that you won’t cause any conflicts by installing AllTalk's requirements within that Python environment, then load up your Python environment and install the requirements. For everyone else, here is a basic guide on installing AllTalk in its own custom Python environment (there will be small variations between OS's, but the principle is the same). + +**Note:** A standard VENV can cause module path issues, hence Conda is the correct method to create a Python environment for AllTalk. + +#### 🟩 Building a custom Python Environment with Miniconda +1) Open a terminal/command prompt, and confirm that both `python --version` and `pip` both work. Neither of them should give an error message. If they do you may need to install Python (and maybe Pip).

[Python Website](https://www.python.org/downloads/)
[Pip Website](https://pip.pypa.io/en/stable/installation/)

Once you have those working you can now continue on.

+ +2) Assuming you don't already have Miniconda installed, we will need to download and install Minoconda as this is what we will use to build our custom Python environment.

[Miniconda Website](https://docs.conda.io/projects/miniconda/en/latest/)

Download the Miniconda version that is correct for your OS and then install it. Miniconda will create some items in your Start Menu or Application Launcher called something like `Anaconda Prompt`

You will start the `Anaconda Prompt` and your prompt in your terminal/command prompt will say something like:

`(base) C:\users\myaccount\>`

The important bit is that it has the `(base)`, the location after that doesn't matter. `(base)` signifies we are in the base conda Python environment and can now create custom Python environments with Conda.

+ +3) In your Anaconda Prompt command prompt/terminal, move into the folder where you want to download AllTalk to and then git clone this repository. For simplicity I am going to assume that location is `c:\myfiles\`. So you will:

`cd myfiles`

`git clone https://github.com/erew123/alltalk_tts`

+ +4) Now we will create our custom Conda Python 3.11.5 environment and give it the name alltalkenv by typing the following at the prompt:

`conda create --name alltalkenv python=3.11.5`

You will be prompted if you want to continue and you say yes.

Once that process has completed, we now need to change from the `(base)` Conda Python environment, to our newly created `(alltalkenv)` environment. So type the following:

`conda activate alltalkenv`

Your prompt should now change from `(base) C:\myfiles\>` to `(alltalkenv) C:\myfiles\>`

+ +5) **If you are NOT using an Nvidia card and CUDA, skip to step 6**. To force PyTorch to install with CUDA, perform the following:

`pip cache purge` (Clear the current Pip cache)

`pip uninstall torch torchaudio` (Uninstall Torch and Torchaudio from the `alltalkenv` environment)

+You can now either install the CUDA 11.8 or 12.1 version of Torch and Torchaudio with **one** of the following

**CUDA 11.8** > `pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118`

**CUDA 12.1** > `pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121`

+ +6) Move into the alltalk_tts folder and install one of the two requirements files. Whichever one of the two is correct for your machine type:

`cd alltalk_tts`

+**Nvidia graphics card machines** - `pip install -r requirements_nvidia.txt`

+**Other machines (mac, amd etc)** - `pip install -r requirements_other.txt`

+ +7) Start AllTalk with `python script.py`

+ +🟩 Anytime you wish to use AllTalk, update its requirements, install some other requirements such as DeepSpeed, you will need to start its Conda Python Environment in the terminal/prompt, which you will do with `conda activate alltalkenv` as long as the Conda executable is accessible at the command prompt/terminal you are in.

It is fully possible to create a batch file or script file that you can use as a launcher from your Start Menu/Application Launcher that will start the environment and AllTalk for you. + +Deepspeed and other such things can be installed. Please read the relevant instructions for those items, however, make the relevant changes to load your correct Python environment when installing any requirements files and starting AllTalk.

+ + Any time you need to make changes to AllTalk, or use Finetuning etc, always start your Python environment first. + + Please read the `🟩 Other installation notes` (also additional voices are available there). + + Finetuning & DeepSpeed have other installation requirements (depending on your OS) so please read any instructions in the setup utility and refer back here to this page for detailed instructions (as needed).

+ +
+ #### 🟩 Other installation notes On first startup, AllTalk will download the Coqui XTTSv2 2.0.2 model to its **models** folder (1.8GB space required). Check the command prompt/terminal window if you want to know what its doing. After it says "Model Loaded" the Text generation webUI is usually available on its IP address a few seconds later, for you to connect to in your browser. @@ -70,7 +166,11 @@ Once the extension is loaded, please find all documentation and settings on the **Where to find voices** https://aiartes.com/voiceai or https://commons.wikimedia.org/ or interviews on youtube etc. Instructions on how to cut down and prepare a voice sample are within the built in documentation. -#### 🟩 The one thing I cant easily work around +Please read the note below about start-up times and also the note about ensuring your character cards are set up [correctly](https://github.com/erew123/alltalk_tts#-a-note-on-character-cards--greeting-messages) + +Some extra voices for AllTalk are downloadable [here](https://drive.google.com/file/d/1bYdZdr3L69kmzUN3vSiqZmLRD7-A3M47/view?usp=drive_link) + +#### 🟩 A note on Character Cards & Greeting Messages Messages intended for the Narrator should be enclosed in asterisks `*` and those for the character inside quotation marks `"`. However, AI systems often deviate from these rules, resulting in text that is neither in quotes nor asterisks. Sometimes, text may appear with only a single asterisk, and AI models may vary their formatting mid-conversation. For example, they might use asterisks initially and then switch to unmarked text. A properly formatted line should look like this: `"`Hey! I'm so excited to finally meet you. I've heard so many great things about you and I'm eager to pick your brain about computers.`"` `*`She walked across the room and picked up her cup of coffee`*` @@ -80,21 +180,33 @@ Most narrator/character systems switch voices upon encountering an asterisk or q With improvements to the splitter/processor, I'm confident it's functioning well. You can monitor what AllTalk identifies as Narrator lines on the command line and adjust its behavior if needed (Text Not Inside - Function). ## 🟪 Updating +
+ UPDATING - I am using Text-Generation-webui
This is pretty much a repeat of the installation process. 1) In a command prompt/terminal window you need to move into your Text generation webUI folder:

`cd text-generation-webui` and start the Python environment for your OS with whichever **one** of the below is correct for your OS:

`cmd_windows.bat`, `./cmd_linux.sh`, `cmd_macos.sh` or `cmd_wsl.bat`
-2) Move into your extensions and alltalk_tts folder:

-`cd extensions` then `cd alltalk_tts` +2) Move into your extensions and alltalk_tts folder e.g. `cd extensions` then `cd alltalk_tts` -3) At the command prompt/terminal, type:

-`git pull` +3) At the command prompt/terminal, type `git pull` 4) Install the correct requirements for your machine:

**Nvidia graphics card machines** - `pip install -r requirements_nvidia.txt`

-**Other machines (mac, amd etc)** - `pip install -r requirements_other.txt` +**Other machines (mac, amd etc)** - `pip install -r requirements_other.txt`

+
+
+ UPDATING - I am running as a Standalone Application
+ +1) In a command prompt/terminal window you need to move into your `alltalk_tts` folder and run `start_environment.bat` or `/start_environment.sh` to load the Python environment. + +2) At the command prompt/terminal, type `git pull` and wait for it to complete the download. + +3) Install the correct requirements for your machine:

+**Nvidia graphics card machines** - `pip install -r requirements_nvidia.txt`

+**Other machines (mac, amd etc)** - `pip install -r requirements_other.txt`

+
#### 🟪 Updating "git pull" error @@ -147,16 +259,16 @@ Assuming its all working fine and you are happy, you can delete the old alltalk_ ## 🟫 Screenshots -|![image](https://github.com/erew123/alltalk_tts/assets/35898566/15c6ba0b-e6b0-4570-8253-b920dd76174d) | ![image](https://github.com/erew123/alltalk_tts/assets/35898566/548619c8-5f1b-47d0-a73d-54d2fee3f3db) | -|:---:|:---:| -|![image](https://github.com/erew123/alltalk_tts/assets/35898566/b0e13dba-c6b1-4ab7-845d-244ac1158330) | ![image](https://github.com/erew123/alltalk_tts/assets/35898566/e35e987c-543a-486b-b4fb-ee6ebe6f59c6) | -|![image](https://github.com/erew123/alltalk_tts/assets/35898566/68edc968-bdd2-4f98-9d56-b9ca40decf26) | ![image](https://github.com/erew123/alltalk_tts/assets/35898566/eb5e617b-2345-418d-8267-a9d30f705a10) | -|![image](https://github.com/erew123/alltalk_tts/assets/35898566/4bd0f02e-298d-4e46-95ea-afc003c1cb70) | +|![image](https://github.com/erew123/screenshots/raw/main/textgensettings.jpg) | ![image](https://github.com/erew123/screenshots/raw/main/setuputilitys.jpg) | ![image](https://github.com/erew123/screenshots/raw/main/deepspeed.jpg) |![image](https://github.com/erew123/screenshots/raw/main/textgen.jpg) | +|:---:|:---:|:---:|:---:| +|![image](https://github.com/erew123/screenshots/raw/main/settingsanddocs.jpg) | ![image](https://github.com/erew123/screenshots/raw/main/finetune1.jpg) | ![image](https://github.com/erew123/screenshots/raw/main/finetune2.jpg) |![image](https://github.com/erew123/screenshots/raw/main/sillytavern.jpg)| ## 🟨 Help with problems + ####      🔄 **Minor updates/bug fixes list** can be found [here](https://github.com/erew123/alltalk_tts/issues/25) #### 🟨 How to make a diagnostics report file +If you are on a Windows machine or a Linux machine, you should be able to use the `atsetup.bat` or `./atsetup.sh` utility to create a diagnositcs file. If you are unable to use the `atsetup` utility, please follow the instructions below.
Click to expand @@ -286,6 +398,7 @@ If you have a voice that the model doesnt quite reproduce correctly, or indeed y - To install a few portions of the Nvidia CUDA 11.8 Toolkit (this will not impact text-generation-webui's cuda setup. - 18GB of disk space free (most of this is used temporarily) - At least 2 minutes of good quality speech from your chosen speaker in mp3, wav or flacc format, in one or more files (have tested as far as 20 minutes worth of audio). +- As a side note, many people seem to think that the Whisper v2 model (used on Step 1) is giving better results at generating training datasets, so you may prefer to try that, as opposed to the Whisper 3 model. #### ⚫ How will this work/How complicated is it? Everything has been done to make this as simple as possible. At its simplest, you can literally just download a large chunk of audio from an interview, and tell the finetuning to strip through it, find spoken parts and build your dataset. You can literally click 4 buttons, then copy a few files and you are done. At it's more complicated end you will clean up the audio a little beforehand, but its still only 4x buttons and copying a few files. @@ -294,8 +407,8 @@ Everything has been done to make this as simple as possible. At its simplest, yo I would suggest that if its in an interview format, you cut out the interviewer speaking in audacity or your chosen audio editing package. You dont have to worry about being perfect with your cuts, the finetuning Step 1 will go and find spoken audio and cut it out for you. Is there is music over the spoken parts, for best quality you would cut out those parts, though its not 100% necessary. As always, try to avoid bad quality audio with noises in it (humming sounds, hiss etc). You can try something like [Audioenhancer](https://audioenhancer.ai/) to try clean up noisier audio. There is no need to down-sample any of the audio, all of that is handled for you. Just give the finetuning some good quality audio to work with. #### ⚫ Important requirements CUDA 11.8 -As mentioned you must have a small portion of the Nvidia CUDA Toolkit **11.8** installed. Not higher or lower versions. Specifically **11.8**. You do not have to uninstall any other versions, change any graphics drivers, reinstall torch or anything like that. To keep the download+install as small as possible, you will need to: -- Download the **xxx (network)** install of the Nvidia Cuda Toolkit 11.8 from [here](https://developer.nvidia.com/cuda-11-8-0-download-archive) +As mentioned you must have a small portion of the Nvidia CUDA Toolkit **11.8** installed. Not higher or lower versions. Specifically **11.8**. You do not have to uninstall any other versions, change any graphics drivers, reinstall torch or anything like that. This requirement is for Step1 of Finetuning. To keep the download+install as small as possible, you will need to: +- Download the **xxx (network)** install of the Nvidia CUDA Toolkit 11.8 from [here](https://developer.nvidia.com/cuda-11-8-0-download-archive) - Run the installer. At minimum, you need to [minimally] install the `nvcc` compiler and the `CUBLAS` development and runtime libraries: - Select **Custom Advanced** as your installation type. - Uncheck all the checkboxes in the list. @@ -314,6 +427,8 @@ As mentioned you must have a small portion of the Nvidia CUDA Toolkit **11.8** i - When you have made the changes, open a **new** terminal/command prompt (in order to load the new search paths) and `nvcc --version`. It should report back `Cuda compilation tools, release 11.8.` at which point, you are good to go. - If it doesn't report that, check you have correctly set the search environment paths, dont have overlapping other versions of cuda paths etc. +**Note:** Its also important that your Torch and Torchaudio have cuda installed (of any version). If you run the AllTalk diagnostics you can see your Torch and Torchaudio versions listed there. Cuda 11.8 will be listed as `cu118` and Cuda 12.1 as `cu121`. This Torch and Torchaudio is seperate to the above requirement to have the Nvidia CUDA Toolkit installed, so dont confuse the two different requirements. If you dont have Cuda installed on Torch and Torchaudio, Step 2 of Finetuning will fail. + #### ⚫ Starting Finetuning **NOTE:** Please make sure you have started AllTalk at least once after updating, so that it downloads the additional files needed for finetuning. @@ -361,6 +476,8 @@ If you've compacted and moved your model, its highly unlikely you would want to ➡️DeepSpeed requires an Nvidia Graphics card!⬅️ +- #### DeepSpeed Installation for Text generation webUI + 1) Preferably use your built in package manager to install CUDA tools. Alternatively download and install the Nvidia Cuda Toolkit for Linux [Nvidia Cuda Toolkit 11.8 or 12.1](https://developer.nvidia.com/cuda-toolkit-archive)

2) Open a terminal console.

3) Install libaio-dev (however your Linux version installs things) e.g. `sudo apt install libaio-dev`

@@ -372,14 +489,36 @@ If you've compacted and moved your model, its highly unlikely you would want to `export CUDA_HOME=/etc/alternatives/cuda`

**every** time you run `./cmd_linux.sh`.

If you try to start DeepSpeed with the CUDA_HOME path set incorrectly, expect an error similar to `[Errno 2] No such file or directory: /home/yourname/text-generation-webui/installer_files/env/bin/nvcc`

-9) Now install deepspeed with pip install deepspeed

-10) You can now start Text generation webUI `python server.py` ensuring to activate your extensions.

+8) Now install deepspeed with pip install deepspeed

+9) You can now start Text generation webUI `python server.py` ensuring to activate your extensions.

Just to reiterate, starting Text-generation-webUI with `./start_linux.sh` will overwrite the CUDA_HOME variable unless you have permanently changed it, hence always starting it with `./cmd_linux.sh` **then** setting the environment variable manually (step 7) and **then** `python server.py`, which is how you would need to run it each time, unless you permanently set the environment variable for CUDA_HOME within Text-generation-webUI's standard Python environment.

**Removal** - If it became necessary to uninstall DeepSpeed, you can do so with `./cmd_linux.sh` and then `pip uninstall deepspeed`

+ +- #### DeepSpeed Installation for standalone alltalk_tts app + +1) Preferably use your built in package manager to install CUDA tools. Alternatively download and install the Nvidia Cuda Toolkit for Linux [Nvidia Cuda Toolkit 11.8 or 12.1](https://developer.nvidia.com/cuda-toolkit-archive)

+2) Open a terminal console.

+3) Install libaio-dev (however your Linux version installs things) e.g. `sudo apt install libaio-dev` If you are using an RPM based distribution the package will probably be named `libaio-devel`.

+4) Navigate to your alltalk_tts folder e.g. `cd alltext_tts`

+5) Start the alltalk_tts Python environment `conda activate alltalkenv`

+6) You will need to set the **CUDA_HOME** environment variable. Details to change it each time are on the next step. Below is a link to Conda's manual and changing environment variables permanently though its possible changing it permanently could affect other extensions, you would have to test.

+[Conda manual - Environment variables](https://conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#set-env-vars)

+7) Set the environment path for CUDA_HOME e.g. `export CUDA_HOME=/usr/local/cuda-12.1/bin` or `export CUDA_HOME=/etc/alternative/cuda`. Verify the path on your Linux distro before setting this variable.
+If you try to start DeepSpeed with the CUDA_HOME path set incorrectly, expect an error similar to `[Errno 2] No such file or directory` or `CUDA_HOME does not exist`

+If you set the `env` variable and you receive an error of `[Errno 2] No such file or directory: '/usr/local/cuda-12.1/bin/bin/nvcc'` with a double `bin/bin` path, you can remove the env variable with `unset CUDA_HOME` and add the path to your linux PATH env variable with: `export PATH=$PATH:/usr/local/cuda-12.1/bin`. (Verify the path on your Linux distro before setting this variable.) However if you are using other applications that use CUDA, you will want to verify that configuring the path did not break those applications.

+8) Now install deepspeed with `pip install deepspeed`

+9) You can now start the alltalk_tts webUI with `python script.py`

+

+**Removal** - If it became necessary to uninstall DeepSpeed, you can do so by entering your `alltalkenv` enviroment `conda activate alltalkenv` and then running `pip uninstall deepspeed`

+ +
- + #### 🟢🟡 Windows Installation +
+ Click to Expand: Windows DeepSpeed options
+ DeepSpeed v11.2 will work on the current default text-generation-webui Python 3.11 environment! You have 2x options for how to setup DeepSpeed on Windows. A quick way (🟢Option 1) and a long way (🟡Option 2). Thanks to [@S95Sedan](https://github.com/S95Sedan) - They managed to get DeepSpeed 11.2 working on Windows via making some edits to the original Microsoft DeepSpeed v11.2 installation. The original post is [here](https://github.com/oobabooga/text-generation-webui/issues/4734#issuecomment-1843984142). @@ -504,42 +643,7 @@ def get_shm_size():

**Removal** - If it became necessary to uninstall DeepSpeed, you can do so with `cmd_windows.bat` and then `pip uninstall deepspeed`

- -## 🟦 Running AllTalk as a standalone app -AllTalk will run as a standalone app, as long as you install its requirements files into whatever Python environment you are using. You can follow the steps to install the AllTalk's requirements into whatever Python environment you wish. Because I dont know what Python environment you are wanting to use, I can only give you a loose set of installation instructions. - -Please note, at time of writing, the TTS engine requires Python **3.9.x** to **3.11.x** [TTS Engine details here](https://pypi.org/project/TTS/). AllTalk and its requirements are tested on Python **3.11.x**. - -#### 🟦 (Option 1) I already have AllTalk installed as an extension of Text-generation-webui -If you already have AllTalk as a extension of Text-generation-webui, and wish to run it as standalone, load Text-generation-webui's Python environment `cmd_windows.bat`, `./cmd_linux.sh`, `cmd_macos.sh` or `cmd_wsl.bat`, move into the AllTalk folder `cd extensions` > `cd alltalk_tts` and start AllTalk with `python script.py`. There is nothing beyond this you would need to do. - -#### 🟦 (Option 2) I wish to do a custom install of AllTalk -#### 🔹 A very quick understanding of Python Environments. -Different Python applications have different requirements, some of those requirement’s conflict with other Python applications requirements. To work around this problem, you can create different Python environments that remain separated from one another. A simple way of looking at Python environments, is just like how your house has different rooms for specific purposes (Kitchen, Bathroom, Bedroom etc). You can create a Python environment that is built/customised specifically for your current applications needs/purposes and will **not** interfere with any other Python applications environments/installations.

If you are adept at managing Python environments, have an existing Python environment and know that you won’t cause any conflicts by installing AllTalk's requirements within that Python environment, then load up your Python environment and install the requirements. For everyone else, here is a basic guide on installing AllTalk in its own custom Python environment (there will be small variations between OS's, but the principle is the same). - -**Note:** A standard VENV can cause module path issues, hence Conda is the correct method to create a Python environment for AllTalk. - -#### 🔹 Building a custom Python Environment with Miniconda -1) Open a terminal/command prompt, and confirm that both `python --version` and `pip` both work. Neither of them should give an error message. If they do you may need to install Python (and maybe Pip).

[Python Website](https://www.python.org/downloads/)
[Pip Website](https://pip.pypa.io/en/stable/installation/)

Once you have those working you can now continue on.

- -2) Assuming you don't already have Miniconda installed, we will need to download and install Minoconda as this is what we will use to build our custom Python environment.

[Miniconda Website](https://docs.conda.io/projects/miniconda/en/latest/)

Download the Miniconda version that is correct for your OS and then install it. Miniconda will create some items in your Start Menu or Application Launcher called something like `Anaconda Prompt`

You will start the `Anaconda Prompt` and your prompt in your terminal/command prompt will say something like:

`(base) C:\users\myaccount\>`

The important bit is that it has the `(base)`, the location after that doesn't matter. `(base)` signifies we are in the base conda Python environment and can now create custom Python environments with Conda.

- -3) In your Anaconda Prompt command prompt/terminal, move into the folder where you want to download AllTalk to and then git clone this repository. For simplicity I am going to assume that location is `c:\myfiles\`. So you will:

`cd myfiles`

`git clone https://github.com/erew123/alltalk_tts`

- -4) Now we will create our custom Conda Python 3.11.5 environment and give it the name alltalkenv by typing the following at the prompt:

`conda create --name alltalkenv python=3.11.5`

You will be prompted if you want to continue and you say yes.

Once that process has completed, we now need to change from the `(base)` Conda Python environment, to our newly created `(alltalkenv)` environment. So type the following:

`conda activate alltalkenv`

Your prompt should now change from `(base) C:\myfiles\>` to `(alltalkenv) C:\myfiles\>`

- -5) **If you are NOT using an Nvidia card and CUDA, skip to step 6**. To force PyTorch to install with CUDA, perform the following:

`pip cache purge` (Clear the current Pip cache)

`pip uninstall torch torchaudio` (Uninstall Torch and Torchaudio from the `alltalkenv` environment)

-You can now either install the CUDA 11.8 or 12.1 version of Torch and Torchaudio with **one** of the following

**CUDA 11.8** > `pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118`

**CUDA 12.1** > `pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121`

- -6) Move into the alltalk_tts folder and install one of the two requirements files. Whichever one of the two is correct for your machine type:

`cd alltalk_tts`

-**Nvidia graphics card machines** - `pip install -r requirements_nvidia.txt`

-**Other machines (mac, amd etc)** - `pip install -r requirements_other.txt`

- -7) Start AllTalk with `python script.py`

- -🔹Anytime you wish to use AllTalk, update its requirements, install some other requirements such as DeepSpeed, you will need to start its Conda Python Environment in the terminal/prompt, which you will do with `conda activate alltalkenv` as long as the Conda executable is accessible at the command prompt/terminal you are in.

It is fully possible to create a batch file or script file that you can use as a launcher from your Start Menu/Application Launcher that will start the environment and AllTalk for you. - -Deepspeed and other such things can be installed. Please read the relevant instructions for those items, however, make the relevant changes to load your correct Python environment when installing any requirements files and starting AllTalk.

+ ## ⬜ AllTalk TTS Generator AllTalk TTS Generator is the solution for converting large volumes of text into speech using the voice of your choice. Whether you're creating audio content or just want to hear text read aloud, the TTS Generator is equipped to handle it all efficiently. Please see here for a quick [demo](https://www.youtube.com/watch?v=hunvXn0mLzc)

The link to open the TTS generator can be found on the built-in Settings and Documentation page.

**DeepSpeed** is **highly** recommended to speed up generation. **Low VRAM** would be best turned off and your LLM model unloaded from your GPU VRAM (unload your model). **No Playback** will reduce memory overhead on very large generations (15,000 words or more). Splitting **Export to Wav** into smaller groups will also reduce memory overhead at the point of exporting your wav files (so good for low memory systems). @@ -600,6 +704,22 @@ Retrieve a list of available voices for generating speech. JSON return: `{"voices": ["voice1.wav", "voice2.wav", "voice3.wav"]}` +#### 🟠 Current Settings Endpoint
+Retrieve a list of available voices for generating speech. + +- URL: `http://127.0.0.1:7851/api/currentsettings`
- Method: `GET`
+ + `curl -X GET "http://127.0.0.1:7851/api/currentsettings"` + + JSON return: ```{"models_available":[{"name":"Coqui","model_name":"API TTS"},{"name":"Coqui","model_name":"API Local"},{"name":"Coqui","model_name":"XTTSv2 Local"}],"current_model_loaded":"XTTSv2 Local","deepspeed_available":true,"deepspeed_status":true,"low_vram_status":true,"finetuned_model":false}``` + + `name & model_name` = listing the currently available models.
+ `current_model_loaded` = what model is currently loaded into VRAM.
+ `deepspeed_available` = was DeepSpeed detected on startup and available to be activated.
+ `deepspeed_status` = If DeepSpeed was detected, is it currently activated.
+ `low_vram_status` = Is Low VRAM currently enabled.
+ `finetuned_model` = Was a finetuned model detected. (XTTSv2 FT).
+ #### 🟠 Preview Voice Endpoint Generate a preview of a specified voice with hardcoded settings. @@ -644,12 +764,12 @@ Generate a preview of a specified voice with hardcoded settings. JSON return `{"status": "lowvram-success"}` -### 🟠 TTS Generation Endpoint +### 🟠 TTS Generation Endpoint (Standard Generation) - URL: `http://127.0.0.1:7851/api/tts-generate`
- Method: `POST`
- Content-Type: `application/x-www-form-urlencoded`
-### 🟠 Example command lines -Standard TTS speech Example (standard text) generating a time-stamped file
+### 🟠 Example command lines (Standard Generation) +Standard TTS generation supports Narration and will generate a wav file/blob. Standard TTS speech Example (standard text) generating a time-stamped file
`curl -X POST "http://127.0.0.1:7851/api/tts-generate" -d "text_input=All of this is text spoken by the character. This is text not inside quotes, though that doesnt matter in the slightest" -d "text_filtering=standard" -d "character_voice_gen=female_01.wav" -d "narrator_enabled=false" -d "narrator_voice_gen=male_01.wav" -d "text_not_inside=character" -d "language=en" -d "output_file_name=myoutputfile" -d "output_file_timestamp=true" -d "autoplay=true" -d "autoplay_volume=0.8"`
@@ -752,7 +872,32 @@ Example JSON TTS Generation Response: `{"status":"generate-success","output_file_path":"C:\\text-generation-webui\\extensions\\alltalk_tts\\outputs\\myoutputfile_1704141936.wav","output_file_url":"http://127.0.0.1:7851/audio/myoutputfile_1704141936.wav","output_cache_url":"http://127.0.0.1:7851/audiocache/myoutputfile_1704141936.wav"}` +### 🟠 Example command lines (Streaming Generation) +Streaming TTS generation does NOT support Narration and will generate an audio stream. Streaming TTS speech JavaScript Example:
+ +- URL: `http://localhost:7851/api/tts-generate-streaming`
- Method: `POST`
- Content-Type: `application/x-www-form-urlencoded`

+ +``` +// Example parameters +const text = "Here is some text"; +const voice = "female_01.wav"; +const language = "en"; +const outputFile = "stream_output.wav"; +// Encode the text for URL +const encodedText = encodeURIComponent(text); +// Create the streaming URL +const streamingUrl = `http://localhost:7851/api/tts-generate-streaming?text=${encodedText}&voice=${voice}&language=${language}&output_file=${outputFile}`; +// Create and play the audio element +const audioElement = new Audio(streamingUrl); +audioElement.play(); // Play the audio stream directly +``` +- **Text (text):** This is the actual text you want to convert to speech. It should be a string and must be URL-encoded to ensure that special characters (like spaces and punctuation) are correctly transmitted in the URL. Example: `Hello World` becomes `Hello%20World` when URL-encoded.
+- **Voice (voice):** This parameter specifies the voice type to be used for the TTS. The value should match one of the available voice options in AllTalks voices folder. This is a string representing the file, like `female_01.wav`.
+- **Language (language):** This setting determines the language in which the text should be spoken. A two-letter language code (like `en` for English, `fr` for French, etc.).
+- **Output File (output_file):** This parameter names the output file where the audio will be streamed. It should be a string representing the file name, such as `stream_output.wav`. AllTalk will not save this as a file in its outputs folder.
+
+ ### 🔴 Future to-do list -- SillyTavern Support - Possibly add some additional TTS engines (TBD). +- Tagging support within text. - Have a break! diff --git a/atsetup.bat b/atsetup.bat new file mode 100644 index 0000000..b996754 --- /dev/null +++ b/atsetup.bat @@ -0,0 +1,503 @@ +@echo off +cd /D "%~dp0" + +set PATH=%PATH%;%SystemRoot%\system32 + +:: Generate the ESC character +for /F %%a in ('echo prompt $E ^| cmd') do set "ESC=%%a" + +:: Standard Colors +set "BLACK=%ESC%[30m" +set "RED=%ESC%[31m" +set "GREEN=%ESC%[32m" +set "YELLOW=%ESC%[33m" +set "BLUE=%ESC%[34m" +set "MAGENTA=%ESC%[35m" +set "CYAN=%ESC%[36m" +set "WHITE=%ESC%[37m" +set "L_BLACK=%ESC%[90m" +set "L_RED=%ESC%[91m" +set "L_GREEN=%ESC%[92m" +set "L_YELLOW=%ESC%[93m" +set "L_BLUE=%ESC%[94m" +set "L_MAGENTA=%ESC%[95m" +set "L_CYAN=%ESC%[96m" +set "L_WHITE=%ESC%[97m" +set "RESET=%ESC%[0m" + +@rem Check if curl is available +curl --version >nul 2>&1 +if "%ERRORLEVEL%" NEQ "0" ( + echo curl is not available on this system. Please install curl then re-run the script https://curl.se/ + echo or perform a manual installation of a Conda Python environment. + goto end +) + +:MainMenu +cls +echo. +echo %L_BLUE%ALLTALK WINDOWS SETUP UTILITY%RESET% +echo. +echo INSTALLATION TYPE +echo 1) I am using AllTalk as part of %L_GREEN%Text-generation-webui%RESET% +echo 2) I am using AllTalk as a %L_GREEN%Standalone Application%RESET% +echo. +echo 9) %L_RED%Exit/Quit%RESET% +echo. +set /p UserOption=" Enter your choice: " + +if "%UserOption%"=="1" goto WebUIMenu +if "%UserOption%"=="2" goto StandaloneMenu +if "%UserOption%"=="9" goto End +goto MainMenu + +:WebUIMenu +cls +echo. +echo %L_BLUE%TEXT-GENERATION-WEBUI SETUP%RESET% +echo. +echo Please ensure you have started your Text-generation-webui Python +echo environment. If you have NOT done this, please run %L_GREEN%cmd_windows.bat%RESET% +echo in the %L_GREEN%text-generation-webui%RESET% folder and then re-run this script. +echo. +echo BASE REQUIREMENTS +echo 1) Install the requirements for a %L_GREEN%Nvidia machine%RESET%. +echo 2) Install the requirements for a %L_GREEN%AMD or MAC machine%RESET%. +echo. +echo OPTIONAL +echo 3) Install the Finetuning requirements. +echo 4) Install DeepSpeed v11.2 for CUDA %L_GREEN%11.8%RESET% and Python-3.11.x. +echo 5) Install DeepSpeed v11.2 for CUDA %L_GREEN%12.1%RESET% and Python-3.11.x. +echo 6) Uninstall DeepSpeed. +echo. +echo. OTHER +echo 7) Generate a diagnostics file. +echo. +echo 9) %L_RED%Exit/Quit%RESET% +echo. +set /p WebUIOption="Enter your choice: " +if "%WebUIOption%"=="1" goto InstallNvidiaTextGen +if "%WebUIOption%"=="2" goto InstallOtherTextGen +if "%WebUIOption%"=="3" goto InstallFinetuneTextGen +if "%WebUIOption%"=="4" goto InstallDeepSpeed118TextGen +if "%WebUIOption%"=="5" goto InstallDeepSpeed121TextGen +if "%WebUIOption%"=="6" goto UnInstallDeepSpeed +if "%WebUIOption%"=="7" goto GenerateDiagsTextGen +if "%WebUIOption%"=="9" goto End +goto WebUIMenu + +:StandaloneMenu +cls +echo. +echo %L_BLUE%ALLTALK STANDALONE APPLICATION SETUP%RESET% +echo. +echo BASE REQUIREMENTS +echo 1) Install AllTalk as a Standalone Application +echo. +echo OPTIONAL +echo 2) Delete AllTalk's custom Python environment +echo 3) Install the Finetuning requirements +echo. +echo. OTHER +echo 4) Generate a diagnostics file +echo. +echo 9) %L_RED%Exit/Quit%RESET% +echo. +set /p StandaloneOption="Enter your choice: " +if "%StandaloneOption%"=="1" goto InstallCustomStandalone +if "%StandaloneOption%"=="2" goto DeleteCustomStandalone +if "%StandaloneOption%"=="3" goto InstallFinetuneStandalone +if "%StandaloneOption%"=="4" goto GenerateDiagsStandalone +if "%StandaloneOption%"=="9" goto End +goto StandaloneMenu + +:InstallNvidiaTextGen +pip install -r requirements_nvidia.txt +if %ERRORLEVEL% neq 0 ( + echo. + echo There was an error installing the Nvidia requirements. + echo Press any key to return to the menu. + echo. + pause + goto WebUIMenu +) +Echo. +echo. +Echo Nvidia machine requirements installed successfully. +Echo. +pause +goto WebUIMenu + +:InstallOtherTextGen +pip install -r requirements_other.txt +if %ERRORLEVEL% neq 0 ( + echo. + echo There was an error installing the Other requirements. + echo Press any key to return to the menu. + echo. + pause + goto WebUIMenu +) +Echo. +echo. +Echo Other machine requirements installed successfully. +Echo. +pause +goto WebUIMenu + +:InstallFinetuneTextGen +pip install -r requirements_finetune.txt +if %ERRORLEVEL% neq 0 ( + echo. + echo There was an error installing the Finetune requirements. + echo Press any key to return to the menu. + echo. + pause + goto WebUIMenu +) +Echo. +Echo Finetune requirements installed successfully. Please ensure you have installed CUDA 11.8. +Echo Finetune needs access to %L_GREEN%cublas64_11.dll%RESET% and will error if this is not setup correctly. +Echo Please see here for details %L_GREEN%https://github.com/erew123/alltalk_tts#-finetuning-a-model%RESET% +Echo. +pause +goto WebUIMenu + +:InstallDeepSpeed118TextGen +echo Downloading DeepSpeed... +curl -LO https://github.com/erew123/alltalk_tts/releases/download/deepspeed/deepspeed-0.11.2+cuda118-cp311-cp311-win_amd64.whl +if %ERRORLEVEL% neq 0 ( + echo. + echo Failed to download DeepSpeed wheel file. + echo Please check your internet connection or try again later. + echo Press any key to return to the menu. + echo. + pause + goto WebUIMenu +) +echo DeepSpeed wheel file downloaded successfully. +echo Installing DeepSpeed... +pip install deepspeed-0.11.2+cuda118-cp311-cp311-win_amd64.whl +if %ERRORLEVEL% neq 0 ( + echo. + echo Failed to install DeepSpeed. + echo Please check if the wheel file is compatible with your system. + echo Press any key to return to the menu. + echo. + pause + goto WebUIMenu +) +Echo. +echo. +Echo DeepSpeed installed successfully. +Echo. +del deepspeed-0.11.2+cuda118-cp311-cp311-win_amd64.whl +pause +goto WebUIMenu + +:InstallDeepSpeed121TextGen +echo Downloading DeepSpeed... +curl -LO https://github.com/erew123/alltalk_tts/releases/download/deepspeed/deepspeed-0.11.2+cuda121-cp311-cp311-win_amd64.whl +if %ERRORLEVEL% neq 0 ( + echo. + echo Failed to download DeepSpeed wheel file. + echo Please check your internet connection or try again later. + echo Press any key to return to the menu. + echo. + pause + goto WebUIMenu +) +echo DeepSpeed wheel file downloaded successfully. +echo Installing DeepSpeed... +pip install deepspeed-0.11.2+cuda121-cp311-cp311-win_amd64.whl +if %ERRORLEVEL% neq 0 ( + echo. + echo Failed to install DeepSpeed. + echo Please check if the wheel file is compatible with your system. + echo Press any key to return to the menu. + echo. + pause + goto WebUIMenu +) +Echo. +echo. +Echo DeepSpeed installed successfully. +Echo. +del deepspeed-0.11.2+cuda121-cp311-cp311-win_amd64.whl +pause +goto WebUIMenu + +:UnInstallDeepSpeed +pip uninstall deepspeed +if %ERRORLEVEL% neq 0 ( + echo. + echo There was an error uninstalling DeepSpeed. + echo Press any key to return to the menu. + echo. + pause + goto WebUIMenu +) +echo. +Echo DeepSpeed uninstalled successfully. +Echo. +pause +goto WebUIMenu + +:GenerateDiagsTextGen +Python diagnostics.py +if %ERRORLEVEL% neq 0 ( + echo. + echo There was an error running diagnostics. Have you correctly started your + echo Text-generation-webui Python environment with %L_GREEN%cmd_windows.bat%RESET%? + echo. + pause + goto WebUIMenu +) +Echo. +echo. +Echo Diagnostics.log generated. Please scroll up to look over the log. +Echo. +pause +goto WebUIMenu + +:InstallCustomStandalone +set PATH=%PATH%;%SystemRoot%\system32 + +@rem Check if curl is available +curl --version >nul 2>&1 +if "%ERRORLEVEL%" NEQ "0" ( + echo curl is not available on this system. Please install curl then re-run the script https://curl.se/ + echo or perform a manual installation of a Conda Python environment. + goto end +) +echo "%CD%"| findstr /C:" " >nul && echo This script relies on Miniconda which can not be silently installed under a path with spaces. Please correct your folder names and re-try installation. && goto exit + +@rem Check for special characters in installation path +set "SPCHARMESSAGE=WARNING: Special characters were detected in the installation path! This can cause the installation to fail!" +echo "%CD%"| findstr /R /C:"[!#\$%&()\*+,;<=>?@\[\]\^`{|}~]" >nul && ( + call :PrintBigMessage %SPCHARMESSAGE% +) +set SPCHARMESSAGE= + +@rem fix failed install when installing to a separate drive +set TMP=%cd%\alltalk_environment +set TEMP=%cd%\alltalk_environment + +@rem deactivate existing conda envs as needed to avoid conflicts +(call conda deactivate && call conda deactivate && call conda deactivate) 2>nul + +@rem config +set INSTALL_DIR=%cd%\alltalk_environment +set CONDA_ROOT_PREFIX=%cd%\alltalk_environment\conda +set INSTALL_ENV_DIR=%cd%\alltalk_environment\env +set MINICONDA_DOWNLOAD_URL=https://repo.anaconda.com/miniconda/Miniconda3-py310_23.3.1-0-Windows-x86_64.exe +set conda_exists=F + +@rem figure out whether git and conda need to be installed +call "%CONDA_ROOT_PREFIX%\_conda.exe" --version >nul 2>&1 +if "%ERRORLEVEL%" EQU "0" ( + set conda_exists=T + call "%CONDA_ROOT_PREFIX%\condabin\conda.bat" activate "%INSTALL_ENV_DIR%" + goto RunScript +) + +@rem download and install conda if not installed +echo Downloading Miniconda from %MINICONDA_DOWNLOAD_URL% to %INSTALL_DIR%\miniconda_installer.exe +mkdir "%INSTALL_DIR%" +call curl -Lk "%MINICONDA_DOWNLOAD_URL%" > "%INSTALL_DIR%\miniconda_installer.exe" || ( echo. && echo Miniconda failed to download. && goto end ) +echo Installing Miniconda to %CONDA_ROOT_PREFIX% +start /wait "" "%INSTALL_DIR%\miniconda_installer.exe" /InstallationType=JustMe /NoShortcuts=1 /AddToPath=0 /RegisterPython=0 /NoRegistry=1 /S /D=%CONDA_ROOT_PREFIX% +echo Miniconda version: +call "%CONDA_ROOT_PREFIX%\_conda.exe" --version || ( echo. && echo Miniconda not found. && goto end ) + +@rem create the installer env +echo Packages to install: %PACKAGES_TO_INSTALL% +call "%CONDA_ROOT_PREFIX%\_conda.exe" create --no-shortcuts -y -k --prefix "%INSTALL_ENV_DIR%" python=3.11 || ( echo. && echo Conda environment creation failed. && goto end ) + +@rem check if conda environment was actually created +if not exist "%INSTALL_ENV_DIR%\python.exe" ( echo. && echo Conda environment is empty. && goto end ) + +@rem activate installer env +call "%CONDA_ROOT_PREFIX%\condabin\conda.bat" activate "%INSTALL_ENV_DIR%" || ( echo. && echo Miniconda hook not found. && goto end ) + +@rem Ask user for the type of requirements to install +echo. +echo %L_BLUE%Choose the type of requirements to install%RESET% +echo. +echo 1. Nvidia graphics card machines +echo 2. Other machines (mac, amd, etc) +echo. +set /p UserChoice="Enter your choice (1 or 2): " + +@rem Install requirements based on user choice +if "%UserChoice%" == "1" ( + echo. + echo Installing Nvidia requirements... + echo Downloading Pytorch with CUDA.. + pip install torch==2.1.0 torchvision==0.16.0 torchaudio==2.1.0 --upgrade --force-reinstall --index-url https://download.pytorch.org/whl/cu121 + pip install numpy==1.24.4 + pip install typing-extensions>=4.8.0 + pip install networkx<3.0.0,>=2.5.0 + pip install fsspec>=2023.6.0 + pip install soundfile==0.12.1 + pip install uvicorn==0.24.0.post1 + pip install TTS==0.21.3 + pip install fastapi==0.104.1 + pip install Jinja2==3.1.2 + pip install requests==2.31.0 + pip install tqdm==4.66.1 + pip install importlib-metadata==4.8.1 + pip install packaging==23.2 + pip install pydantic==1.10.13 + pip install sounddevice==0.4.6 + pip install python-multipart==0.0.6 + echo Downloading DeepSpeed... + curl -LO https://github.com/erew123/alltalk_tts/releases/download/DeepSpeed-12.7/deepspeed-0.12.7+d058d4b-cp311-cp311-win_amd64.whl + echo Installing DeepSpeed... + pip install deepspeed-0.12.7+d058d4b-cp311-cp311-win_amd64.whl + del deepspeed-0.12.7+d058d4b-cp311-cp311-win_amd64.whl +) else ( + echo Installing requirements for other machines... + pip install -r requirements_other.txt +) + +@rem Create start_environment.bat to run AllTalk environment +echo @echo off > start_environment.bat +echo cd /D "%~dp0" >> start_environment.bat +echo set CONDA_ROOT_PREFIX=%cd%\alltalk_environment\conda >> start_environment.bat +echo set INSTALL_ENV_DIR=%cd%\alltalk_environment\env >> start_environment.bat +echo call "%CONDA_ROOT_PREFIX%\condabin\conda.bat" activate "%INSTALL_ENV_DIR%" >> start_environment.bat +echo exit >> start_environment.bat +@rem Create start_alltalk.bat to run AllTalk +echo @echo off > start_alltalk.bat +echo cd /D "%~dp0" >> start_alltalk.bat +echo set CONDA_ROOT_PREFIX=%cd%\alltalk_environment\conda >> start_alltalk.bat +echo set INSTALL_ENV_DIR=%cd%\alltalk_environment\env >> start_alltalk.bat +echo call "%CONDA_ROOT_PREFIX%\condabin\conda.bat" activate "%INSTALL_ENV_DIR%" >> start_alltalk.bat +echo call python script.py >> start_alltalk.bat +echo exit >> start_alltalk.bat +Echo. +Echo start_alltalk.bat has been created. +Echo You can now start AllTalk with %L_YELLOW%start_alltalk.bat%RESET% +Echo. +pause +goto StandaloneMenu + +:DeleteCustomStandalone +@rem Check if the alltalk_environment directory exists +if not exist "%cd%\alltalk_environment\" ( + echo. + echo %L_GREEN%alltalk_environment%RESET% directory does not exist. No need to delete. + echo. + pause + goto StandaloneMenu +) +@rem Check if a Conda environment is active +if not defined CONDA_PREFIX goto NoCondaEnvDeleteCustomStandalone +@rem Deactivate Conda environment if it's active +Echo Exiting the Conda Environment. Please run %L_GREEN%atsetup.bat%RESET% again and delete the environment. +conda deactivate +:NoCondaEnvDeleteCustomStandalone +echo Deleting "alltalk_environment". Please wait. +rd /s /q "alltalk_environment" +del start_alltalk.bat +del start_environment.bat +if %ERRORLEVEL% neq 0 ( + echo. + echo Failed to delete alltalk_environment folder. + echo Please make sure it is not in use and try again. + echo. + pause + goto StandaloneMenu +) +echo. +Echo. +echo Environment %L_GREEN%alltalk_environment%RESET% deleted. Please set up the environment again. +Echo. +pause +goto StandaloneMenu + +:InstallFinetuneStandalone +cd /D "%~dp0" +set CONDA_ROOT_PREFIX=%cd%\alltalk_environment\conda +set INSTALL_ENV_DIR=%cd%\alltalk_environment\env +@rem Check if the Conda environment exists +if not exist "%INSTALL_ENV_DIR%\python.exe" ( + echo. + echo The Conda environment at "%INSTALL_ENV_DIR%" does not exist. + echo Please install the environment before proceeding. + echo. + pause + goto StandaloneMenu +) +@rem Attempt to activate the Conda environment +call "%CONDA_ROOT_PREFIX%\condabin\conda.bat" activate "%INSTALL_ENV_DIR%" +if errorlevel 1 ( + echo. + echo Failed to activate the Conda environment. + echo Please check your installation and try again. + echo. + pause + goto StandaloneMenu +) +@rem Proceed with installing requirements +pip install -r requirements_finetune.txt +if %ERRORLEVEL% neq 0 ( + echo. + echo There was an error installing the Finetune requirements. + echo Press any key to return to the menu. + echo. + pause + goto StandaloneMenu +) +Echo. +Echo Finetune requirements installed successfully. Please ensure you have installed CUDA 11.8. +Echo Finetune needs access to %L_GREEN%cublas64_11.dll%RESET% and will error if this is not setup correctly. +Echo Please see here for details %L_GREEN%https://github.com/erew123/alltalk_tts#-finetuning-a-model%RESET% +Echo. +pause +goto StandaloneMenu + +:GenerateDiagsStandalone +cd /D "%~dp0" +set CONDA_ROOT_PREFIX=%cd%\alltalk_environment\conda +set INSTALL_ENV_DIR=%cd%\alltalk_environment\env +@rem Check if the Conda environment exists +if not exist "%INSTALL_ENV_DIR%\python.exe" ( + echo. + echo The Conda environment at "%INSTALL_ENV_DIR%" does not exist. + echo Please install the environment before proceeding. + echo. + pause + goto StandaloneMenu +) +@rem Attempt to activate the Conda environment +call "%CONDA_ROOT_PREFIX%\condabin\conda.bat" activate "%INSTALL_ENV_DIR%" +if errorlevel 1 ( + echo. + echo Failed to activate the Conda environment. + echo Please check your installation and try again. + echo. + pause + goto StandaloneMenu +) +@rem Run diagnostics +Python diagnostics.py +if %ERRORLEVEL% neq 0 ( + echo. + echo There was an error running diagnostics. + echo Press any key to return to the menu. + echo. + pause + goto StandaloneMenu +) +Echo. +Echo. +Echo Diagnostics.log generated. Please scroll up to look over the log. +Echo. +pause +goto StandaloneMenu + +:End +echo Exiting AllTalk Setup Utility... +exit /b diff --git a/atsetup.sh b/atsetup.sh new file mode 100644 index 0000000..3700b82 --- /dev/null +++ b/atsetup.sh @@ -0,0 +1,440 @@ +#!/bin/bash + +# ANSI color codes +RED='\033[1;31m' +GREEN='\033[1;32m' +YELLOW='\033[1;33m' +BLUE='\033[1;34m' +CYAN='\033[1;36m' +MAGENTA='\033[1;35m' +WHITE='\033[1;37m' +L_RED='\033[1;91m' +L_GREEN='\033[1;92m' +L_YELLOW='\033[1;93m' +L_BLUE='\033[1;94m' +L_CYAN='\033[1;96m' +L_MAGENTA='\033[1;95m' +NC='\033[0m' # No Color + +# Navigate to the script's directory +cd "$(dirname "$0")" + +# Function to check if curl is installed +check_curl() { + if ! command -v curl &> /dev/null; then + echo "curl is not available on this system. Please install curl then re-run the script https://curl.se/" + echo "or perform a manual installation of a Conda Python environment." + exit 1 + fi +} + +# Main Menu +main_menu() { + while true; do + clear + echo + echo -e " ${L_BLUE}ALLTALK LINUX SETUP UTILITY${NC}" + echo + echo " INSTALLATION TYPE" + echo -e " 1) I am using AllTalk as part of ${L_GREEN}Text-generation-webui${NC}" + echo -e " 2) I am using AllTalk as a ${L_GREEN}Standalone Application${NC}" + echo + echo -e " 9)${L_RED} Exit/Quit${NC}" + echo + read -p " Enter your choice: " user_option + + case $user_option in + 1) webui_menu ;; + 2) standalone_menu ;; + 9) exit 0 ;; + *) echo "Invalid option"; sleep 2 ;; + esac + done +} + +# Text-generation-webui Menu +webui_menu() { + while true; do + clear + echo + echo -e " ${L_BLUE}TEXT-GENERATION-WEBUI SETUP${NC}" + echo + echo -e " Please ensure you have started your Text-generation-webui Python" + echo -e " environment. If you have NOT done this, please run ${L_GREEN}/cmd_linux.sh${NC}" + echo -e " in the ${L_GREEN}text-generation-webui${NC} folder and then re-run this script." + echo + echo " BASE REQUIREMENTS" + echo -e " 1) Install the AllTalk requirements for an ${L_GREEN}Nvidia GPU machine.${NC}" + echo -e " 2) Install the AllTalk requirements for an ${L_GREEN}AMD or MAC machine.${NC}" + echo + echo " OPTIONAL" + echo " 3) Install the Finetuning requirements." + echo " 4) Install DeepSpeed." + echo " 5) Uninstall DeepSpeed." + echo + echo " OTHER" + echo " 6) Generate a diagnostics file." + echo + echo -e " 9)${L_RED} Exit/Quit${NC}" + echo + read -p " Enter your choice: " webui_option + + case $webui_option in + 1) install_nvidia_textgen ;; + 2) install_other_textgen ;; + 3) install_finetune_textgen ;; + 4) install_deepspeed ;; + 5) uninstall_deepspeed ;; + 6) generate_diagnostics_textgen ;; + 9) exit 0 ;; + *) echo "Invalid option"; sleep 2 ;; + esac + done +} + +install_nvidia_textgen() { + local requirements_file="requirements_nvidia.txt" + echo " Installing Finetune requirements from $requirements_file..." + if ! pip install -r "$requirements_file"; then + echo " There was an error installing the Nvidia requirements." + echo " Please check the output for details." + read -p " Press any key to return to the menu. " -n 1 + echo + return + fi + echo " Nvidia requirements installed successfully." + read -p " Press any key to continue. " -n 1 + echo +} + +install_other_textgen() { + local requirements_file="requirements_other.txt" + echo " Installing Other machine requirements from $requirements_file..." + if ! pip install -r "$requirements_file"; then + echo " There was an error installing the Other requirements." + echo " Please check the output for details." + read -p " Press any key to return to the menu. " -n 1 + echo + return + fi + echo " Other machine requirements installed successfully." + read -p " Press any key to continue. " -n 1 + echo +} + +install_finetune_textgen() { + local requirements_file="requirements_finetune.txt" + echo " Installing Finetune requirements from $requirements_file..." + if ! pip install -r "$requirements_file"; then + echo " There was an error installing the Finetune requirements." + echo " Please check the output for details." + read -p " Press any key to return to the menu. " -n 1 + echo + return + fi + echo + echo " Finetune base requirements installed successfully." + echo + echo -e " Finetuning requires access to ${GREEN}Cublas${NC} from the ${GREEN}Nvidia CUDA Toolkit${NC}, so will need installing." + echo + echo -e " Instructions can be found here:" + echo -e " ${GREEN}https://github.com/erew123/alltalk_tts?#-finetuning-a-model${NC}" + echo + read -p " Press any key to continue. " -n 1 + echo +} + +# Function to install DeepSpeed +install_deepspeed() { + clear + echo + echo -e " ${L_BLUE}DEEPSPEED INSTALLATION REQUIREMENTS${NC}" + echo + echo -e " - NVIDIA CUDA Toolkit must be installed from your ${L_GREEN}Linux package manager${NC}" + echo -e " or ${L_GREEN}https://developer.nvidia.com/cuda-toolkit-archive${NC}" + echo -e " - The environment variable ${L_GREEN}CUDA_HOME${NC} must be set in your Python env" + echo -e " e.g. export CUDA_HOME=/usr/local/cuda." + echo -e " - ${L_GREEN}libaio-dev${NC} must be installed." + echo + echo -e " See the following link for full instructions:" + echo -e " ${L_GREEN}https://github.com/erew123/alltalk_tts?#-deepspeed-installation-options${NC}" + echo + echo -e " ${L_BLUE}Please ${L_RED}confirm${L_BLUE} that you have completed these steps before continuing" + echo -e " with the DeepSpeed installation, or the install ${L_RED}will fail.${NC}" + echo + read -p " Have you completed all the above steps? (y/n): " confirm + + if [ "$confirm" != "y" ]; then + echo -e " ${RED}DeepSpeed installation cannot proceed without completing the prerequisites.${NC}" + return + fi + + echo -e "\n ${GREEN}Proceeding with DeepSpeed installation...${NC}" + pip install deepspeed + if [ $? -ne 0 ]; then + echo -e " ${RED}There was an error installing DeepSpeed.${NC}" + return + fi + + echo -e " ${GREEN}DeepSpeed installed successfully.${NC}" + read -p " Press any key to continue. " -n 1 + echo +} + +uninstall_deepspeed() { + echo "Uninstalling DeepSpeed..." + pip uninstall -y deepspeed + if [ $? -ne 0 ]; then + echo " There was an error uninstalling DeepSpeed." + echo " Press any key to return to the menu." + read -n 1 + return + fi + echo " DeepSpeed uninstalled successfully." + echo " Press any key to continue." + read -n 1 +} + +generate_diagnostics_textgen() { + # Run diagnostics + if ! python diagnostics.py; then + echo " There was an error running diagnostics." + read -p " Press any key to return to the menu. " -n 1 + echo + return + fi + + echo " Diagnostics completed successfully." + read -p " Press any key to continue. " -n 1 + echo +} + +# Standalone Menu +standalone_menu() { + while true; do + clear + echo + echo -e " ${L_BLUE}ALLTALK STANDALONE APPLICATION SETUP${NC}" + echo + echo " BASE REQUIREMENTS" + echo " 1) Install AllTalk as a Standalone Application" + echo + echo " OPTIONAL" + echo " 2) Delete AllTalk's custom Python environment" + echo " 3) Install the Finetuning requirements" + echo " 4) Install DeepSpeed." + echo + echo " OTHER" + echo " 5) Generate a diagnostics file" + echo + echo -e " 9)${L_RED} Exit/Quit${NC}" + echo + read -p " Enter your choice: " standalone_option + + case $standalone_option in + 1) install_custom_standalone ;; + 2) delete_custom_standalone ;; + 3) install_finetune_standalone ;; + 4) install_deepspeed ;; + 5) generate_diagnostics_standalone ;; + 9) exit 0 ;; + *) echo "Invalid option"; sleep 2 ;; + esac + done +} + +install_custom_standalone() { + cd "$(dirname "${BASH_SOURCE[0]}")" + + if [[ "$(pwd)" =~ " " ]]; then + echo "This script relies on Miniconda which can not be silently installed under a path with spaces." + exit + fi + + # Deactivate existing conda envs as needed to avoid conflicts + { conda deactivate && conda deactivate && conda deactivate; } 2> /dev/null + + OS_ARCH=$(uname -m) + case "${OS_ARCH}" in + x86_64*) OS_ARCH="x86_64" ;; + arm64* | aarch64*) OS_ARCH="aarch64" ;; + *) echo "Unknown system architecture: $OS_ARCH! This script runs only on x86_64 or arm64" && exit ;; + esac + + # Config + INSTALL_DIR="$(pwd)/alltalk_environment" + CONDA_ROOT_PREFIX="${INSTALL_DIR}/conda" + INSTALL_ENV_DIR="${INSTALL_DIR}/env" + MINICONDA_DOWNLOAD_URL="https://repo.anaconda.com/miniconda/Miniconda3-py310_23.3.1-0-Linux-${OS_ARCH}.sh" + + if [ ! -x "${CONDA_ROOT_PREFIX}/bin/conda" ]; then + echo "Downloading Miniconda from $MINICONDA_DOWNLOAD_URL to ${INSTALL_DIR}/miniconda_installer.sh" + mkdir -p "${INSTALL_DIR}" + curl -L "${MINICONDA_DOWNLOAD_URL}" -o "${INSTALL_DIR}/miniconda_installer.sh" + chmod +x "${INSTALL_DIR}/miniconda_installer.sh" + bash "${INSTALL_DIR}/miniconda_installer.sh" -b -p "${CONDA_ROOT_PREFIX}" + echo "Miniconda installed." + fi + + if [ ! -d "${INSTALL_ENV_DIR}" ]; then + "${CONDA_ROOT_PREFIX}/bin/conda" create -y --prefix "${INSTALL_ENV_DIR}" python=3.11 + echo "Conda environment created at ${INSTALL_ENV_DIR}." + fi + + # Activate the environment and install requirements + source "${CONDA_ROOT_PREFIX}/etc/profile.d/conda.sh" + conda activate "${INSTALL_ENV_DIR}" + + echo + echo -e " ${L_BLUE}Choose the type of requirements to install${NC}" + echo + echo " 1. Nvidia graphics card machines" + echo " 2. Other machines (mac, amd, etc)" + echo + read -p " Enter your choice (1 or 2): " user_choice + + if [ "$user_choice" == "1" ]; then + pip install torch==2.1.0 torchvision==0.16.0 torchaudio==2.1.0 --upgrade --force-reinstall --index-url https://download.pytorch.org/whl/cu121 + pip install numpy==1.24.4 + pip install typing-extensions>=4.8.0 + pip install networkx<3.0.0,>=2.5.0 + pip install fsspec>=2023.6.0 + pip install soundfile==0.12.1 + pip install uvicorn==0.24.0.post1 + pip install TTS==0.21.3 + pip install fastapi==0.104.1 + pip install Jinja2==3.1.2 + pip install requests==2.31.0 + pip install tqdm==4.66.1 + pip install importlib-metadata==4.8.1 + pip install packaging==23.2 + pip install pydantic==1.10.13 + pip install sounddevice==0.4.6 + pip install python-multipart==0.0.6 + else + pip install -r requirements_other.txt + fi + # Create start_environment.sh to run AllTalk + cat << EOF > start_environment.sh +#!/bin/bash +cd "$(dirname "${BASH_SOURCE[0]}")" +if [[ "$(pwd)" =~ " " ]]; then echo This script relies on Miniconda which can not be silently installed under a path with spaces. && exit; fi +# deactivate existing conda envs as needed to avoid conflicts +{ conda deactivate && conda deactivate && conda deactivate; } 2> /dev/null +# config +CONDA_ROOT_PREFIX="$(pwd)/alltalk_environment/conda" +INSTALL_ENV_DIR="$(pwd)/alltalk_environment/env" +# environment isolation +export PYTHONNOUSERSITE=1 +unset PYTHONPATH +unset PYTHONHOME +export CUDA_PATH="$INSTALL_ENV_DIR" +export CUDA_HOME="$CUDA_PATH" +# activate env +bash --init-file <(echo "source \"$CONDA_ROOT_PREFIX/etc/profile.d/conda.sh\" && conda activate \"$INSTALL_ENV_DIR\"") +EOF + # Create start_alltalk.sh to run AllTalk + cat << EOF > start_alltalk.sh +#!/bin/bash +source "${CONDA_ROOT_PREFIX}/etc/profile.d/conda.sh" +conda activate "${INSTALL_ENV_DIR}" +python script.py +EOF + chmod +x start_alltalk.sh + chmod +x start_environment.sh + echo + echo + echo -e " start_alltalk.sh has been created." + echo -e " You can now start AllTalk with ${L_YELLOW}./start_alltalk.sh${NC}" + echo + read -p " Press any key to continue. " -n 1 +} + + +delete_custom_standalone() { + local env_dir="$PWD/alltalk_environment" + # Check if the alltalk_environment directory exists + if [ ! -d "$env_dir" ]; then + echo " \"$env_dir\" directory does not exist. No need to delete." + read -p " Press any key to continue. " -n 1 + echo + return + fi + # Check if a Conda environment is active and deactivate it + if [ -n "$CONDA_PREFIX" ]; then + echo " Exiting the Conda environment. You may need to start ./atstart.sh again" + conda deactivate + fi + echo "Deleting \"$env_dir\". Please wait." + rm -rf "$env_dir" + if [ -d "$env_dir" ]; then + echo " Failed to delete \"$env_dir\" folder." + echo " Please make sure it is not in use and try again." + else + echo " Environment \"$env_dir\" deleted successfully." + fi + read -p " Press any key to continue. " -n 1 + echo +} + +install_finetune_standalone() { + local env_dir="$PWD/alltalk_environment" + local conda_root_prefix="${env_dir}/conda" + local install_env_dir="${env_dir}/env" + if [ ! -d "${install_env_dir}" ]; then + echo " The Conda environment at '${install_env_dir}' does not exist." + echo " Please install the environment before proceeding." + read -p " Press any key to return to the menu. " -n 1 + echo + return + fi + source "${conda_root_prefix}/etc/profile.d/conda.sh" + conda activate "${install_env_dir}" + local requirements_file="requirements_finetune.txt" + echo "Installing Finetune requirements from $requirements_file..." + if ! pip install -r "$requirements_file"; then + echo " There was an error installing the Finetune requirements." + echo " Please check the output for details." + read -p " Press any key to return to the menu. " -n 1 + echo + return + fi + echo + echo " Finetune base requirements installed successfully." + echo + echo -e " Finetuning requires access to ${GREEN}Cublas${NC} from the ${GREEN}Nvidia CUDA Toolkit${NC}, so will need installing." + echo + echo -e " Instructions can be found here:" + echo -e " ${GREEN}https://github.com/erew123/alltalk_tts?#-finetuning-a-model${NC}" + echo + read -p " Press any key to continue. " -n 1 + echo +} + +generate_diagnostics_standalone() { + local env_dir="$PWD/alltalk_environment" + local conda_root_prefix="${env_dir}/conda" + local install_env_dir="${env_dir}/env" + if [ ! -d "${install_env_dir}" ]; then + echo " The Conda environment at '${install_env_dir}' does not exist." + echo " Please install the environment before proceeding." + read -p " Press any key to return to the menu. " -n 1 + echo + return + fi + source "${conda_root_prefix}/etc/profile.d/conda.sh" + conda activate "${install_env_dir}" + if ! python diagnostics.py; then + echo " There was an error running diagnostics." + read -p " Press any key to return to the menu. " -n 1 + echo + return + fi + echo " Diagnostics completed successfully." + read -p " Press any key to continue. " -n 1 + echo +} + + +# Start the main menu +main_menu diff --git a/diagnostics.py b/diagnostics.py index ca5466a..eca4ad4 100644 --- a/diagnostics.py +++ b/diagnostics.py @@ -215,6 +215,13 @@ def log_system_info(): # Print colored output print(f"{package_name.ljust(max_package_length)} Required: {color_required}{operator} {required_version.ljust(12)}\033[0m Installed: {color_installed}{installed_version}\033[0m") + + print("\nOn Nvidia Graphics cards machines, if your \033[92mInstalled\033[0m version of \033[92mTorch\033[0m and \033[92mTorchaudio\033[0m does") + print("not have \033[92m+cu118\033[0m (Cuda 11.8) or \033[92m+cu121\033[0m (Cuda 12.1) listed after them, you do not have CUDA") + print("installed for Torch or Torchaudio in this Python environment. This will cause you problems") + print("with \033[94mAllTalk\033[0m and \033[94mFinetuning.\033[0m You may have to 'pip install' a new version of torch and") + print("torchaudio, using '\033[94m--upgrade --force-reinstall\033[0m' with the correct version of PyTorch for\033[0m") + print("your Python environment.\033[0m") print("\033[94m\nRequirements file specifier meanings:\033[0m") explanation = textwrap.dedent(""" == Exact version != Any version except < Less than diff --git a/finetune.py b/finetune.py index 75aba76..5583137 100644 --- a/finetune.py +++ b/finetune.py @@ -917,7 +917,9 @@ if __name__ == "__main__": ####      - Place your audio files in {str(audio_folder)} ####      - Your audio samples can be in the format mp3, wav, or flac. ####      - You will need a minimum of 2 minutes of audio in either one or multiple audio files. Very small sample files cause errors, so I would suggest 30 seconds and longer samples. - ####      - When you have completed Steps 1, 2, and 3, you are welcome to delete your samples from "put-voice-samples-in-here".
+ ####      - When you have completed Steps 1, 2, and 3, you are welcome to delete your samples from "put-voice-samples-in-here". + ####      - FYI Anecdotal evidence suggests that the Whisper 2 model may yield superior results in audio splitting and dataset creation. + ####      - If this step is failing, it will be worth running the diagnostics with atsetup and confirming you have cu118 or cu112 listed against your torch and torchaudio.
### 🟨 What this step is doing ####      - With step one, we are going to be stripping your audio file(s) into smaller files, using Whisper to find spoken words/sentences, compile that into excel sheets of training data, ready for finetuning the model on Step 2. ####      - Whilst you can choose multiple Whisper models, its best only to use the 1 model as each one is about 3GB in size and will download to your local huggingface cache on first-time use. If and when you have completed training, you wish to delete this 3GB model from your system, you are welcome to do so. diff --git a/script.py b/script.py index 81b1f96..677dd2d 100644 --- a/script.py +++ b/script.py @@ -237,10 +237,12 @@ process = subprocess.Popen(["python", script_path]) # Check if the subprocess has started successfully if process.poll() is None: print(f"[{params['branding']}Startup] TTS Subprocess starting") + print(f"[{params['branding']}Startup]") print( - f"[{params['branding']}Startup] Readme available here:", - f"http://{params['ip_address']}:{params['port_number']}", + f"[{params['branding']}Startup] \033[94mSettings & Documentation:\033[00m", + f"\033[92mhttp://{params['ip_address']}:{params['port_number']}\033[00m", ) + print(f"[{params['branding']}Startup]") else: print( f"[{params['branding']}Startup] \033[91mWarning\033[0m TTS Subprocess Webserver failing to start process" @@ -862,7 +864,8 @@ def ui(): voice.change(lambda x: params.update({"voice": x}), voice, None) language.change(lambda x: params.update({"language": x}), language, None) - # TTS Settings (Not yet parsed to api/implemented) + + # TS Settings (Not yet parsed to api/implemented) # local_temperature_gr.change(lambda x: params.update({"local_temperature": x}), local_temperature_gr, None) # local_repetition_penalty_gr.change(lambda x: params.update({"local_repetition_penalty": x}), local_repetition_penalty_gr, None) @@ -885,4 +888,4 @@ if running_in_standalone: try: time.sleep(1) # Add a small delay to avoid high CPU usage except KeyboardInterrupt: - break # Allow graceful exit on Ctrl+C + break # Allow graceful exit on Ctrl+C \ No newline at end of file diff --git a/templates/STfiles/alltalk.js b/templates/STfiles/alltalk.js new file mode 100644 index 0000000..e44b312 --- /dev/null +++ b/templates/STfiles/alltalk.js @@ -0,0 +1,938 @@ +import { doExtrasFetch, getApiUrl, modules } from '../../extensions.js'; +import { saveTtsProviderSettings } from './index.js'; + +export { AllTalkTtsProvider }; + +const css = ` +.settings-row { + display: flex; + justify-content: space-between; /* Distribute space evenly between the children */ + align-items: center; + width: 100%; /* Full width to spread out the children */ +} + +.settings-option { + flex: 1; /* Each child will take equal space */ + margin: 0 10px; /* Adjust spacing as needed */ +} + +.endpoint-option { + flex: 1; /* Each child will take equal space */ + margin: 0 10px; /* Adjust spacing as needed */ + margin-right: 25px; + width: 38%; /* Full width to spread out the children */ +} + +.website-row { + display: flex; + justify-content: start; /* Aligns items to the start of the row */ + align-items: center; + margin-top: 10px; /* Top margin */ + margin-bottom: 10px; /* Bottom margin */ +} + +.website-option { + flex: 1; /* Allows each option to take equal space */ + margin-right: 10px; /* Spacing between the two options, adjust as needed */ + margin-left: 10px; /* Adjust this value to align as desired */ + /* Remove the individual margin top and bottom declarations if they are set globally on the .settings-row */ +} + +.settings-separator { + margin-top: 10px; + margin-bottom: 10px; + padding: 18x; + font-weight: bold; + border-top: 1px solid #e1e1e1; /* Grey line */ + border-bottom: 1px solid #e1e1e1; /* Grey line */ + text-align: center; +} + +.status-message { + flex: 1; /* Adjust as needed */ + margin: 0 10px; + /* Additional styling */ +} + +.model-endpoint-row { + display: flex; + justify-content: space-between; + align-items: center; + width: 100%; +} + +.model-option, .endpoint-option { + flex: 1; + margin: 0 10px; + margin-left: 10px; /* Adjust this value to align as desired */ + /* Adjust width as needed to fit in one line */ +} + +.endpoint-option { + /* Adjust width to align with model dropdown */ + width: 38%; +} + +#status_info { + color: lightgreen; /* Or any specific shade of green you prefer */ +} +`; + +const style = document.createElement('style'); +style.type = 'text/css'; +style.appendChild(document.createTextNode(css)); +document.head.appendChild(style); + +class AllTalkTtsProvider { + //########// + // Config // + //########// + + settings; + ready = false; + voices = []; + separator = '. '; + audioElement = document.createElement('audio'); + + languageLabels = { + 'Arabic': 'ar', + 'Brazilian Portuguese': 'pt', + 'Chinese': 'zh-cn', + 'Czech': 'cs', + 'Dutch': 'nl', + 'English': 'en', + 'French': 'fr', + 'German': 'de', + 'Italian': 'it', + 'Polish': 'pl', + 'Russian': 'ru', + 'Spanish': 'es', + 'Turkish': 'tr', + 'Japanese': 'ja', + 'Korean': 'ko', + 'Hungarian': 'hu', + 'Hindi': 'hi', + }; + + Settings = { + provider_endpoint: 'http://localhost:7851', + language: 'en', + voiceMap: {}, + at_generation_method: 'standard_generation', + narrator_enabled: 'false', + at_narrator_text_not_inside: 'narrator', + narrator_voice_gen: 'female_01.wav', + finetuned_model: 'false' + }; + + get settingsHtml() { + let html = `
AllTalk Settings
`; + + html += `
+ +
+ + +
+
`; + + html += `
+ +
+ + +
+ +
+ + +
+ +
`; + + html += `
+
+ + +
+
+ + +
+
`; + + + html += `
+
+ + +
+ +
+ + +
+
`; + + + html += `
+
+ + +
+
+ + +
+
+ Status: Ready +
+
+ +
+
`; + + + html += `
+
+ AllTalk Config & Docs. +
+ +
+ AllTalk Website. +
+
`; + +html += `
+
+Text-generation-webui users - Uncheck Enable TTS in Text-generation-webui. +
+
`; + + + + return html; + } + + + //#################// + // Startup ST & AT // + //#################// + + async loadSettings(settings) { + updateStatus('Offline'); + // Use default settings if no settings are provided + this.settings = Object.keys(settings).length === 0 ? this.Settings : settings; + + try { + // Check if TTS provider is ready + await this.checkReady(); + await this.fetchTtsVoiceObjects(); // Fetch voices only if service is ready + await this.updateSettingsFromServer(); + this.updateNarratorVoicesDropdown(); + this.updateLanguageDropdown(); + this.setupEventListeners(); + this.applySettingsToHTML(); + updateStatus('Ready'); + } catch (error) { + console.error("Error loading settings:", error); + // Set status to Error if there is an issue in loading settings + updateStatus('Offline'); + } + } + + applySettingsToHTML() { + // Load AllTalk specific settings first + const loadedSettings = loadAllTalkSettings(); + // Apply loaded settings or use defaults + this.settings = loadedSettings ? { ...this.Settings, ...loadedSettings } : this.Settings; + const narratorVoiceSelect = document.getElementById('narrator_voice'); + const atNarratorSelect = document.getElementById('at_narrator_enabled'); + const textNotInsideSelect = document.getElementById('at_narrator_text_not_inside'); + const generationMethodSelect = document.getElementById('at_generation_method'); + // Apply settings to Narrator Voice dropdown + if (narratorVoiceSelect && this.settings.narrator_voice) { + narratorVoiceSelect.value = this.settings.narrator_voice.replace('.wav', ''); + this.settings.narrator_voice_gen = this.settings.narrator_voice; + //console.log(this.settings.narrator_voice_gen) + } + // Apply settings to AT Narrator Enabled dropdown + if (atNarratorSelect) { + //console.log(this.settings.narrator_enabled) + // Sync the state with the checkbox in index.js + const ttsPassAsterisksCheckbox = document.getElementById('tts_pass_asterisks'); // Access the checkbox from index.js + const ttsNarrateQuotedCheckbox = document.getElementById('tts_narrate_quoted'); // Access the checkbox from index.js + const ttsNarrateDialoguesCheckbox = document.getElementById('tts_narrate_dialogues'); // Access the checkbox from index.js + // Sync the state with the checkbox in index.js + if (this.settings.narrator_enabled) { + ttsPassAsterisksCheckbox.checked = false; + $('#tts_pass_asterisks').click(); // Simulate a click event + $('#tts_pass_asterisks').trigger('change'); + } + if (!this.settings.narrator_enabled) { + ttsPassAsterisksCheckbox.checked = true; + $('#tts_pass_asterisks').click(); // Simulate a click event + $('#tts_pass_asterisks').trigger('change'); + } + // Uncheck and set tts_narrate_quoted to false if narrator is enabled + if (this.settings.narrator_enabledd) { + ttsNarrateQuotedCheckbox.checked = true; + ttsNarrateDialoguesCheckbox.checked = true; + // Trigger click events instead of change events + $('#tts_narrate_quoted').click(); + $('#tts_narrate_quoted').trigger('change'); + $('#tts_narrate_dialogues').click(); + $('#tts_narrate_dialogues').trigger('change'); + } + atNarratorSelect.value = this.settings.narrator_enabled.toString(); + this.settings.narrator_enabled = this.settings.narrator_enabled.toString(); + } + // Apply settings to the Language dropdown + const languageSelect = document.getElementById('language_options'); + if (languageSelect && this.settings.language) { + languageSelect.value = this.settings.language; + //console.log(this.settings.language) + } + // Apply settings to Text Not Inside dropdown + if (textNotInsideSelect && this.settings.text_not_inside) { + textNotInsideSelect.value = this.settings.text_not_inside; + this.settings.at_narrator_text_not_inside = this.settings.text_not_inside; + //console.log(this.settings.at_narrator_text_not_inside) + } + // Apply settings to Generation Method dropdown + if (generationMethodSelect && this.settings.at_generation_method) { + generationMethodSelect.value = this.settings.at_generation_method; + //console.log(this.settings.at_generation_method) + } + // Additional logic to disable/enable dropdowns based on the selected generation method + const isStreamingEnabled = this.settings.at_generation_method === 'streaming_enabled'; + if (isStreamingEnabled) { + // Disable certain dropdowns when streaming is enabled + if (atNarratorSelect) atNarratorSelect.disabled = true; + if (textNotInsideSelect) textNotInsideSelect.disabled = true; + if (narratorVoiceSelect) narratorVoiceSelect.disabled = true; + } else { + // Enable dropdowns for standard generation + if (atNarratorSelect) atNarratorSelect.disabled = false; + if (textNotInsideSelect) textNotInsideSelect.disabled = !this.settings.narrator_enabled; + if (narratorVoiceSelect) narratorVoiceSelect.disabled = !this.settings.narrator_enabled; + } + const modelSelect = document.getElementById('switch_model'); + if (this.settings.finetuned_model === 'true') { + const ftOption = document.createElement('option'); + ftOption.value = 'XTTSv2 FT'; + ftOption.textContent = 'XTTSv2 FT'; + modelSelect.appendChild(ftOption); + } + //console.log("Settings applied to HTML."); + } + + //##############################// + // Check AT Server is Available // + //##############################// + + async checkReady() { + try { + const response = await fetch(`${this.settings.provider_endpoint}/api/ready`); + // Check if the HTTP request was successful + if (!response.ok) { + throw new Error(`HTTP Error Response: ${response.status} ${response.statusText}`); + } + const statusText = await response.text(); + // Check if the response is 'Ready' + if (statusText === 'Ready') { + this.ready = true; // Set the ready flag to true + console.log('TTS service is ready.'); + } else { + this.ready = false; + console.log('TTS service is not ready.'); + } + } catch (error) { + console.error('Error checking TTS service readiness:', error); + this.ready = false; // Ensure ready flag is set to false in case of error + } + } + + //######################// + // Get Available Voices // + //######################// + + async fetchTtsVoiceObjects() { + const response = await fetch(`${this.settings.provider_endpoint}/api/voices`); + if (!response.ok) { + const errorText = await response.text(); + throw new Error(`HTTP ${response.status}: ${errorText}`); + } + const data = await response.json(); + const voices = data.voices.map(filename => { + const voiceName = filename.replace('.wav', ''); + return { + name: voiceName, + voice_id: voiceName, + preview_url: null, // Preview URL will be dynamically generated + lang: 'en' // Default language + }; + }); + this.voices = voices; // Assign to the class property + return voices; // Also return this list + } + + //##########################################// + // Get Current AT Server Config & Update ST // + //##########################################// + + async updateSettingsFromServer() { + try { + // Fetch current settings + const response = await fetch(`${this.settings.provider_endpoint}/api/currentsettings`); + if (!response.ok) { + throw new Error(`Failed to fetch current settings: ${response.statusText}`); + } + const currentSettings = await response.json(); + // Update internal settings + this.settings.modelsAvailable = currentSettings.models_available; + this.settings.currentModel = currentSettings.current_model_loaded; + this.settings.deepspeed_available = currentSettings.deepspeed_available; + this.settings.deepSpeedEnabled = currentSettings.deepspeed_status; + this.settings.lowVramEnabled = currentSettings.low_vram_status; + this.settings.finetuned_model = currentSettings.finetuned_model + // Update HTML elements + this.updateModelDropdown(); + this.updateCheckboxes(); + } catch (error) { + console.error(`Error updating settings from server: ${error}`); + } + } + + //###################################################// + // Get Current AT Server Config & Update ST (Models) // + //###################################################// + + updateModelDropdown() { + const modelSelect = document.getElementById('switch_model'); + if (modelSelect) { + modelSelect.innerHTML = ''; // Clear existing options + this.settings.modelsAvailable.forEach(model => { + const option = document.createElement('option'); + option.value = model.model_name; + option.textContent = model.model_name; // Use model_name instead of name + option.selected = model.model_name === this.settings.currentModel; + modelSelect.appendChild(option); + }); + } + } + + //#######################################################// + // Get Current AT Server Config & Update ST (DS and LVR) // + //#######################################################// + + updateCheckboxes() { + const deepspeedCheckbox = document.getElementById('deepspeed'); + const lowVramCheckbox = document.getElementById('low_vram'); + if (lowVramCheckbox) lowVramCheckbox.checked = this.settings.lowVramEnabled; + if (deepspeedCheckbox) { + deepspeedCheckbox.checked = this.settings.deepSpeedEnabled; + deepspeedCheckbox.disabled = !this.settings.deepspeed_available; // Disable checkbox if deepspeed is not available + } + } + + //###############################################################// + // Get Current AT Server Config & Update ST (AT Narrator Voices) // + //###############################################################// + + updateNarratorVoicesDropdown() { + const narratorVoiceSelect = document.getElementById('narrator_voice'); + if (narratorVoiceSelect && this.voices) { + // Clear existing options + narratorVoiceSelect.innerHTML = ''; + // Add new options + for (let voice of this.voices) { + const option = document.createElement('option'); + option.value = voice.voice_id; + option.textContent = voice.name; + narratorVoiceSelect.appendChild(option); + } + } + } + + //######################################################// + // Get Current AT Server Config & Update ST (Languages) // + //######################################################// + + updateLanguageDropdown() { + const languageSelect = document.getElementById('language_options'); + if (languageSelect) { + // Ensure default language is set + this.settings.language = this.Settings.language; + + languageSelect.innerHTML = ''; + for (let language in this.languageLabels) { + const option = document.createElement('option'); + option.value = this.languageLabels[language]; + option.textContent = language; + if (this.languageLabels[language] === this.settings.language) { + option.selected = true; + } + languageSelect.appendChild(option); + } + } + } + + //########################################// + // Start AT TTS extenstion page listeners // + //########################################// + + setupEventListeners() { + + let debounceTimeout; + const debounceDelay = 500; // Milliseconds + + // Define the event handler function + const onModelSelectChange = async (event) => { + console.log("Model select change event triggered"); // Debugging statement + const selectedModel = event.target.value; + console.log(`Selected model: ${selectedModel}`); // Debugging statement + // Set status to Processing + updateStatus('Processing'); + try { + const response = await fetch(`${this.settings.provider_endpoint}/api/reload?tts_method=${encodeURIComponent(selectedModel)}`, { + method: 'POST' + }); + if (!response.ok) { + throw new Error(`HTTP Error: ${response.status}`); + } + const data = await response.json(); + console.log("POST response data:", data); // Debugging statement + // Set status to Ready if successful + updateStatus('Ready'); + } catch (error) { + console.error("POST request error:", error); // Debugging statement + // Set status to Error in case of failure + updateStatus('Error'); + } + + // Handle response or error + }; + + const debouncedModelSelectChange = (event) => { + clearTimeout(debounceTimeout); + debounceTimeout = setTimeout(() => { + onModelSelectChange(event); + }, debounceDelay); + }; + + // Switch Model Listener + const modelSelect = document.getElementById('switch_model'); + if (modelSelect) { + // Remove the event listener if it was previously added + modelSelect.removeEventListener('change', debouncedModelSelectChange); + // Add the debounced event listener + modelSelect.addEventListener('change', debouncedModelSelectChange); + } + + // DeepSpeed Listener + const deepspeedCheckbox = document.getElementById('deepspeed'); + if (deepspeedCheckbox) { + deepspeedCheckbox.addEventListener('change', async (event) => { + const deepSpeedValue = event.target.checked ? 'True' : 'False'; + // Set status to Processing + updateStatus('Processing'); + try { + const response = await fetch(`${this.settings.provider_endpoint}/api/deepspeed?new_deepspeed_value=${deepSpeedValue}`, { + method: 'POST' + }); + if (!response.ok) { + throw new Error(`HTTP Error: ${response.status}`); + } + const data = await response.json(); + console.log("POST response data:", data); // Debugging statement + // Set status to Ready if successful + updateStatus('Ready'); + } catch (error) { + console.error("POST request error:", error); // Debugging statement + // Set status to Error in case of failure + updateStatus('Error'); + } + }); + } + + // Low VRAM Listener + const lowVramCheckbox = document.getElementById('low_vram'); + if (lowVramCheckbox) { + lowVramCheckbox.addEventListener('change', async (event) => { + const lowVramValue = event.target.checked ? 'True' : 'False'; + // Set status to Processing + updateStatus('Processing'); + try { + const response = await fetch(`${this.settings.provider_endpoint}/api/lowvramsetting?new_low_vram_value=${lowVramValue}`, { + method: 'POST' + }); + if (!response.ok) { + throw new Error(`HTTP Error: ${response.status}`); + } + const data = await response.json(); + console.log("POST response data:", data); // Debugging statement + // Set status to Ready if successful + updateStatus('Ready'); + } catch (error) { + console.error("POST request error:", error); // Debugging statement + // Set status to Error in case of failure + updateStatus('Error'); + } + }); + } + + // Narrator Voice Dropdown Listener + const narratorVoiceSelect = document.getElementById('narrator_voice'); + if (narratorVoiceSelect) { + narratorVoiceSelect.addEventListener('change', (event) => { + this.settings.narrator_voice_gen = `${event.target.value}.wav`; + this.onSettingsChangeAllTalk(); // Save the settings after change + }); + } + + const textNotInsideSelect = document.getElementById('at_narrator_text_not_inside'); + if (textNotInsideSelect) { + textNotInsideSelect.addEventListener('change', (event) => { + this.settings.text_not_inside = event.target.value; + this.onSettingsChangeAllTalk(); // Save the settings after change + }); + } + + // AT Narrator Dropdown Listener + const atNarratorSelect = document.getElementById('at_narrator_enabled'); + const ttsPassAsterisksCheckbox = document.getElementById('tts_pass_asterisks'); // Access the checkbox from index.js + const ttsNarrateQuotedCheckbox = document.getElementById('tts_narrate_quoted'); // Access the checkbox from index.js + const ttsNarrateDialoguesCheckbox = document.getElementById('tts_narrate_dialogues'); // Access the checkbox from index.js + + if (atNarratorSelect && textNotInsideSelect && narratorVoiceSelect) { + atNarratorSelect.addEventListener('change', (event) => { + const isNarratorEnabled = event.target.value === 'true'; + this.settings.narrator_enabled = isNarratorEnabled; // Update the setting here + textNotInsideSelect.disabled = !isNarratorEnabled; + narratorVoiceSelect.disabled = !isNarratorEnabled; + + // Sync the state with the checkbox in index.js + if (isNarratorEnabled) { + ttsPassAsterisksCheckbox.checked = false; + $('#tts_pass_asterisks').click(); // Simulate a click event + $('#tts_pass_asterisks').trigger('change'); + } + if (!isNarratorEnabled) { + ttsPassAsterisksCheckbox.checked = true; + $('#tts_pass_asterisks').click(); // Simulate a click event + $('#tts_pass_asterisks').trigger('change'); + } + // Uncheck and set tts_narrate_quoted to false if narrator is enabled + if (isNarratorEnabled) { + ttsNarrateQuotedCheckbox.checked = true; + ttsNarrateDialoguesCheckbox.checked = true; + // Trigger click events instead of change events + $('#tts_narrate_quoted').click(); + $('#tts_narrate_quoted').trigger('change'); + $('#tts_narrate_dialogues').click(); + $('#tts_narrate_dialogues').trigger('change'); + } + this.onSettingsChangeAllTalk(); // Save the settings after change + }); + } + + + + // Event Listener for AT Generation Method Dropdown + const atGenerationMethodSelect = document.getElementById('at_generation_method'); + const atNarratorEnabledSelect = document.getElementById('at_narrator_enabled'); + if (atGenerationMethodSelect) { + atGenerationMethodSelect.addEventListener('change', (event) => { + const selectedMethod = event.target.value; + + if (selectedMethod === 'streaming_enabled') { + // Disable and unselect AT Narrator + atNarratorEnabledSelect.disabled = true; + atNarratorEnabledSelect.value = 'false'; + textNotInsideSelect.disabled = true; + narratorVoiceSelect.disabled = true; + } else if (selectedMethod === 'standard_generation') { + // Enable AT Narrator + atNarratorEnabledSelect.disabled = false; + } + this.settings.at_generation_method = selectedMethod; // Update the setting here + this.onSettingsChangeAllTalk(); // Save the settings after change + }); + } + + // Listener for Language Dropdown + const languageSelect = document.getElementById('language_options'); + if (languageSelect) { + languageSelect.addEventListener('change', (event) => { + this.settings.language = event.target.value; + this.onSettingsChangeAllTalk(); // Save the settings after change + }); + } + } + + //#############################// + // Store ST interface settings // + //#############################// + + onSettingsChange() { + // Update settings that SillyTavern will save + this.settings.provider_endpoint = $('#at_server').val(); + this.settings.language = $('#language_options').val(); + saveTtsProviderSettings(); // This function should save settings handled by SillyTavern + // Call the function to handle AllTalk specific settings + this.onSettingsChangeAllTalk(); // Save the settings after change + } + + onSettingsChangeAllTalk() { + // Update AllTalk specific settings and save to localStorage + this.settings.narrator_enabled = $('#at_narrator_enabled').val() === 'true'; + this.settings.narrator_voice = $('#narrator_voice').val() + '.wav'; // assuming you need to append .wav + this.settings.text_not_inside = $('#at_narrator_text_not_inside').val(); // "character" or "narrator" + this.settings.at_generation_method = $('#at_generation_method').val(); // Streaming or standard + this.settings.language = $('#language_options').val(); // Streaming or standard + + // Call the save function with the current settings + saveAllTalkSettings({ + narrator_voice: this.settings.narrator_voice, + narrator_enabled: this.settings.narrator_enabled, + text_not_inside: this.settings.text_not_inside, + at_generation_method: this.settings.at_generation_method, + language: this.settings.language + }); + } + + //#########################// + // ST Handle Reload button // + //#########################// + + async onRefreshClick() { + await this.checkReady(); // Check if the TTS provider is ready + await this.loadSettings(this.settings); // Reload the settings + // Additional actions as needed + } + + //##################// + // Preview AT Voice // + //##################// + + async previewTtsVoice(voiceName) { + try { + // Prepare data for POST request + const postData = new URLSearchParams(); + postData.append("voice", `${voiceName}.wav`); + // Making the POST request + const response = await fetch(`${this.settings.provider_endpoint}/api/previewvoice/`, { + method: "POST", + headers: { + 'Content-Type': 'application/x-www-form-urlencoded' + }, + body: postData, + }); + if (!response.ok) { + const errorText = await response.text(); + console.error(`[previewTtsVoice] Error Response Text:`, errorText); + throw new Error(`HTTP ${response.status}: ${errorText}`); + } + // Assuming the server returns a URL to the .wav file + const data = await response.json(); + if (data.output_file_url) { + // Use an audio element to play the .wav file + const audioElement = new Audio(data.output_file_url); + audioElement.play().catch(e => console.error("Error playing audio:", e)); + } else { + console.warn("[previewTtsVoice] No output file URL received in the response"); + throw new Error("No output file URL received in the response"); + } + + } catch (error) { + console.error("[previewTtsVoice] Exception caught during preview generation:", error); + throw error; + } + } + + //#####################// + // Populate ST voices // + //#####################// + + async getVoice(voiceName, generatePreview = false) { + // Ensure this.voices is populated + if (this.voices.length === 0) { + // Fetch voice objects logic + } + // Find the object where the name matches voiceName + const match = this.voices.find(voice => voice.name === voiceName); + if (!match) { + // Error handling + } + // Generate preview URL only if requested + if (!match.preview_url && generatePreview) { + // Generate preview logic + } + return match; // Return the found voice object + } + + //##########################################// + // Generate TTS Streaming or call Standard // + //##########################################// + + async generateTts(inputText, voiceId) { + try { + if (this.settings.at_generation_method === 'streaming_enabled') { + // For streaming method + const streamingUrl = `${this.settings.provider_endpoint}/api/tts-generate-streaming?text=${encodeURIComponent(inputText)}&voice=${encodeURIComponent(voiceId)}.wav&language=${encodeURIComponent(this.settings.language)}&output_file=stream_output.wav`; + const audioElement = new Audio(streamingUrl); + audioElement.play(); // Play the audio stream directly + console.log("Streaming URL:", streamingUrl); + return new Response(null, { + status: 200, + statusText: "OK", + headers: { + "Content-Type": "audio/wav", + "Content-Location": streamingUrl + } + }); + } else { + // For standard method + const outputUrl = await this.fetchTtsGeneration(inputText, voiceId); + // Fetch the audio data as a blob from the URL + const audioResponse = await fetch(outputUrl); + if (!audioResponse.ok) { + throw new Error(`HTTP ${audioResponse.status}: Failed to fetch audio data`); + } + const audioBlob = await audioResponse.blob(); + if (!audioBlob.type.startsWith('audio/')) { + throw new Error(`Invalid audio data format. Expecting audio/*, got ${audioBlob.type}`); + } + return new Response(audioBlob, { + status: 200, + statusText: "OK", + headers: { + "Content-Type": audioBlob.type + } + }); + } + } catch (error) { + console.error("Error in generateTts:", error); + throw error; + } + } + + //####################// + // Generate Standard // + //####################// + + async fetchTtsGeneration(inputText, voiceId) { + // Prepare the request payload + const requestBody = new URLSearchParams({ + 'text_input': inputText, + 'text_filtering': "standard", + 'character_voice_gen': voiceId + ".wav", + 'narrator_enabled': this.settings.narrator_enabled, + 'narrator_voice_gen': this.settings.narrator_voice_gen, + 'text_not_inside': this.settings.text_not_inside, + 'language': this.settings.language, + 'output_file_name': "st_output", + 'output_file_timestamp': "true", + 'autoplay': "false", + 'autoplay_volume': "0.8" + }).toString(); + + try { + const response = await doExtrasFetch( + `${this.settings.provider_endpoint}/api/tts-generate`, + { + method: 'POST', + headers: { + 'Content-Type': 'application/x-www-form-urlencoded', + 'Cache-Control': 'no-cache', + }, + body: requestBody + } + ); + + if (!response.ok) { + const errorText = await response.text(); + console.error(`[fetchTtsGeneration] Error Response Text:`, errorText); + // Uncomment the following line if you have a UI element for displaying errors + // toastr.error(response.statusText, 'TTS Generation Failed'); + throw new Error(`HTTP ${response.status}: ${errorText}`); + } + const data = await response.json(); + const outputUrl = data.output_file_url; + return outputUrl; // Return only the output_file_url + } catch (error) { + console.error("[fetchTtsGeneration] Exception caught:", error); + throw error; // Rethrow the error for further handling + } + } +} + +//#########################// +// Update Status Messages // +//#########################// + +function updateStatus(message) { + const statusElement = document.getElementById('status_info'); + if (statusElement) { + statusElement.textContent = message; + switch (message) { + case 'Offline': + statusElement.style.color = 'red'; + break; + case 'Ready': + statusElement.style.color = 'lightgreen'; + break; + case 'Processing': + statusElement.style.color = 'blue'; + break; + case 'Error': + statusElement.style.color = 'red'; + break; + // Add more cases as needed + } + } +} + +//########################// +// Save/load AT Settings // +//########################// + +function saveAllTalkSettings(settingsToSave) { + // Save the specific settings to localStorage + console.log("settings", settingsToSave) + localStorage.setItem('AllTalkSettings', JSON.stringify(settingsToSave)); +} + +function loadAllTalkSettings() { + // Retrieve the settings from localStorage + const settings = localStorage.getItem('AllTalkSettings'); + // If settings exist, parse them back into an object + if (settings) { + return JSON.parse(settings); + } + return null; +} diff --git a/templates/STfiles/index.js b/templates/STfiles/index.js new file mode 100644 index 0000000..d07aea6 --- /dev/null +++ b/templates/STfiles/index.js @@ -0,0 +1,1028 @@ +import { callPopup, cancelTtsPlay, eventSource, event_types, name2, saveSettingsDebounced } from '../../../script.js'; +import { ModuleWorkerWrapper, doExtrasFetch, extension_settings, getApiUrl, getContext, modules } from '../../extensions.js'; +import { delay, escapeRegex, getStringHash } from '../../utils.js'; +import { EdgeTtsProvider } from './edge.js'; +import { ElevenLabsTtsProvider } from './elevenlabs.js'; +import { SileroTtsProvider } from './silerotts.js'; +import { CoquiTtsProvider } from './coqui.js'; +import { SystemTtsProvider } from './system.js'; +import { NovelTtsProvider } from './novel.js'; +import { power_user } from '../../power-user.js'; +import { registerSlashCommand } from '../../slash-commands.js'; +import { OpenAITtsProvider } from './openai.js'; +import { XTTSTtsProvider } from './xtts.js'; +import { AllTalkTtsProvider } from './alltalk.js'; +export { talkingAnimation }; + +const UPDATE_INTERVAL = 1000; + +let voiceMapEntries = []; +let voiceMap = {}; // {charName:voiceid, charName2:voiceid2} +let storedvalue = false; +let lastChatId = null; +let lastMessageHash = null; + +const DEFAULT_VOICE_MARKER = '[Default Voice]'; +const DISABLED_VOICE_MARKER = 'disabled'; + +export function getPreviewString(lang) { + const previewStrings = { + 'en-US': 'The quick brown fox jumps over the lazy dog', + 'en-GB': 'Sphinx of black quartz, judge my vow', + 'fr-FR': 'Portez ce vieux whisky au juge blond qui fume', + 'de-DE': 'Victor jagt zwölf Boxkämpfer quer über den großen Sylter Deich', + 'it-IT': 'Pranzo d\'acqua fa volti sghembi', + 'es-ES': 'Quiere la boca exhausta vid, kiwi, piña y fugaz jamón', + 'es-MX': 'Fabio me exige, sin tapujos, que añada cerveza al whisky', + 'ru-RU': 'В чащах юга жил бы цитрус? Да, но фальшивый экземпляр!', + 'pt-BR': 'Vejo xá gritando que fez show sem playback.', + 'pt-PR': 'Todo pajé vulgar faz boquinha sexy com kiwi.', + 'uk-UA': 'Фабрикуймо гідність, лящім їжею, ґав хапаймо, з\'єднавці чаш!', + 'pl-PL': 'Pchnąć w tę łódź jeża lub ośm skrzyń fig', + 'cs-CZ': 'Příliš žluťoučký kůň úpěl ďábelské ódy', + 'sk-SK': 'Vyhŕňme si rukávy a vyprážajme čínske ryžové cestoviny', + 'hu-HU': 'Árvíztűrő tükörfúrógép', + 'tr-TR': 'Pijamalı hasta yağız şoföre çabucak güvendi', + 'nl-NL': 'De waard heeft een kalfje en een pinkje opgegeten', + 'sv-SE': 'Yxskaftbud, ge vårbygd, zinkqvarn', + 'da-DK': 'Quizdeltagerne spiste jordbær med fløde, mens cirkusklovnen Walther spillede på xylofon', + 'ja-JP': 'いろはにほへと　ちりぬるを　わかよたれそ　つねならむ　うゐのおくやま　けふこえて　あさきゆめみし　ゑひもせす', + 'ko-KR': '가나다라마바사아자차카타파하', + 'zh-CN': '我能吞下玻璃而不伤身体', + 'ro-RO': 'Muzicologă în bej vând whisky și tequila, preț fix', + 'bg-BG': 'Щъркелите се разпръснаха по цялото небе', + 'el-GR': 'Ταχίστη αλώπηξ βαφής ψημένη γη, δρασκελίζει υπέρ νωθρού κυνός', + 'fi-FI': 'Voi veljet, miksi juuri teille myin nämä vehkeet?', + 'he-IL': 'הקצינים צעקו: "כל הכבוד לצבא הצבאות!"', + 'id-ID': 'Jangkrik itu memang enak, apalagi kalau digoreng', + 'ms-MY': 'Muzik penyanyi wanita itu menggambarkan kehidupan yang penuh dengan duka nestapa', + 'th-TH': 'เป็นไงบ้างครับ ผมชอบกินข้าวผัดกระเพราหมูกรอบ', + 'vi-VN': 'Cô bé quàng khăn đỏ đang ngồi trên bãi cỏ xanh', + 'ar-SA': 'أَبْجَدِيَّة عَرَبِيَّة', + 'hi-IN': 'श्वेता ने श्वेता के श्वेते हाथों में श्वेता का श्वेता चावल पकड़ा', + }; + const fallbackPreview = 'Neque porro quisquam est qui dolorem ipsum quia dolor sit amet'; + + return previewStrings[lang] ?? fallbackPreview; +} + +let ttsProviders = { + ElevenLabs: ElevenLabsTtsProvider, + Silero: SileroTtsProvider, + XTTSv2: XTTSTtsProvider, + System: SystemTtsProvider, + Coqui: CoquiTtsProvider, + Edge: EdgeTtsProvider, + Novel: NovelTtsProvider, + OpenAI: OpenAITtsProvider, + AllTalk: AllTalkTtsProvider, +}; +let ttsProvider; +let ttsProviderName; + +let ttsLastMessage = null; + +async function onNarrateOneMessage() { + audioElement.src = '/sounds/silence.mp3'; + const context = getContext(); + const id = $(this).closest('.mes').attr('mesid'); + const message = context.chat[id]; + + if (!message) { + return; + } + + resetTtsPlayback(); + ttsJobQueue.push(message); + moduleWorker(); +} + +async function onNarrateText(args, text) { + if (!text) { + return; + } + + audioElement.src = '/sounds/silence.mp3'; + + // To load all characters in the voice map, set unrestricted to true + await initVoiceMap(true); + + const baseName = args?.voice || name2; + const name = (baseName === 'SillyTavern System' ? DEFAULT_VOICE_MARKER : baseName) || DEFAULT_VOICE_MARKER; + + const voiceMapEntry = voiceMap[name] === DEFAULT_VOICE_MARKER + ? voiceMap[DEFAULT_VOICE_MARKER] + : voiceMap[name]; + + if (!voiceMapEntry || voiceMapEntry === DISABLED_VOICE_MARKER) { + toastr.info(`Specified voice for ${name} was not found. Check the TTS extension settings.`); + return; + } + + resetTtsPlayback(); + ttsJobQueue.push({ mes: text, name: name }); + await moduleWorker(); + + // Return back to the chat voices + await initVoiceMap(false); +} + +async function moduleWorker() { + // Primarily determining when to add new chat to the TTS queue + const enabled = $('#tts_enabled').is(':checked'); + $('body').toggleClass('tts', enabled); + if (!enabled) { + return; + } + + const context = getContext(); + const chat = context.chat; + + processTtsQueue(); + processAudioJobQueue(); + updateUiAudioPlayState(); + + // Auto generation is disabled + if (extension_settings.tts.auto_generation == false) { + return; + } + + // no characters or group selected + if (!context.groupId && context.characterId === undefined) { + return; + } + + // Chat changed + if ( + context.chatId !== lastChatId + ) { + currentMessageNumber = context.chat.length ? context.chat.length : 0; + saveLastValues(); + + // Force to speak on the first message in the new chat + if (context.chat.length === 1) { + lastMessageHash = -1; + } + + return; + } + + // take the count of messages + let lastMessageNumber = context.chat.length ? context.chat.length : 0; + + // There's no new messages + let diff = lastMessageNumber - currentMessageNumber; + let hashNew = getStringHash((chat.length && chat[chat.length - 1].mes) ?? ''); + + // if messages got deleted, diff will be < 0 + if (diff < 0) { + // necessary actions will be taken by the onChatDeleted() handler + return; + } + + // if no new messages, or same message, or same message hash, do nothing + if (diff == 0 && hashNew === lastMessageHash) { + return; + } + + // If streaming, wait for streaming to finish before processing new messages + if (context.streamingProcessor && !context.streamingProcessor.isFinished) { + return; + } + + // clone message object, as things go haywire if message object is altered below (it's passed by reference) + const message = structuredClone(chat[chat.length - 1]); + + // if last message within current message, message got extended. only send diff to TTS. + if (ttsLastMessage !== null && message.mes.indexOf(ttsLastMessage) !== -1) { + let tmp = message.mes; + message.mes = message.mes.replace(ttsLastMessage, ''); + ttsLastMessage = tmp; + } else { + ttsLastMessage = message.mes; + } + + // We're currently swiping. Don't generate voice + if (!message || message.mes === '...' || message.mes === '') { + return; + } + + // Don't generate if message doesn't have a display text + if (extension_settings.tts.narrate_translated_only && !(message?.extra?.display_text)) { + return; + } + + // Don't generate if message is a user message and user message narration is disabled + if (message.is_user && !extension_settings.tts.narrate_user) { + return; + } + + // New messages, add new chat to history + lastMessageHash = hashNew; + currentMessageNumber = lastMessageNumber; + + console.debug( + `Adding message from ${message.name} for TTS processing: "${message.mes}"`, + ); + ttsJobQueue.push(message); +} + +function talkingAnimation(switchValue) { + if (!modules.includes('talkinghead')) { + console.debug('Talking Animation module not loaded'); + return; + } + + const apiUrl = getApiUrl(); + const animationType = switchValue ? 'start' : 'stop'; + + if (switchValue !== storedvalue) { + try { + console.log(animationType + ' Talking Animation'); + doExtrasFetch(`${apiUrl}/api/talkinghead/${animationType}_talking`); + storedvalue = switchValue; // Update the storedvalue to the current switchValue + } catch (error) { + // Handle the error here or simply ignore it to prevent logging + } + } + updateUiAudioPlayState(); +} + +function resetTtsPlayback() { + // Stop system TTS utterance + cancelTtsPlay(); + + // Clear currently processing jobs + currentTtsJob = null; + currentAudioJob = null; + + // Reset audio element + audioElement.currentTime = 0; + audioElement.src = ''; + + // Clear any queue items + ttsJobQueue.splice(0, ttsJobQueue.length); + audioJobQueue.splice(0, audioJobQueue.length); + + // Set audio ready to process again + audioQueueProcessorReady = true; +} + +function isTtsProcessing() { + let processing = false; + + // Check job queues + if (ttsJobQueue.length > 0 || audioJobQueue.length > 0) { + processing = true; + } + // Check current jobs + if (currentTtsJob != null || currentAudioJob != null) { + processing = true; + } + return processing; +} + +function debugTtsPlayback() { + console.log(JSON.stringify( + { + 'ttsProviderName': ttsProviderName, + 'voiceMap': voiceMap, + 'currentMessageNumber': currentMessageNumber, + 'audioPaused': audioPaused, + 'audioJobQueue': audioJobQueue, + 'currentAudioJob': currentAudioJob, + 'audioQueueProcessorReady': audioQueueProcessorReady, + 'ttsJobQueue': ttsJobQueue, + 'currentTtsJob': currentTtsJob, + 'ttsConfig': extension_settings.tts, + }, + )); +} +window.debugTtsPlayback = debugTtsPlayback; + +//##################// +// Audio Control // +//##################// + +let audioElement = new Audio(); +audioElement.id = 'tts_audio'; +audioElement.autoplay = true; + +let audioJobQueue = []; +let currentAudioJob; +let audioPaused = false; +let audioQueueProcessorReady = true; + +async function playAudioData(audioBlob) { + // Since current audio job can be cancelled, don't playback if it is null + if (currentAudioJob == null) { + console.log('Cancelled TTS playback because currentAudioJob was null'); + } + const reader = new FileReader(); + reader.onload = function (e) { + const srcUrl = e.target.result; + audioElement.src = srcUrl; + }; + reader.readAsDataURL(audioBlob); + audioElement.addEventListener('ended', completeCurrentAudioJob); + audioElement.addEventListener('canplay', () => { + console.debug('Starting TTS playback'); + audioElement.play(); + }); +} + +window['tts_preview'] = function (id) { + const audio = document.getElementById(id); + + if (audio && !$(audio).data('disabled')) { + audio.play(); + } + else { + ttsProvider.previewTtsVoice(id); + } +}; + +async function onTtsVoicesClick() { + let popupText = ''; + + try { + const voiceIds = await ttsProvider.fetchTtsVoiceObjects(); + + for (const voice of voiceIds) { + popupText += ` +
+ ${voice.lang || ''} + ${voice.name} + +
`; + if (voice.preview_url) { + popupText += ``; + } + } + } catch { + popupText = 'Could not load voices list. Check your API key.'; + } + + callPopup(popupText, 'text'); +} + +function updateUiAudioPlayState() { + if (extension_settings.tts.enabled == true) { + $('#ttsExtensionMenuItem').show(); + let img; + // Give user feedback that TTS is active by setting the stop icon if processing or playing + if (!audioElement.paused || isTtsProcessing()) { + img = 'fa-solid fa-stop-circle extensionsMenuExtensionButton'; + } else { + img = 'fa-solid fa-circle-play extensionsMenuExtensionButton'; + } + $('#tts_media_control').attr('class', img); + } else { + $('#ttsExtensionMenuItem').hide(); + } +} + +function onAudioControlClicked() { + audioElement.src = '/sounds/silence.mp3'; + let context = getContext(); + // Not pausing, doing a full stop to anything TTS is doing. Better UX as pause is not as useful + if (!audioElement.paused || isTtsProcessing()) { + resetTtsPlayback(); + talkingAnimation(false); + } else { + // Default play behavior if not processing or playing is to play the last message. + ttsJobQueue.push(context.chat[context.chat.length - 1]); + } + updateUiAudioPlayState(); +} + +function addAudioControl() { + + $('#extensionsMenu').prepend(` +
+
+ TTS Playback +
`); + $('#ttsExtensionMenuItem').attr('title', 'TTS play/pause').on('click', onAudioControlClicked); + updateUiAudioPlayState(); +} + +function completeCurrentAudioJob() { + audioQueueProcessorReady = true; + currentAudioJob = null; + talkingAnimation(false); //stop lip animation + // updateUiPlayState(); +} + +/** + * Accepts an HTTP response containing audio/mpeg data, and puts the data as a Blob() on the queue for playback + * @param {Response} response + */ +async function addAudioJob(response) { + const audioData = await response.blob(); + if (!audioData.type.startsWith('audio/')) { + throw `TTS received HTTP response with invalid data format. Expecting audio/*, got ${audioData.type}`; + } + audioJobQueue.push(audioData); + console.debug('Pushed audio job to queue.'); +} + +async function processAudioJobQueue() { + // Nothing to do, audio not completed, or audio paused - stop processing. + if (audioJobQueue.length == 0 || !audioQueueProcessorReady || audioPaused) { + return; + } + try { + audioQueueProcessorReady = false; + currentAudioJob = audioJobQueue.pop(); + playAudioData(currentAudioJob); + talkingAnimation(true); + } catch (error) { + console.error(error); + audioQueueProcessorReady = true; + } +} + +//################// +// TTS Control // +//################// + +let ttsJobQueue = []; +let currentTtsJob; // Null if nothing is currently being processed +let currentMessageNumber = 0; + +function completeTtsJob() { + console.info(`Current TTS job for ${currentTtsJob?.name} completed.`); + currentTtsJob = null; +} + +function saveLastValues() { + const context = getContext(); + lastChatId = context.chatId; + lastMessageHash = getStringHash( + (context.chat.length && context.chat[context.chat.length - 1].mes) ?? '', + ); +} + +async function tts(text, voiceId, char) { + let response = await ttsProvider.generateTts(text, voiceId); + + // RVC injection + if (extension_settings.rvc.enabled && typeof window['rvcVoiceConversion'] === 'function') + response = await window['rvcVoiceConversion'](response, char, text); + + addAudioJob(response); + completeTtsJob(); +} + +async function processTtsQueue() { + // Called each moduleWorker iteration to pull chat messages from queue + if (currentTtsJob || ttsJobQueue.length <= 0 || audioPaused) { + return; + } + + console.debug('New message found, running TTS'); + currentTtsJob = ttsJobQueue.shift(); + let text = extension_settings.tts.narrate_translated_only ? (currentTtsJob?.extra?.display_text || currentTtsJob.mes) : currentTtsJob.mes; + + if (extension_settings.tts.skip_codeblocks) { + text = text.replace(/^\s{4}.*$/gm, '').trim(); + text = text.replace(/```.*?```/gs, '').trim(); + } + + if (!extension_settings.tts.pass_asterisks) { + text = extension_settings.tts.narrate_dialogues_only + ? text.replace(/\*[^*]*?(\*|$)/g, '').trim() // remove asterisks content + : text.replaceAll('*', '').trim(); // remove just the asterisks + } + + if (extension_settings.tts.narrate_quoted_only) { + const special_quotes = /[“”]/g; // Extend this regex to include other special quotes + text = text.replace(special_quotes, '"'); + const matches = text.match(/".*?"/g); // Matches text inside double quotes, non-greedily + const partJoiner = (ttsProvider?.separator || ' ... '); + text = matches ? matches.join(partJoiner) : text; + } + + if (typeof ttsProvider?.processText === 'function') { + text = await ttsProvider.processText(text); + } + + // Collapse newlines and spaces into single space + text = text.replace(/\s+/g, ' ').trim(); + + console.log(`TTS: ${text}`); + const char = currentTtsJob.name; + + // Remove character name from start of the line if power user setting is disabled + if (char && !power_user.allow_name2_display) { + const escapedChar = escapeRegex(char); + text = text.replace(new RegExp(`^${escapedChar}:`, 'gm'), ''); + } + + try { + if (!text) { + console.warn('Got empty text in TTS queue job.'); + completeTtsJob(); + return; + } + + const voiceMapEntry = voiceMap[char] === DEFAULT_VOICE_MARKER ? voiceMap[DEFAULT_VOICE_MARKER] : voiceMap[char]; + + if (!voiceMapEntry || voiceMapEntry === DISABLED_VOICE_MARKER) { + throw `${char} not in voicemap. Configure character in extension settings voice map`; + } + const voice = await ttsProvider.getVoice(voiceMapEntry); + const voiceId = voice.voice_id; + if (voiceId == null) { + toastr.error(`Specified voice for ${char} was not found. Check the TTS extension settings.`); + throw `Unable to attain voiceId for ${char}`; + } + tts(text, voiceId, char); + } catch (error) { + console.error(error); + currentTtsJob = null; + } +} + +// Secret function for now +async function playFullConversation() { + const context = getContext(); + const chat = context.chat; + ttsJobQueue = chat; +} +window.playFullConversation = playFullConversation; + +//#############################// +// Extension UI and Settings // +//#############################// + +function loadSettings() { + if (Object.keys(extension_settings.tts).length === 0) { + Object.assign(extension_settings.tts, defaultSettings); + } + for (const key in defaultSettings) { + if (!(key in extension_settings.tts)) { + extension_settings.tts[key] = defaultSettings[key]; + } + } + $('#tts_provider').val(extension_settings.tts.currentProvider); + $('#tts_enabled').prop( + 'checked', + extension_settings.tts.enabled, + ); + $('#tts_narrate_dialogues').prop('checked', extension_settings.tts.narrate_dialogues_only); + $('#tts_narrate_quoted').prop('checked', extension_settings.tts.narrate_quoted_only); + $('#tts_auto_generation').prop('checked', extension_settings.tts.auto_generation); + $('#tts_narrate_translated_only').prop('checked', extension_settings.tts.narrate_translated_only); + $('#tts_narrate_user').prop('checked', extension_settings.tts.narrate_user); + $('#tts_pass_asterisks').prop('checked', extension_settings.tts.pass_asterisks); + $('body').toggleClass('tts', extension_settings.tts.enabled); +} + +const defaultSettings = { + voiceMap: '', + ttsEnabled: false, + currentProvider: 'ElevenLabs', + auto_generation: true, + narrate_user: false, +}; + +function setTtsStatus(status, success) { + $('#tts_status').text(status); + if (success) { + $('#tts_status').removeAttr('style'); + } else { + $('#tts_status').css('color', 'red'); + } +} + +function onRefreshClick() { + Promise.all([ + ttsProvider.onRefreshClick(), + // updateVoiceMap() + ]).then(() => { + extension_settings.tts[ttsProviderName] = ttsProvider.settings; + saveSettingsDebounced(); + setTtsStatus('Successfully applied settings', true); + console.info(`Saved settings ${ttsProviderName} ${JSON.stringify(ttsProvider.settings)}`); + initVoiceMap(); + updateVoiceMap(); + }).catch(error => { + console.error(error); + setTtsStatus(error, false); + }); +} + +function onEnableClick() { + extension_settings.tts.enabled = $('#tts_enabled').is( + ':checked', + ); + updateUiAudioPlayState(); + saveSettingsDebounced(); +} + + +function onAutoGenerationClick() { + extension_settings.tts.auto_generation = !!$('#tts_auto_generation').prop('checked'); + saveSettingsDebounced(); +} + + +function onNarrateDialoguesClick() { + extension_settings.tts.narrate_dialogues_only = !!$('#tts_narrate_dialogues').prop('checked'); + saveSettingsDebounced(); +} + +function onNarrateUserClick() { + extension_settings.tts.narrate_user = !!$('#tts_narrate_user').prop('checked'); + saveSettingsDebounced(); +} + +function onNarrateQuotedClick() { + extension_settings.tts.narrate_quoted_only = !!$('#tts_narrate_quoted').prop('checked'); + saveSettingsDebounced(); +} + + +function onNarrateTranslatedOnlyClick() { + extension_settings.tts.narrate_translated_only = !!$('#tts_narrate_translated_only').prop('checked'); + saveSettingsDebounced(); +} + +function onSkipCodeblocksClick() { + extension_settings.tts.skip_codeblocks = !!$('#tts_skip_codeblocks').prop('checked'); + saveSettingsDebounced(); +} + +function onPassAsterisksClick() { + extension_settings.tts.pass_asterisks = !!$('#tts_pass_asterisks').prop('checked'); + saveSettingsDebounced(); +} + +//##############// +// TTS Provider // +//##############// + +async function loadTtsProvider(provider) { + //Clear the current config and add new config + $('#tts_provider_settings').html(''); + + if (!provider) { + return; + } + + // Init provider references + extension_settings.tts.currentProvider = provider; + ttsProviderName = provider; + ttsProvider = new ttsProviders[provider]; + + // Init provider settings + $('#tts_provider_settings').append(ttsProvider.settingsHtml); + if (!(ttsProviderName in extension_settings.tts)) { + console.warn(`Provider ${ttsProviderName} not in Extension Settings, initiatilizing provider in settings`); + extension_settings.tts[ttsProviderName] = {}; + } + await ttsProvider.loadSettings(extension_settings.tts[ttsProviderName]); + await initVoiceMap(); +} + +function onTtsProviderChange() { + const ttsProviderSelection = $('#tts_provider').val(); + extension_settings.tts.currentProvider = ttsProviderSelection; + loadTtsProvider(ttsProviderSelection); +} + +// Ensure that TTS provider settings are saved to extension settings. +export function saveTtsProviderSettings() { + extension_settings.tts[ttsProviderName] = ttsProvider.settings; + updateVoiceMap(); + saveSettingsDebounced(); + console.info(`Saved settings ${ttsProviderName} ${JSON.stringify(ttsProvider.settings)}`); +} + + +//###################// +// voiceMap Handling // +//###################// + +async function onChatChanged() { + await resetTtsPlayback(); + const voiceMapInit = initVoiceMap(); + await Promise.race([voiceMapInit, delay(1000)]); + ttsLastMessage = null; +} + +async function onChatDeleted() { + const context = getContext(); + + // update internal references to new last message + lastChatId = context.chatId; + currentMessageNumber = context.chat.length ? context.chat.length : 0; + + // compare against lastMessageHash. If it's the same, we did not delete the last chat item, so no need to reset tts queue + let messageHash = getStringHash((context.chat.length && context.chat[context.chat.length - 1].mes) ?? ''); + if (messageHash === lastMessageHash) { + return; + } + lastMessageHash = messageHash; + ttsLastMessage = (context.chat.length && context.chat[context.chat.length - 1].mes) ?? ''; + + // stop any tts playback since message might not exist anymore + await resetTtsPlayback(); +} + +/** + * Get characters in current chat + * @param {boolean} unrestricted - If true, will include all characters in voiceMapEntries, even if they are not in the current chat. + * @returns {string[]} - Array of character names + */ +function getCharacters(unrestricted) { + const context = getContext(); + + if (unrestricted) { + const names = context.characters.map(char => char.name); + names.unshift(DEFAULT_VOICE_MARKER); + return names; + } + + let characters = []; + if (context.groupId === null) { + // Single char chat + characters.push(DEFAULT_VOICE_MARKER); + characters.push(context.name1); + characters.push(context.name2); + } else { + // Group chat + characters.push(DEFAULT_VOICE_MARKER); + characters.push(context.name1); + const group = context.groups.find(group => context.groupId == group.id); + for (let member of group.members) { + // Remove suffix + if (member.endsWith('.png')) { + member = member.slice(0, -4); + } + characters.push(member); + } + } + return characters; +} + +function sanitizeId(input) { + // Remove any non-alphanumeric characters except underscore (_) and hyphen (-) + let sanitized = input.replace(/[^a-zA-Z0-9-_]/g, ''); + + // Ensure first character is always a letter + if (!/^[a-zA-Z]/.test(sanitized)) { + sanitized = 'element_' + sanitized; + } + + return sanitized; +} + +function parseVoiceMap(voiceMapString) { + let parsedVoiceMap = {}; + for (const [charName, voiceId] of voiceMapString + .split(',') + .map(s => s.split(':'))) { + if (charName && voiceId) { + parsedVoiceMap[charName.trim()] = voiceId.trim(); + } + } + return parsedVoiceMap; +} + + + +/** + * Apply voiceMap based on current voiceMapEntries + */ +function updateVoiceMap() { + const tempVoiceMap = {}; + for (const voice of voiceMapEntries) { + if (voice.voiceId === null) { + continue; + } + tempVoiceMap[voice.name] = voice.voiceId; + } + if (Object.keys(tempVoiceMap).length !== 0) { + voiceMap = tempVoiceMap; + console.log(`Voicemap updated to ${JSON.stringify(voiceMap)}`); + } + if (!extension_settings.tts[ttsProviderName].voiceMap) { + extension_settings.tts[ttsProviderName].voiceMap = {}; + } + Object.assign(extension_settings.tts[ttsProviderName].voiceMap, voiceMap); + saveSettingsDebounced(); +} + +class VoiceMapEntry { + name; + voiceId; + selectElement; + constructor(name, voiceId = DEFAULT_VOICE_MARKER) { + this.name = name; + this.voiceId = voiceId; + this.selectElement = null; + } + + addUI(voiceIds) { + let sanitizedName = sanitizeId(this.name); + let defaultOption = this.name === DEFAULT_VOICE_MARKER ? + `` : + ``; + let template = ` +
+ ${this.name} + +
+ `; + $('#tts_voicemap_block').append(template); + + // Populate voice ID select list + for (const voiceId of voiceIds) { + const option = document.createElement('option'); + option.innerText = voiceId.name; + option.value = voiceId.name; + $(`#tts_voicemap_char_${sanitizedName}_voice`).append(option); + } + + this.selectElement = $(`#tts_voicemap_char_${sanitizedName}_voice`); + this.selectElement.on('change', args => this.onSelectChange(args)); + this.selectElement.val(this.voiceId); + } + + onSelectChange(args) { + this.voiceId = this.selectElement.find(':selected').val(); + updateVoiceMap(); + } +} + +/** + * Init voiceMapEntries for character select list. + * @param {boolean} unrestricted - If true, will include all characters in voiceMapEntries, even if they are not in the current chat. + */ +export async function initVoiceMap(unrestricted = false) { + // Gate initialization if not enabled or TTS Provider not ready. Prevents error popups. + const enabled = $('#tts_enabled').is(':checked'); + if (!enabled) { + return; + } + + // Keep errors inside extension UI rather than toastr. Toastr errors for TTS are annoying. + try { + await ttsProvider.checkReady(); + } catch (error) { + const message = `TTS Provider not ready. ${error}`; + setTtsStatus(message, false); + return; + } + + setTtsStatus('TTS Provider Loaded', true); + + // Clear existing voiceMap state + $('#tts_voicemap_block').empty(); + voiceMapEntries = []; + + // Get characters in current chat + const characters = getCharacters(unrestricted); + + // Get saved voicemap from provider settings, handling new and old representations + let voiceMapFromSettings = {}; + if ('voiceMap' in extension_settings.tts[ttsProviderName]) { + // Handle previous representation + if (typeof extension_settings.tts[ttsProviderName].voiceMap === 'string') { + voiceMapFromSettings = parseVoiceMap(extension_settings.tts[ttsProviderName].voiceMap); + // Handle new representation + } else if (typeof extension_settings.tts[ttsProviderName].voiceMap === 'object') { + voiceMapFromSettings = extension_settings.tts[ttsProviderName].voiceMap; + } + } + + // Get voiceIds from provider + let voiceIdsFromProvider; + try { + voiceIdsFromProvider = await ttsProvider.fetchTtsVoiceObjects(); + } + catch { + toastr.error('TTS Provider failed to return voice ids.'); + } + + // Build UI using VoiceMapEntry objects + for (const character of characters) { + if (character === 'SillyTavern System') { + continue; + } + // Check provider settings for voiceIds + let voiceId; + if (character in voiceMapFromSettings) { + voiceId = voiceMapFromSettings[character]; + } else if (character === DEFAULT_VOICE_MARKER) { + voiceId = DISABLED_VOICE_MARKER; + } else { + voiceId = DEFAULT_VOICE_MARKER; + } + const voiceMapEntry = new VoiceMapEntry(character, voiceId); + voiceMapEntry.addUI(voiceIdsFromProvider); + voiceMapEntries.push(voiceMapEntry); + } + updateVoiceMap(); +} + +$(document).ready(function () { + function addExtensionControls() { + const settingsHtml = ` +
+
+
+ TTS +
+
+
+
+
+ Select TTS Provider
+
+ + +
+
+ + + + + + + + +
+
+
+
+
+
+
+ +
+
+
+
+ + `; + $('#extensions_settings').append(settingsHtml); + $('#tts_refresh').on('click', onRefreshClick); + $('#tts_enabled').on('click', onEnableClick); + $('#tts_narrate_dialogues').on('click', onNarrateDialoguesClick); + $('#tts_narrate_quoted').on('click', onNarrateQuotedClick); + $('#tts_narrate_translated_only').on('click', onNarrateTranslatedOnlyClick); + $('#tts_skip_codeblocks').on('click', onSkipCodeblocksClick); + $('#tts_pass_asterisks').on('click', onPassAsterisksClick); + $('#tts_auto_generation').on('click', onAutoGenerationClick); + $('#tts_narrate_user').on('click', onNarrateUserClick); + $('#tts_voices').on('click', onTtsVoicesClick); + for (const provider in ttsProviders) { + $('#tts_provider').append($('