Custom Start-up Settings: Adjust your default start-up settings. Screenshot
Narrarator: Use different voices for main character and narration. Example Narration
Low VRAM mode: Improve generation performance if your VRAM is filled by your LLM. Screenshot
DeepSpeed: A 3-4x performance boost generating TTS. DeepSpeed Windows/Linux Instructions Screenshot
Local/Custom models: Use any of the XTTSv2 models (API Local and XTTSv2 Local).
Optional wav file maintenance: Configurable deletion of old output wav files. Screenshot
Documentation: Fully documented with a built in webpage. Screenshot
Console output Clear command line output for any warnings or issues.
Standalone/3rd Party support via JSON calls Can be used with 3rd party applications via JSON calls.

Index

🟩 Installation
🟪 Updating & problems with updating
🟫 Screenshots
🟨 Help with problems
🔵🟢🟡 DeepSpeed Installation (Windows & Linux)
🔴 Future to-do list & Upcoming updates

Updates

The latest build (13 Dec 2023) has had the entire text filtering engine and narration engine rebuilt from scratch. It's highly complicated how its actually working, but the end result it a much clearer TTS output and much better control over the narrator option and how to handle text that isnt within quotes or asterisks. Its a highly recommened update, for the improved quality it gives to the TTS output, if nothing else.

Should you want the older version of the narrator engine+text filtering, I will leave this older copy here

DeepSpeed v11.2 can be installed within the default text-generation-webui Python 3.11 environment. Instructions here (or scroll down). Please note, this is not an official Microsoft method (currently) so they will not support you with this style installation. Officially, only DeepSpeed v8.3 is installing on Python 3.9.x.

The one thing I cant easily work around

With a RP chat with your AI, on your character card parameters menu > character tab > greeting make sure that anything in there that is the narrator is in asterisks and anything spoken is in double quotes, then hit the save (disk) button. Greeting paragraphs/sentences are handled differently from how the AI sends text and so its difficut to account for them both.

I could force a delimeter in at this stage, but I know it would/may affect things further down the line in the chat and I need a good think about that before just making a change. This issue only affects the greeting card/start of conversation and the "example" card that comes with text-generation-webui suffers this issue (if you want to try it for yourself). So you would put double quotes around like this (from the example card):

"Hey! I'm so excited to finally meet you. I've heard so many great things about you and I'm eager to pick your brain about computers. I'm sure you have a wealth of knowledge that I can learn from."

🟩 Installation on Text generation web UI

This has been tested on the current Dec 2023 release of Text generation webUI. If you have not updated it for a while, you may wish to update Text generation webUI, instructions here

In a command prompt/terminal window you need to move into your Text generation webUI folder:

cd text-generation-webui
Start the Text generation webUI Python environment for your OS:

cmd_windows.bat, ./cmd_linux.sh, cmd_macos.sh or cmd_wsl.bat
Move into your extensions folder:

cd extensions
Once there git clone this repository:

git clone https://github.com/erew123/alltalk_tts
Move into the alltalk_tts folder:

cd alltalk_tts
Install the requirements:

Nvidia graphics card machines - pip install -r requirements_nvidia.txt

Other machines (mac, amd etc) - pip install -r requirements_other.txt
(Optional DeepSpeed) If you have an Nvidia Graphics card on a system running Linux or Windows and wish to use DeepSpeed please follow these instructions here. Though you may wish to check things are generally working with the below steps before installing DeepSpeed.
You can now start move back to the main Text generation webUI folder cd .. (a few times), start Text generation webUI (start_windows.bat,./start_linux.sh, start_macos.sh or start_wsl.bat) and load the AllTalk extension in the Text generation webUI session tab.
Please read the note below about start-up times and also the note above about ensuring your character cards are set up correctly
Some extra voices downloadable here

Note: It can take a while to start up. Check the command prompt/terminal window if you want to know what its doing. After it says "Model Loaded" the Text generation webUI is usually available on its IP address a few seconds later, for you to connect to in your browser.

Documentation: Click on the link when inside Text generation webUI as shown in the screenshot here

Where to find voices https://aiartes.com/voiceai or https://commons.wikimedia.org/ or interviews on youtube etc. Instructions on how to cut down and prepare a voice sample are within the built in documentation.

🟩 Other installation notes

On first startup, AllTalk will download the Coqui XTTSv2 2.0.2 model to its models folder (1.8GB space required). You can customse your model or use the TTS latest model within the interface (details in documentation).

Once the extension is loaded, please find all documentation and settings on the link provided in the interface (as shown in the screenshot below).

To start AllTalk every Text generation webUI loads, edit the Text generation webUI CMD_FLAGS.txt file in the main text-generation-webui folder and add --extensions alltalk_tts.

🟪 Updating

This is pretty much a repeat of the installation process.

In a command prompt/terminal window you need to move into your Text generation webUI folder:

cd text-generation-webui
Move into your extensions and alltalk_tts folder:

cd extensions then cd alltalk_tts
At the command prompt/terminal, type:

git pull
Install the requirements:

Nvidia graphics card machines - pip install -r requirements_nvidia.txt

Other machines (mac, amd etc) - pip install -r requirements_other.txt

🟪 git pull error

I did leave a mistake in the /extensions/alltalk_tts/.gitignore file at one point. If your git pull doesnt work, you can either follow the Problems Updating section below, or edit the .gitignore file and replace its entire contents with the below, save the file, then re-try the git pull

voices/*.*
models/*.*
outputs/*.*
config.json
models.json

🟪 Problems Updating

If you do experience any problems, the simplest method to resolve this will be:

re-name the existing alltalk_tts folder to something like alltalk_tts.old
Start a console/terminal then:

cd text-generation-webui and start your python environment cmd_windows.bat, ./cmd_linux.sh, cmd_macos.sh or cmd_wsl.bat
Move into the extensions folder, same as if you were doing a fresh installation:

cd extensions then

git clone https://github.com/erew123/alltalk_tts

This will download a fresh installation.

Move into the alltalk_tts folder:

cd alltalk_tts
Install the requirements:

Nvidia graphics card machines - pip install -r requirements_nvidia.txt

Other machines (mac, amd etc) - pip install -r requirements_other.txt
Before starting it up, copy/merge the models, voices and outputs folders over from the alltalk_tts.old folder to the newly created alltalk_tts folder. This will keep your voices history and also stop it re-downloading the model again.

You can now start text-generation-webui or AllTalk (standalone) and it should start up fine. You will need to re-set any saved configuration changes on the configuration page.

Assuming its all working fine and you are happy, you can delete the old alltalk_tts.old folder.

🟫 Screenshots

🟨 Help with problems

🟨 I activated DeepSpeed in the settings page, but I didnt install DeepSpeed yet and now I have issues starting up

You can either follow the Problems Updating and fresh install your config. Or you can edit the confignew.json file within the alltalk_tts folder. You would look for '"deepspeed_activate": true,' and change the word true to false `"deepspeed_activate": false,' ,then save the file and try starting again.

If you want to use DeepSpeed, you need an Nvidia Graphics card and to install DeepSpeed on your system. Instructions are here.

🟨 I am having problems getting AllTalk to start after changing settings or making a custom setup/model setup.

I would suggest following Problems Updating and if you still have issues after that, you can raise an issue here

🟨 I see some red "asyncio" messages

As far as I am aware, these are to do with the chrome browser the gradio text-generation-webui in some way. I raised an issue about this on the text-generation-webui here where you can see that AllTalk is not loaded and the messages persist. Either way, this is more a warning than an actual issue, so shouldnt affect any functionality of either AllTalk or text-generation-webui, they are more just an annoyance.

🟨 I am having problems updating/some other issue where it wont start up/Im sure this is a bug

Please see Problems Updating. If that doesnt help you can raise an ticket here. It would be handy to have any log files from the console where your error is being shown. I can only losely support custom built Python environments and give general pointers.

Also, is your text-generation-webui up to date? instructions here

🔵🟢🟡 DeepSpeed Installation Options

🔵 For Linux

➡️DeepSpeed requires an Nvidia Graphics card!⬅️

Covered in the online/buit-in documentation, but a nice easy install.

🟢🟡 For Windows & Python 3.11.x or 3.10.x

DeepSpeed v11.1 and v11.2 will work on the current text-generation-webui Python 3.11 environment! You have 2x options for how to setup DeepSpeed on Windows. A quick way (🟢Option 1) and a long way (🟡Option 2).

Thanks to @S95Sedan - They managed to get DeepSpeed 11.2 working on Windows via making some edits to the original Microsoft DeepSpeed v11.2 installation. The original post is here.

🟢 OPTION 1 - Quick and easy!

Pre-Compiled Wheel Deepspeed v11.1 (for Windows and Python 3.11)

➡️DeepSpeed requires an Nvidia Graphics card!⬅️

Note: In my tests, with this method you will not need to install the Nvidia CUDA toolkit to make this work, but AllTalk may warn you when starting DeepSpeed that it doesnt see the CUDA Toolkit, however, it works fine for TTS purposes.

Download the correct wheel version for your Python/Cuda from here and save the file it inside your text-generation-webui folder.
Open a command prompt window, move into your text-generation-webui folder, you can now start the Python environment for text-generation-webui:

cmd_windows.bat
With the file that you saved in the text-generation-webui folder you now type the following:

pip install "deepspeed-0.11.1+**THE-VERSION-YOU-ARE-USING**-win_amd64.whl" (The version you are using shouid match the file name you downloaded).
This should install through cleanly and you should now have DeepSpeed v11.1 installed within the Python 3.11 environment of text-generation-webui.
When you start up text-generation-webui, and AllTalk starts, you should see [AllTalk Startup] DeepSpeed Detected
Within AllTalk, you will now have a checkbox for Activate DeepSpeed though remember you can only change 1x setting every 15 or so seconds, so dont try to activate DeepSpeed and LowVRAM/Change your model simultantiously. Do one of those, wait 15-20 seconds until the change is confirmed in the terminal/command prompt, then you can change the other. When you are happy it works, you can set the default start-up settings in the settings page.

🟡 OPTION 2 - A bit more complicated!

Manual Build DeepSpeed v11.2 (for Windows and Python 3.11)

➡️DeepSpeed requires an Nvidia Graphics card!⬅️

DeepSpeed Version 11.2 with CUDA 11.8 or 12.1 - Installation Instructions:

Download the 11.2 release of DeepSpeed extract it to a folder.
Install Visual C++ build tools, such as VS2019 C++ x64/x86 build tools.
Download and install the Nvidia Cuda Toolkit 11.8 or 12.1
OPTIONAL If you do not have an python environment already created, you can install Miniconda, then at a command prompt, create and activate your environment with:

conda create -n pythonenv python=3.11
activate pythonenv

If you are using the text-generation-webui python environment, then in the text-generation-webui folder you will run cmd_windows.bat to start the python evnironment.
Launch the Command Prompt cmd with Administrator privilege as it requires admin to allow creating symlink folders.
Install PyTorch, 2.1.0 with CUDA 11.8 or 12.1 into your Python 3.1x.x environment e.g:

activate pythonenv (activate your python environment)
conda install pytorch==2.1.0 torchvision==0.16.0 torchaudio==2.1.0 pytorch-cuda=12.1 -c pytorch -c nvidia
or
conda install pytorch==2.1.0 torchvision==0.16.0 torchaudio==2.1.0 pytorch-cuda=11.8 -c pytorch -c nvidia
Set your CUDA Windows environment variables in the command prompt to ensure that CUDA_HOME and CUDA_PATH are set to your Nvidia Cuda Toolkit path. (The folder above the bin folder that nvcc.exe is installed in). Examples are:

set CUDA_HOME=C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.1
set CUDA_PATH=C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.1
or
set CUDA_HOME=C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.8
set CUDA_PATH=C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.8
Navigate to wherever you extracted the deepspeed folder in the Command Prompt:

cd c:\DeepSpeed-0.11.2 (wherever you extracted it to)
Modify the following files:
(These modified files are included in the git-pull of AllTalk, in the DeepSpeed Windows folder and so can just be copied over the top of the exsting folders/files, but if you want to modify them yourself, please follow the below)

deepspeed-0.11.2/build_win.bat - at the top of the file, add:

set DS_BUILD_EVOFORMER_ATTN=0

deepspeed-0.11.2/csrc/quantization/pt_binding.cpp - lines 244-250 - change to:

    std::vector<int64_t> sz_vector(input_vals.sizes().begin(), input_vals.sizes().end());
    sz_vector[sz_vector.size() - 1] = sz_vector.back() / devices_per_node;  // num of GPU per nodes
    at::IntArrayRef sz(sz_vector);
    auto output = torch::empty(sz, output_options);

    const int elems_per_in_tensor = at::numel(input_vals) / devices_per_node;
    const int elems_per_in_group = elems_per_in_tensor / (in_groups / devices_per_node);
    const int elems_per_out_group = elems_per_in_tensor / out_groups;

deepspeed-0.11.2/csrc/transformer/inference/csrc/pt_binding.cpp lines 541-542 - change to:

									 {static_cast<unsigned>(hidden_dim * InferenceContext::Instance().GetMaxTokenLength()),
									  static_cast<unsigned>(k * InferenceContext::Instance().GetMaxTokenLength()),

lines 550-551 - change to:

						 {static_cast<unsigned>(hidden_dim * InferenceContext::Instance().GetMaxTokenLength()),
						  static_cast<unsigned>(k * InferenceContext::Instance().GetMaxTokenLength()),

line 1581 - change to:

		at::from_blob(intermediate_ptr, {input.size(0), input.size(1), static_cast<int64_t>(mlp_1_out_neurons)}, options);

deepspeed-0.11.2/deepspeed/env_report.py line 10 - add:

import psutil

line 83 - 100 - change to:

def get_shm_size():
    try:
        temp_dir = os.getenv('TEMP') or os.getenv('TMP') or os.path.join(os.path.expanduser('~'), 'tmp')
        shm_stats = psutil.disk_usage(temp_dir)
        shm_size = shm_stats.total
        shm_hbytes = human_readable_size(shm_size)
        warn = []
        if shm_size < 512 * 1024**2:
            warn.append(
                f" {YELLOW} [WARNING] Shared memory size might be too small, consider increasing it. {END}"
            )
            # Add additional warnings specific to your use case if needed.
        return shm_hbytes, warn
    except Exception as e:
        return "UNKNOWN", [f"Error getting shared memory size: {e}"]

While still in your command line with python environment enabled run:
build_win.bat and wait 10-20 minutes.
Now cd dist to go into your dist folder and you can now pip install deepspeed-YOURFILENAME.whl (or whatever your WHL file is called).

🔴 Future to-do list

Complete & document the new/full standalone mode API.
Voice output within the command prompt/terminal (TBD).
Correct an issue on incorrect output folder path when running as a standalone app.
Correct a few spelling mistakes in the documnentation.
Possibly add some additional TTS engines (TBD).
Have a break!