Files
alltalk_tts/templates/admin.html
T
erew123 e183e3a7bb Filtering Demo page
removed some of the demo filtering as it was stripping too much from the text.
2024-01-03 09:36:33 +00:00

1434 lines
84 KiB
HTML
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
<!DOCTYPE html>
<html lang="en">
<head>
<link rel="shortcut icon" href="#">
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>AllTalk TTS for Text generation webUI</title>
<style>
body {
font-family: Arial, sans-serif;
max-width: 1200px;
/* Adjusted max-width for better readability */
margin: 40px auto;
padding: 20px;
background-color: #f4f4f4;
border: 1px solid #ddd;
border-radius: 8px;
box-shadow: 0 0 10px rgba(0, 0, 0, 0.1);
}
h1,
h2 {
color: black;
text-decoration: underline;
}
p,
span {
color: #555;
font-size: 16px;
/* Increased font size for better readability */
margin-top: 0;
/* Remove top margin for paragraphs */
}
code {
background-color: #f8f8f8;
border: 2px solid #ddd;
border-radius: 4px;
padding: 0px 0px;
font-size: 14px;
margin-bottom: 20px;
color: #3366ff;
}
pre {
background-color: #f8f8f8;
border: 1px solid #ddd;
border-radius: 4px;
padding: 10px;
overflow-x: auto;
font-size: 14px;
/* Adjusted font size for better visibility */
}
ul {
color: #555;
list-style-type: square;
/* Set the bullet style */
margin-left: 2px;
/* Adjust the left margin to create an indent */
}
li {
font-size: 16px;
/* Set the font size for list items */
margin-bottom: 8px;
/* Add some space between list items */
}
.key {
color: black;
/* Color for keys */
font-size: 14px;
/* Increased font size for better readability */
}
.value {
font-size: 14px;
/* Increased font size for better readability */
color: blue;
/* Color for values */
}
a {
color: #0077cc;
text-decoration: none;
}
a:hover {
text-decoration: underline;
}
strong {
font-weight: bold;
}
.option-a {
color: #33cc33;
font-weight: bold;
}
.option-b {
color: red;
font-weight: bold;
}
/* New styles for TTS Request Page */
#container {
max-width: 1000px;
margin: 50px auto;
padding: 20px;
background-color: #fff;
border-radius: 8px;
box-shadow: 0 0 10px rgba(0, 0, 0, 0.1);
}
form {
display: flex;
flex-direction: column;
gap: 4px;
}
#outputFileContainer {
display: flex;
flex-direction: column;
gap: 4px;
}
label {
font-weight: bold;
font-size: 14px;
padding: 2px;
}
textarea,
input,
select,
button {
padding: 4px;
font-size: 14px;
border: 1px solid #ccc;
border-radius: 4px;
padding: 4px;
}
textarea {
resize: vertical;
}
button {
background-color: #4caf50;
color: #fff;
cursor: pointer;
transition: background-color 0.3s;
}
button:hover {
background-color: #45a049;
}
p {
margin-top: 20px;
}
#outputFilePath {
font-weight: bold;
}
#audioSource {
display: block;
margin: auto;
}
#outputFilePath {
display: none;
}
table {
border-collapse: collapse;
width: 100%;
margin-bottom: 20px;
}
th,
td {
border: 1px solid #ddd;
padding: 8px;
text-align: left;
}
/* Style for the first table */
#configuration-details table {
font-size: 16px;
}
/* Style for the nested table */
#modeldownload-table table {
font-size: 16px;
}
#modeldownload-table table td {
padding: 4px;
/* Adjust padding as needed for the nested table */
}
</style>
</head>
<body>
<h1 id="toc">AllTalk TTS for Text generation webUI</h1>
<iframe src="http://{{ params.ip_address }}:{{ params.port_number }}/settings" width="100%" height="500"
frameborder="0" style="margin: 0; padding: 0;"></iframe>
<h2>Table of Contents</h2>
<ul>
<li><a href="#getting-started">Getting Started</a></li>
<li><a href="#server-information">Server Information</a></li>
<li><a href="#demotesttts">Demo/Test TTS output</a></li>
<li><a href="#using-voice-samples">Using Voice Samples</a></li>
<li><a href="#text-not-inside">Text Not inside</a></li>
<li><a href="#low-vram">Low VRAM</a></li>
<li><a href="#finetune">Finetuning (Training the model)</a></li>
<li><a href="#deepspeed">DeepSpeed</a>
<ul>
<li><a href="#deepspeed-linux">DeepSpeed Setup Linux</a></li>
<li><a href="#deepspeed-windows">DeepSpeed Setup Windows</a></li>
</ul>
</li>
<li><a href="#TTSmodels">TTS Models/Methods</a></li>
<li><a href="#temperature-and-repetition-settings">Model temperature and repetition settings</a></li>
<li><a href="#start--up-checks">Start-up checks</a></li>
<li><a href="#customTTSmodels">Custom TTS Models and Model path</a></li>
<li><a href="#configuration-details">Configuration file settings</a></li>
<li><a href="#curl-commands">JSON calls and CURL Commands</a></li>
<li><a href="#debugging-and-tts-generation-information">Debugging and TTS Generation Information</a></li>
<li><a href="#references">Thanks &amp; References</a></li>
</ul>
<h2 id="getting-started">Getting Started</h2>
<p style="padding-left: 30px; text-align: justify;">AllTalk is a web interface, based around the Coqui TTS voice
cloning/speech generation system. To generate TTS, you can use the provided gradio interface or interact with
the server using JSON/CURL commands.&nbsp;</p>
<p style="padding-left: 30px; text-align: justify;"><span style="color: #ff0000;">Note:</span> When loading up a new
character in Text generation webUI it may look like nothing is happening for 20-30 seconds. It's actually
processing the introduction section (greeting message) of the text and once that is completed, it will appear.
You can see the activity occurring in the console window. Refreshing the page multiple times will try force the
TTS engine to keep re-generating the text, so please just wait and check the console if needed.</p>
<p style="padding-left: 30px; text-align: justify;"><span style="color: #ff0000;">Note: </span>Ensure that your RP
character card has asterisks around anything for the narration and double quotes around anything spoken. There
is a complication ONLY with the greeting card so, ensuring it has the correct use of quotes and asterisks will
help make sure that the greeting card sounds correct. I will aim to address this issue in a future update. In
Text-generation-webUI <span style="color: #3366ff;">parameters menu</span> &gt; <span
style="color: #3366ff;">character tab</span> &gt; <span style="color: #3366ff;">greeting</span>&nbsp;make
sure that anything in there that is the narrator is in asterisks and anything spoken is in double quotes, then
hit the <span style="color: #3366ff;">save</span> (💾) button.</p>
<p style="padding-left: 30px;"><span style="text-decoration: underline;"><strong>AllTalk <span
style="color: #ff0000; text-decoration: underline;">Minor Bug Fixes Changelog &amp; Known
issues</span></strong></span></p>
<p style="padding-left: 30px;">If I squash any minor bugs or find any issues, I will try to apply an update asap. If
you think something isnt working correctly or you have a problem, check these two links below first.</p>
<ul>
<li>Minor bug fixes changelog&nbsp;<a href="https://github.com/erew123/alltalk_tts/issues/25" target="_blank"
rel="noopener">link here</a>.</li>
<li>Help and known issues <a href="https://github.com/erew123/alltalk_tts?#-help-with-problems" target="_blank"
rel="noopener">link here</a>.</li>
</ul>
<p style="padding-left: 30px;"><span style="text-decoration: underline;"><strong>AllTalk</strong></span>
<br />Github <a href="https://github.com/erew123/alltalk_tts" target="_blank" rel="noopener">link
here</a><br />Update instructions <a href="https://github.com/erew123/alltalk_tts?#-updating"
target="_blank" rel="noopener">link here</a><br />Help and issues <a
href="https://github.com/erew123/alltalk_tts?#-help-with-problems" target="_blank" rel="noopener">link
here<br /><br /></a><span style="text-decoration: underline;"><strong>Text generation webUI</strong></span>
<br />Web interface&nbsp;<a href="http://{{ params.ip_address }}:7860" target="_blank" rel="noopener">link
here</a> <br />Documentation <a href="https://github.com/oobabooga/text-generation-webui/wiki"
target="_blank" rel="noopener">link here</a>
</p>
<p style="padding-left: 30px;"><a href="#toc">Back to top of page</a></p>
<h2 id="server-information">Server Information</h2>
<ul>
<li>Base URL: <code>http://{{ params.ip_address }}:{{ params.port_number }}</code></li>
<li>Server Status:
<code><a href="http://{{ params.ip_address }}:{{ params.port_number }}/ready">http://{{ params.ip_address }}:{{ params.port_number }}/ready</a></code>
</li>
</ul>
<p style="padding-left: 30px;"><a href="#toc">Back to top of page</a></p>
<h2 id="demotesttts">Demo/Test TTS</h2>
<div id="container">
<form method="post" action="/tts-demo-request" id="ttsForm">
<label for="text">Text:</label>
<textarea id="text" name="text" rows="4" required></textarea>
<label for="voice">Voice:</label>
<select id="voice" name="voice" required>
<!-- Options will be populated here by JavaScript -->
</select>
<label for="language">Language:</label>
<select id="language" name="language" value="English" required>
<option value="en" selected>English</option>
<option value="ar">Arabic</option>
<option value="zh-cn">Chinese</option>
<option value="cs">Czech</option>
<option value="nl">Dutch</option>
<option value="fr">French</option>
<option value="de">German</option>
<option value="hu">Hungarian</option>
<option value="it">Italian</option>
<option value="ja">Japanese</option>
<option value="ko">Korean</option>
<option value="pl">Polish</option>
<option value="pt">Portuguese</option>
<option value="ru">Russian</option>
<option value="es">Spanish</option>
<option value="tr">Turkish</option>
</select>
<div id="outputFileContainer">
<label for="outputFile">Output File:</label>
<input type="text" id="outputFile" name="output_file" value="demo_output.wav" required>
</div>
<label for="streaming">Streaming:</label>
<select id="streaming" name="streaming" value="false" required>
<option value="false" selected>No</option>
<option value="true">Yes</option>
</select>
<!-- Audio player with autoplay -->
<audio controls autoplay id="audioSource">
<source type="audio/wav" />
Your browser does not support the audio element.
</audio>
<span id="outputFilePath" style="height: 0px;">{{ output_file_path }}</span>
<button id="submit" type="submit">
Generate TTS
</button>
</form>
</div>
<script>
document.addEventListener('DOMContentLoaded', function () {
const ipAddress = "{{ params.ip_address }}";
const portNumber = "{{ params.port_number }}";
const url = `http://${ipAddress}:${portNumber}/api/voices`;
fetch(url)
.then(response => response.json())
.then(data => {
const selectElement = document.getElementById('voice');
data.voices.forEach(voice => {
const option = document.createElement('option');
option.value = voice;
option.textContent = voice;
selectElement.appendChild(option);
});
})
.catch(error => console.error('Error fetching voices:', error));
});
</script>
<script>
// Audio player
const audioPlayer = document.getElementById('audioSource');
function enableLoader(enable) {
// Change the submit button text and disable it while loading
const submit = document.getElementById('submit');
submit.disabled = enable ? true : false;
submit.style.opacity = enable ? 0.7 : 1.0;
submit.innerText = enable ? 'Generating...' : 'Generate TTS';
// Disable the audio player while loading
audioPlayer.disabled = enable ? true : false;
audioPlayer.style.opacity = enable ? 0.7 : 1.0;
audioPlayer.style.pointerEvents = enable ? 'none' : 'auto';
}
audioPlayer.addEventListener('canplay', () => {
enableLoader(false);
audioPlayer.play();
});
audioPlayer.addEventListener('abort', () => {
enableLoader(false);
});
audioPlayer.addEventListener('error', (e) => {
enableLoader(false);
});
// Streaming selector
const streaming = document.getElementById('streaming');
streaming.addEventListener('change', (event) => {
const outputFileContainer = document.getElementById('outputFileContainer');
if (event.target.value === 'true') {
outputFileContainer.style.display = 'none';
} else {
outputFileContainer.style.display = 'flex';
}
});
// Form submit
const form = document.getElementById('ttsForm');
const ipAddress = "{{ params.ip_address }}";
const portNumber = "{{ params.port_number }}";
const baseurl = `http://${ipAddress}:${portNumber}`;
form.addEventListener('submit', async (event) => {
event.preventDefault();
enableLoader(true);
audioPlayer.pause();
const formData = new FormData(form);
// Get and clean the text input from the textarea
let textInput = formData.get('text');
let cleanedText = textInput.replace(/ \- | \– /g, ' '); // Replace hyphens
cleanedText = cleanedText.replace(/%/g, " percent");
//cleanedText = cleanedText.replace(/([!?.])\1+/g, '$1');
cleanedText = cleanedText.replace(/[^a-zA-Z0-9\s\.,;:!?\-\'"]/g, '');
cleanedText = cleanedText.replace(/\n+/g, ' ');
cleanedText = cleanedText.replace(/\.\s'/g, ".'"); // Remove space between period and single quote
//cleanedText = cleanedText.replace(/'/g, ""); // Remove all single quotes
// Set the cleaned text back to the form data
formData.set('text', cleanedText);
// Rest of your form submission code...
// Check if streaming or not and handle accordingly
if (formData.get('streaming') === 'true') {
// For streaming, update the audio player src
const searchParams = new URLSearchParams(formData).toString();
audioPlayer.src = baseurl + "/tts-demo-request?" + searchParams;
audioPlayer.load(); // Reload the audio player
} else {
// For non-streaming, send a POST request
const response = await fetch('/tts-demo-request', {
method: 'POST',
body: formData,
cache: "no-store"
});
// Process the response
const result = await response.json();
// Update output file path and audio player source
const outputFilePath = document.getElementById('outputFilePath');
outputFilePath.textContent = result.output_file_path;
audioPlayer.src = baseurl + `/audio/${result.output_file_path}`;
audioPlayer.load(); // Reload the audio player
}
});
</script>
<p style="padding-left: 30px;"><a href="#toc">Back to top of page</a></p>
<h2 id="using-voice-samples"><strong>Using Voice Samples</strong></h2>
<h4 id="where-are-the-sample-voices-stored">Where are the sample voices stored?</h4>
<p style="padding-left: 30px;">Voice samples are stored in <span style="color: #3366ff;">/alltalk_tts/voices/</span>
and should be named using the following format <span style="color: #3366ff;">name.wav</span></p>
<h4 id="where-are-the-outputs-stored">Where are the outputs stored &amp; Automatic output wav file deletion.</h4>
<p style="padding-left: 30px; text-align: justify;">Voice outputs are stored in&nbsp;<span
style="color: #3366ff;">/alltalk_tts/outputs/</span></p>
<p style="padding-left: 30px; text-align: justify;">You can configure automatic maintenance deletion of old wav
files by setting <span style="color: #3366ff;">Del WAV's older than</span> in the settings above.</p>
<p style="padding-left: 30px; text-align: justify;">When <span style="color: #3366ff;">Disabled</span> your output
wav files will be left untouched. When set to a setting <span style="color: #3366ff;">1 Day</span> or greater,
your output wav files older than that time period will be automatically deleted on start-up of AllTalk.</p>
<h4>Where are the models stored?</h4>
<p style="padding-left: 30px; text-align: justify;">This extension will download the 2.0.2 model to <span
style="color: #3366ff;">/alltalk_tts/models/</span></p>
<p style="padding-left: 30px; text-align: justify;">This TTS engine will also download the latest available model
and store it wherever your OS normally stores it (Windows/Linux/Mac).</p>
<h4>How do I create a new voice sample?</h4>
<p style="padding-left: 30px; text-align: justify;">To create a new voice sample, you need to make a wav file that
is
<span style="color: #3366ff;">22050Hz</span>, <span style="color: #3366ff;">Mono</span>, <span
style="color: #3366ff;">16 bit</span> and between 6 to 30 seconds long, though 8 to 10 seconds is usually
good enough. The model can handle up to 30 second samples, however I've not noticed any improvement in voice
output from a much longer clips.
</p>
<p style="padding-left: 30px; text-align: justify;">You want to find a nice clear selection of audio, so lets say
you wanted to clone your favourite celebrity. You may go looking for an interview where they are talking. Pay
close attention to the audio you are listening to and trying to sample. Are there noises in the background, hiss
on the soundtrack, a low humm, some quiet music playing or something? The better quality the audio the better
the final TTS result. Don't forget, the AI that processes the sounds can hear everything in your sample and it
will use them in the voice its trying to recreate.</p>
<p style="padding-left: 30px; text-align: justify;">Try make your clip one of nice flowing speech, like the included
example files. No big pauses, gaps or other sounds. Preferably a sample that the person you are trying to copy
will show a little vocal range and emotion in their voice. Also, try to avoid a clip starting or ending with
breathy sounds (breathing in/out etc).</p>
<h4>Editing your sample!</h4>
<p style="padding-left: 30px; text-align: justify;">So, you’ve downloaded your favourite celebrity interview off
YouTube, from here you need to chop it down to 6 to 30 seconds in length and resample it.</p>
<p style="text-align: justify; padding-left: 30px;">If you need to clean it up, do audio processing, volume level
changes etc, do this before down-sampling.<br /><br />Using the latest version of Audacity <span
style="color: #3366ff;">select/highlight</span> your 6 to 30 second clip and:<br /><br /><span
style="color: #3366ff;">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; Tracks</span>
&gt;<span style="color: #3366ff;"> Resample to 22050Hz</span>&nbsp;then<br /><span
style="color: #3366ff;">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; Tracks</span> &gt;
<span style="color: #3366ff;">Mix</span> &gt; <span style="color: #3366ff;">Stereo to
Mono&nbsp;</span>then<br /><span style="color: #3366ff;">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; File</span> &gt; <span style="color: #3366ff;">Export Audio</span>&nbsp;saving it as a <span
style="color: #3366ff;">WAV</span> of <span style="color: #3366ff;">22050Hz</span>.
</p>
<p style="padding-left: 30px; text-align: justify;">Save your generated wav file in the&nbsp;<span
style="color: #3366ff;">/alltalk_tts/voices/ <span style="color: #808080;">folder.</span></span></p>
<p style="padding-left: 30px; text-align: justify;">Its worth mentioning that using AI generated audio clips may
introduce unwanted sounds as its already a copy/simulation of a voice.</p>
<h4>Why doesnt it sound like XXX Person?</h4>
<p style="padding-left: 30px; text-align: justify;">Maybe you might be interested in trying <a
href="#finetune">Finetuning of the model</a>. Otherwise, the reasons can be that you:</p>
<ul style="text-align: justify;">
<li>Didn't down-sample it as above.</li>
<li>Have a bad quality voice sample.</li>
<li>Try using the 3x different generation methods: <span style="color: #3366ff;">API TTS</span>, <span
style="color: #3366ff;">API Local</span>, and <span style="color: #3366ff;">XTTSv2 Local</span> within
the web interface, as they generate output in different ways and sound different.</li>
</ul>
<p style="padding-left: 30px; text-align: justify;">Some samples just never seem to work correctly, so maybe try a
different sample. Always remember though, this is an AI model attempting to re-create a voice, so you will never
get a 100% match.</p>
<p style="padding-left: 30px;"><a href="#toc">Back to top of page</a></p>
<h2 id="text-not-inside"><strong>Text Not inside</strong></h2>
<p style="padding-left: 30px; text-align: justify;">This only affects the Narrator function. Most AI models should
be using asterisks or double quotes to differentiate between the Narrator or the Character, however, many models
sometimes switch between using asterisks and double quotes or sometimes nothing at all for the text it outputs.
<br /><br />This leaves a bit of a mess because sometimes un-marked text is narration and sometimes its the
character talking, leaving no clear way to know where to split sentences and what voice to use. Whilst there is
no 100% solution at the moment many models lean more one way or the other as to what that unmarked text will be
(character or narrator).
</p>
<p style="padding-left: 30px; text-align: justify;">As such, the "Text not inside" function at least gives you the
choice to set how you want the TTS engine to handle situations of un-marked text.</p>
<div style="text-align: center;"><img src="/static/textnotinside.jpg"
alt="When the AI doesnt use an asterisk or a quote" /></div>
<p style="padding-left: 30px;"><a href="#toc">Back to top of page</a></p>
<h2 id="low-vram"><strong>Low VRAM</strong></h2>
<p style="padding-left: 30px; text-align: justify;">The Low VRAM option is a crucial feature designed to enhance
performance under constrained (VRAM) conditions, as the TTS models require 2GB-3GB of VRAM to run effectively.
This feature strategically manages the relocation of the Text-to-Speech (TTS) model between your system's Random
Access Memory (RAM) and VRAM, moving it between the two on the fly. Obviously, this is very useful for people
who have smaller graphics cards and will use all their VRAM to load in their LLM.</p>
<p style="padding-left: 30px; text-align: justify;">When you don't have enough VRAM free after loading your LLM
model
into your VRAM (Normal Mode example below), you can see that with so little working space, your GPU will have to
swap in and out bits of the TTS model, which causes horrible slowdown.</p>
<p style="padding-left: 30px; text-align: justify;"><span style="color: #ff0000;">Note:</span> An Nvidia Graphics
card is required for the LowVRAM option to work, as you will just be using system RAM otherwise.&nbsp;</p>
<h4>How It Works:</h4>
<p style="padding-left: 30px; text-align: justify;">The Low VRAM mode intelligently orchestrates the relocation of
the entire TTS model and stores the TTS model in your system RAM. When the TTS engine requires VRAM for
processing, the entire model seamlessly moves into VRAM, causing your LLM to unload/displace some layers,
ensuring optimal performance of the TTS engine.</p>
<p style="padding-left: 30px; text-align: justify;">Post-TTS processing, the model moves back to system RAM, freeing
up VRAM space for your Language Model (LLM) to load back in the missing layers. This adds about 1-2 seconds to
both text generation by the LLM and the TTS engine.</p>
<p style="padding-left: 30px; text-align: justify;">By transferring the entire model between RAM and VRAM, the Low
VRAM option avoids fragmentation, ensuring the TTS model remains cohesive and has all the working space it needs
in your GPU, without having to just work on small bits of the TTS model at a time (which causes terrible slow
down).</p>
<p style="padding-left: 30px; text-align: justify;">This creates a TTS generation performance Boost for Low VRAM
Users and is particularly beneficial for users with less than 2GB of free VRAM after loading their LLM,
delivering a substantial 5-10x improvement in TTS generation speed.</p>
<div style="text-align: center;"><img src="/static/lowvrammode.png" alt="How Low VRAM Works" /></div>
<p style="padding-left: 30px;"><a href="#toc">Back to top of page</a></p>
<h2 id="finetune"><strong>Finetuning (Training the model)</strong></h2>
<p style="margin-left:40px; text-align:justify">If you have a voice that the model doesnt quite reproduce correctly,
or indeed you just want to improve the reproduced voice, then finetuning is a way to train your &quot;XTTSv2
local&quot; model <span style="color:#3366ff">(stored in /alltalk_tts/models/xxxxx/)</span> on a specific voice.
For this you will need:</p>
<ul style="margin-left:40px">
<li>An Nvidia graphics card To install a few portions of the Nvidia CUDA <strong>11.8</strong> Toolkit (this
will not impact text-generation-webui&#39;s cuda setup.</li>
<li>18GB of disk space free (most of this is used temporarily)</li>
<li>At least 2 minutes of good quality speech from your chosen speaker in mp3, wav or flacc format, in one or
more files (have tested as far as 20 minutes worth of audio).</li>
</ul>
<h4>How will this work/How complicated is it?</h4>
<p style="margin-left:40px; text-align:justify">Everything has been done to make this as simple as possible. At its
simplest, you can literally just download a large chunk of audio from an interview, and tell the finetuning to
strip through it, find spoken parts and build your dataset. You can literally click 4 buttons, then copy a few
files and you are done. At it&#39;s more complicated end you will clean up the audio a little beforehand, but
its still only 4x buttons and copying a few files.</p>
<h4>The audio you will use:</h4>
<p style="margin-left:40px; text-align:justify">I would suggest that if its in an interview format, you cut out the
interviewer speaking in audacity or your chosen audio editing package. You don't have to worry about being
perfect with your cuts, the finetuning Step 1 will go and find spoken audio and cut it out for you. Is there is
music over the spoken parts, for best quality you would cut out those parts, though its not 100% necessary. As
always, try to avoid bad quality audio with noises in it (humming sounds, hiss etc). You can try something like
Audioenhancer to try clean up noisier audio. There is no need to down-sample any of the audio, all of that is
handled for you. Just give the finetuning some good quality audio to work with.</p>
<h4>Important requirements CUDA 11.8:</h4>
<p style="margin-left:40px; text-align:justify">As mentioned you must have a small portion of the <span
style="color:#3366ff">Nvidia CUDA Toolkit&nbsp;<strong>11.8</strong></span>&nbsp;installed. Not higher or
lower versions. Specifically&nbsp;<strong>11.8</strong>. You do not have to uninstall any other versions, change
any graphics drivers, reinstall torch or anything like that. There are instructions within the finetuning
interface on doing this or you can also find them on this link&nbsp;<a
href="https://github.com/erew123/alltalk_tts#-important-requirements-cuda-118">here</a></p>
<h4>Starting Finetuning:</h4>
<p style="margin-left:40px"><strong>Ensure</strong> you have followed the instructions on setting up the Nvidia CUDA
Toolkit 11.8 <a href="https://github.com/erew123/alltalk_tts#-important-requirements-cuda-118">here</a>&nbsp;or
the below procedure will fail.</p>
<p style="margin-left:40px">The below instructions are also available online&nbsp;<a
href="https://github.com/erew123/alltalk_tts#-finetuning-a-model">here</a></p>
<ol>
<li>
<p>Close all other applications that are using your GPU/VRAM and copy your audio samples into:<br />
<br />
<span style="color:#3366ff">/alltalk_tts/finetune/put-voice-samples-in-here/</span>
</p>
</li>
<li>
<p>In a command prompt/terminal window you need to move into your Text generation webUI folder:<br />
<br />
<span style="color:#3366ff">cd text-generation-webui</span>
</p>
</li>
<li>
<p>Start the Text generation webUI Python environment for your OS:<br />
<br />
<span
style="color:#3366ff">cmd_windows.bat,&nbsp;./cmd_linux.sh,&nbsp;cmd_macos.sh&nbsp;or&nbsp;cmd_wsl.bat</span>
</p>
</li>
<li>
<p>You can double check your search path environment still works correctly with&nbsp;<span
style="color:#3366ff">nvcc --version</span>. It should report back <span
style="color:#3366ff">11.8</span>:<br />
<br />
<span style="color:#3366ff">Cuda compilation tools, release 11.8.</span>
</p>
</li>
<li>
<p>Move into your extensions folder:<br />
<br />
<span style="color:#3366ff">cd extensions</span>
</p>
</li>
<li>
<p>Move into the&nbsp;<span style="color:#3366ff">alltalk_tts</span>&nbsp;folder:<br />
<br />
<span style="color:#3366ff">cd alltalk_tts</span>
</p>
</li>
<li>
<p>Install the finetune requirements file:&nbsp;<span style="color:#3366ff">pip install -r
requirements_finetune.txt</span></p>
</li>
<li>
<p>Type<span style="color:#3366ff">&nbsp;python finetune.py</span>&nbsp;and it should start up.</p>
</li>
<li>
<p>Follow the on-screen instructions when the web interface starts up.</p>
</li>
<li>
<p>When you have finished finetuning, the final tab will tell you what to do with your files and how to move
your newly trained model to the correct location on disk.</p>
</li>
</ol>
<p><a href="#toc">Back to top of page</a></p>
<h2 id="deepspeed"><strong>DeepSpeed</strong></h2>
<p style="padding-left: 30px; text-align: justify;">DeepSpeed provides a 2x-3x speed boost for Text-to-Speech and AI
tasks. It's all about making AI and TTS happen faster and more efficiently.</p>
<ul>
<li><strong>Model Parallelism:</strong> Spreads work across multiple GPUs, making AI/TTS models handle tasks
more efficiently.</li>
<li><strong>Memory Magic:</strong> Optimizes how memory is used, reducing the memory needed for large AI/TTS
models.</li>
<li><strong>Handles More Load:</strong> Scales up to handle larger workloads with improved performance.</li>
<li><strong>Smart Resource Use:</strong> Uses your computer's resources smartly, getting the most out of your
hardware.</li>
</ul>
<div style="text-align: center;"><img src="/static/deepspeedexample.jpg" alt="DeepSpeed on vs off" /></div>
<p style="padding-left: 30px; text-align: justify;">DeepSpeed only works with the <span
style="color: #3366ff;">XTTSv2 Local</span> model and will deactivate when other models are selected, even
if the checkbox still shows as being selected.</p>
<p style="padding-left: 30px; text-align: justify;"><span style="color: #ff0000;">Note:</span> DeepSpeed/AllTalk may
warn if the Nvidia Cuda Toolkit and CUDA_HOME environment variable isnt set correctly. On Linux you need
CUDA_HOME configured correctly; on Windows, if you use the pre-built wheel its ok without.</p>
<p style="padding-left: 30px; text-align: justify;"><span
style="color: #ff0000;">Note:</span><strong>&nbsp;</strong>You do <strong>not</strong> need to set
Text-generation-webUi's --deepspeed setting for AllTalk to be able to use DeepSpeed.</p>
<p style="padding-left: 30px;"><a href="#toc">Back to top of page</a></p>
<h3 id="deepspeed-linux">DeepSpeed Setup - Linux</h3>
<p style="padding-left: 30px;">➡️DeepSpeed requires an Nvidia Graphics card!⬅️</p>
<ol>
<li>Preferably use your built in package manager to install the CUDA toolkit. Alternatively download and install
the Nvidia Cuda Toolkit for Linux <a href="https://developer.nvidia.com/cuda-toolkit-archive">Nvidia Cuda
Toolkit 11.8 or 12.1</a></li>
<li>Open a terminal console.</li>
<li>Install <span style="color: #3366ff;">libaio-dev</span> (however your Linux version installs things) e.g.
<span style="color: #3366ff;">sudo apt install libaio-dev</span>
</li>
<li>Move into your Text generation webUI folder e.g. <span style="color: #0000ff;"><span
style="color: #3366ff;">cd text-generation-webui</span><br /></span></li>
<li>Start the Text generation webUI Python environment <span style="color: #0000ff;"><span
style="color: #3366ff;">./cmd_linux.sh</span><br /></span></li>
<li style="text-align: left;">Text generation webUI <strong>overwrites</strong> the <strong>CUDA_HOME</strong>
environment variable each time you <span style="color: #3366ff;">./cmd_linux.sh</span> or <span
style="color: #3366ff;">./start_linux.sh&nbsp;</span>so you will need to either permanently change
within the python environment OR set CUDA_HOME it each time you <span
style="color: #3366ff;">./cmd_linux.sh</span>. Details to change it each time are on the next step.
Below is a link to Conda's manual and changing environment variables permanently though its possible
changing it permanently could affect other extensions, you would have to test.<a
href="https://conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#set-env-vars">
Conda manual - Environment variables<br /></a></li>
<li>You can temporarily set the <span style="color: #3366ff;">CUDA_HOME</span> environment with (Standard Ubuntu
path below, but it could vary on other Linux flavours):<br /><br /><span style="color: #0000ff;"><span
style="color: #3366ff;">export
CUDA_HOME=/etc/alternatives/cuda</span>&nbsp;</span><strong>every</strong> time you run <span
style="color: #0000ff;"><span style="color: #808080;">.<span
style="color: #3366ff;">/cmd_linux.sh</span></span><br /><br /></span>If you try to start
DeepSpeed with the CUDA_HOME path set incorrectly, expect an error similar to:<br /><br /><span
style="color: #ffcc00;">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;[Errno 2] No such file or directory:
/home/yourname/text-generation-webui/installer_files/env/bin/nvcc<br /><br /></span></li>
<li>Now install deepspeed with <span style="color: #0000ff;"><span style="color: #3366ff;">pip install
deepspeed</span><br /></span></li>
<li>You can now start Text generation webUI <span style="color: #3366ff;">python server.py</span> ensuring to
activate your extensions.<br /><br />Just to reiterate, starting Text-generation-webUI with <span
style="color: #3366ff;">./start_linux.sh</span> will overwrite the CUDA_HOME variable unless you have
permanently changed it, hence always starting it with <span
style="color: #3366ff;">./cmd_linux.sh</span>&nbsp;<strong>then</strong> setting the environment
variable manually <span style="color: #3366ff;">export CUDA_HOME=/etc/alternatives/cuda</span> and
<strong>then</strong> <span style="color: #3366ff;">python server.py</span>&nbsp;which is how you would need
to run it each time, unless you permanently set the environment variable for CUDA_HOME within
Text-generation-webUI's standard Python environment.<br /><br /><span
style="color: #ff0000;">Removal</span><strong> -</strong> If it became necessary to uninstall DeepSpeed,
you can do so with <span style="color: #3366ff;">./cmd_linux.sh</span> and then <span
style="color: #3366ff;">pip uninstall deepspeed</span>
</li>
</ol>
<p style="padding-left: 30px;"><a href="#toc">Back to top of page</a></p>
<h3 id="deepspeed-windows">DeepSpeed Setup - Windows</h3>
<p style="padding-left: 30px;">➡️DeepSpeed requires an Nvidia Graphics card!⬅️</p>
<p style="padding-left: 30px;">DeepSpeed v11.2 will work on the current default text-generation-webui Python 3.11
environment! You have 2x options for how to setup DeepSpeed on Windows. A quick way <span
style="color: #339966;"><strong>Option 1</strong></span>&nbsp;and a long way <span
style="color: #ff9900;"><strong>Option 2</strong></span>.</p>
<p style="padding-left: 30px;">Thanks to <a href="https://github.com/S95Sedan">@S95Sedan</a> They managed to get
DeepSpeed 11.2 working on Windows via making some edits to the original Microsoft DeepSpeed v11.2 installation.
The original post is <a
href="https://github.com/oobabooga/text-generation-webui/issues/4734#issuecomment-1843984142">here</a>.<br /><br />
</p>
<p style="padding-left: 30px;"><strong><span style="text-decoration: underline;"><span
style="color: #339966; text-decoration: underline;">OPTION 1 - <span
style="color: #000000; text-decoration: underline;">Pre-Compiled Wheel Deepspeed v11.2 (Python
3.11 and 3.10)</span></span></span></strong></p>
<ol>
<li>Download the correct wheel version for your Python/Cuda from <a
href="https://github.com/erew123/alltalk_tts/releases/tag/deepspeed" target="_blank">here</a> and save
the file it inside your <span style="color: #3366ff;">text-generation-webui folder</span>.</li>
<li>Open a command prompt window, move into your <span style="color: #3366ff;">text-generation-webui
folder</span> you can now start the Python environment for text-generation-webui <span
style="color: #3366ff;">cmd_windows.bat<br /></span></li>
<li>With the file that you saved in the <span style="color: #3366ff;">text-generation-webui folder</span> you
now type the following, replacing <span style="color: #99cc00;">your-version</span> with the name of the
file you have:<br /><br /><span style="color: #3366ff;">pip install "deepspeed-0.11.2+<span
style="color: #99cc00;">your-version</span>-win_amd64.whl"<br /><br /></span></li>
<li>This should install through cleanly and you should now have DeepSpeed v11.2 installed within the Python
3.11/3.10 environment of text-generation-webui.</li>
<li>When you start up text-generation-webui, and AllTalk starts, you should see:<br /><br /><span
style="background-color: #999999;"><span style="background-color: #ffffff;">&nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; [AllTalk Startup] DeepSpeed</span></span><strong><span
style="background-color: #999999;"><span style="color: #ffff00; background-color: #ffffff;">
Detected<br /></span></span></strong><strong><span style="background-color: #999999;"><span
style="color: #ffff00; background-color: #ffffff;"><br /></span></span></strong></li>
<li>Within AllTalk, you will now have a checkbox for <span style="color: #3366ff;">Activate DeepSpeed</span>
though remember you can only change 1x setting every 15 or so seconds, so don't try to activate DeepSpeed
and
LowVRAM simultaneously. When you are happy it works, you can set the default start-up settings in the
settings page.<br /><br /><span style="color: #ff0000;">Removal</span> - If it became necessary to uninstall
DeepSpeed, you can do so with <span style="color: #3366ff;">cmd_windows.bat</span> and then <span
style="color: #3366ff;">pip uninstall deepspeed<br /></span><span
style="color: #3366ff;"><br /><br /></span><span style="color: #3366ff;"><span
style="text-decoration: underline;"><span
style="color: #ff9900; text-decoration: underline;"><strong>OPTION 2 - <span
style="color: #000000; text-decoration: underline;">Manual build of DeepSpeed&nbsp;v11.2
(Python 3.11 and 3.10)</span></strong></span></span><br /><br /></span>Due to the
complexity of this, and the complicated formatting, instructions can be found on this <a
href="https://github.com/erew123/alltalk_tts?tab=readme-ov-file#-option-2---a-bit-more-complicated"
target="_blank">link</a></li>
</ol>
<p style="padding-left: 30px;"><a href="#toc">Back to top of page</a></p>
<h2 id="TTSmodels">TTS Models/Methods</h2>
<p style="padding-left: 30px; text-align: justify;">It's worth noting that all models and methods can and do sound
different from one another. Many people complained about the quality of audio produced by the 2.0.3 model, so
this extension will download the 2.0.2 model to your models folder and give you the choice to use 2.0.2 (<span
style="color: #3366ff;">API Local</span> and <span style="color: #3366ff;">XTTSv2 Local</span>) or use the
most current model 2.0.3 (<span style="color: #3366ff;">API TTS</span>). As/When a new model is released by
Coqui it will be downloaded by the TTS service on startup and stored wherever the TTS service keeps new models
for your operating system.</p>
<ul>
<li><span style="color: #3366ff;">API TTS:</span> Uses the current TTS model available that's downloaded by the
TTS API process ( version 2.0.3 at the time of writing). This model is not stored in your "models" folder,
but elsewhere on your system and managed by the TTS software.<br /></li>
<li><span style="color: #3366ff;">API Local:</span> Utilizes the 2.0.2 local model stored at <span
style="color: #3366ff;">/alltalk_tts/models/xttsv2_2.0.2<br /></span></li>
<li><span style="color: #3366ff;">XTTSv2 Local: </span>Utilizes the 2.0.2 local model <span
style="color: #3366ff;">/alltalk_tts/models/xttsv2_2.0.2</span> and utilizes a distinct TTS generation
method. <span style="color: #99cc00;">Supports DeepSpeed acceleration</span>.</li>
</ul>
<p style="padding-left: 30px;"><a href="#toc">Back to top of page</a></p>
<h2 id="temperature-and-repetition-settings"><strong>Model Temperature and Repetition Settings</strong></h3>
<p style="padding-left: 30px; text-align: justify;">It is recommended not to modify these settings unless you
fully comprehend their effects. A general overview is provided below for reference.</p>
<p style="padding-left: 30px; text-align: justify;">Changes to these settings won't take effect until you
restart AllTalk/Text generation webUI.</p>
<p style="padding-left: 30px; text-align: justify;">These settings only affect API Local and XTTSv2 Local
methods.</p>
<h4 style="text-align: justify;">Repetition Penalty:</h4>
<p style="padding-left: 30px; text-align: justify;">In the context of text-to-speech (TTS), the Repetition
Penalty<strong>&nbsp;</strong>influences how the model handles the repetition of sounds, phonemes, or
intonation patterns. Here's how it works:</p>
<ul style="text-align: justify;">
<li><strong>Higher Repetition Penalty (e.g. 16.0):</strong> The model is less likely to repeat sounds or
patterns. It promotes diversity in the generated speech. This can result in a more varied and expressive
output, though introduce elements of unpredictability in the TTS output.<br /></li>
<li><strong>Lower Repetition Penalty (e.g. 2.0):</strong> The model is more tolerant of repeating sounds or
patterns. This might lead to more repetition in the generated speech, potentially making it sound more
structured or rhythmically consistent. Lower values can still introduce expressive variations, but to a
lesser extent. This tendency means that the generated speech may remain closer to the original sample.
</li>
</ul>
<p style="padding-left: 30px; text-align: justify;">The factory setting for repetition penalty is 10.0</p>
<h4>Temperature:</h4>
<p style="padding-left: 30px; text-align: justify;">Temperature&nbsp;influences the randomness of the generated
speech. Here's how it affects the output:</p>
<ul style="text-align: justify;">
<li><strong>Higher Temperature (e.g. 0.95):</strong> Increases randomness in how the model selects and
pronounces phonemes or intonation patterns. This can result in more creative, but potentially less
controlled or "stable," speech that may deviate from the input sample. It adds an element of
unpredictability and variety, contributing to expressiveness in the voice output created.<br /></li>
<li><strong>Lower Temperature (e.g. 0.20):</strong> Reduces randomness, making the model more likely to
closely mimic the input sample's voice, intonation, and overall style. This tends to produce more
"coherent" speech that aligns closely with the characteristics of the training data or input voice
sample. It adds a level of predictability and consistency, potentially reducing expressive variations.
So it could end up sounding too monotone.</li>
</ul>
<p style="padding-left: 30px; text-align: justify;">The factory setting for temperature is 0.70</p>
<h4><strong>Temperature and Repetition Settings Examples:</strong></h4>
<ul style="text-align: justify;">
<li><strong>Temp High (0.90) and Repetition High (16.0):</strong><br /> Result: Speech may sound highly
creative and diverse, with reduced repetition. It could be more expressive and unpredictable.<br /></li>
<li><strong>Temp Low (0.20) and Repetition High (16.0):</strong><br /> Result: Output tends to be focused
and deterministic, but with reduced repetition. It may sound structured and less expressive.<br /></li>
<li><strong>Temp High (0.90) and Repetition Low (2.0):</strong><br /> Result: Speech may be more creative
and diverse, with tolerance for repeating sounds. It could have expressive variations but with some
structured patterns.<br /></li>
<li><strong>Temp Low (0.20) and Repetition Low (2.0):</strong><br /> Result: Output is focused and
deterministic, with tolerance for repeating sounds. It may sound more structured and less expressive.
</li>
</ul>
<p style="text-align: justify;">Factory settings should be fine for most people, however if you choose to
experiment, setting extremely high or low values, especially without a good understanding of their effects,
may lead to flat-sounding output or very strange-sounding output. It's advisable to experiment with
adjustments incrementally and observe the impact on the generated speech to find a balance that suits your
desired outcome.</p>
<p style="text-align: justify;"><a href="#toc">Back to top of page</a><br /></p>
<h2 id="start-up-checks">Start-up Checks</h2>
<p style="padding-left: 30px; text-align: justify;">AllTalk performs a variety of checks on startup and will
warn out messages at the console should you need to do something such as update your TTS version.&nbsp;</p>
<p style="padding-left: 30px; text-align: justify;">A basic environment check to ensure everything should work
e.g. is the model already downloaded, are the configuration files set correctly etc.</p>
<p style="padding-left: 30px; text-align: justify;">AllTalk will download the Xtts model (version 2.0.2) into
your models folder. Many people didn't like the quality of the 2.0.3 model, however the latest model will be
accessible on the API TTS setting (2.0.3 at the time of writing) so you have the best of both worlds.</p>
<p style="padding-left: 30px;"><a href="#toc">Back to top of page</a></p>
<h2 id="customTTSmodels">Custom TTS Models and Model path</h2>
<p style="padding-left: 30px; text-align: justify;">Its possible to set a custom model for the <span
style="color: #3366ff;">API Local</span> and <span style="color: #3366ff;">XTTSv2 Local</span> methods,
or indeed point it at the same model that <span style="color: #3366ff;">API TTS</span> uses (wherever it is
stored on your OS of choice).</p>
<p style="padding-left: 30px; text-align: justify;">Many people did not like the sound quality of the Coqui
<span style="color: #000000;">2.0.3</span> model, and as such the AllTalk downloads the <span
style="color: #000000;">2.0.2</span> model separately from the <span
style="color: #000000;">2.0.3</span> model that TTS service downloads and manages.
</p>
<p style="padding-left: 30px; text-align: justify;">Typically the <span style="color: #000000;">2.0.2</span>
model is stored in your <span style="color: #3366ff;">/alltalk_tts/models</span> folder and it is downloaded
on first start-up of the&nbsp;AllTalk_tts extension. However, you may either want to use a custom model
version of your choosing, or re-point AllTalk to a different path on your system, or even point it so
that&nbsp;<span style="color: #3366ff;">API Local</span> and <span style="color: #3366ff;">XTTSv2
Local</span> both use the same model that <span style="color: #3366ff;">API TTS</span> is using.</p>
<p style="padding-left: 30px; text-align: justify;">If you do choose to change the location there are a couple
of things to note.&nbsp;</p>
<ul style="text-align: justify;">
<li>The folder you place the model in, <span style="color: #3366ff;">cannot </span>be called "<span
style="color: #3366ff;">models</span>". This name is reserved solely for the system to identify you
are or are not using a custom model.</li>
<li>On each startup, the&nbsp;AllTalk tts extension will check the custom location and if it does not exist,
it will create it and download the files it needs. It will also re download any missing files in that
location that are needed for the model to function.</li>
<li>There will be extra output at the console to inform you that you are using a custom model and each time
you load up AllTalk extension or switch between models.</li>
</ul>
<p style="padding-left: 30px; text-align: justify;">To change the model path, there are at minimum 2x settings
you need to alter in the <span style="color: #3366ff;">modeldownload.json</span> file, <span
style="color: #3366ff;">base_path </span>and <span style="color: #3366ff;">model_path</span>.</p>
<p style="padding-left: 30px; text-align: justify;">You would edit the settings in the <span
style="color: #3366ff;">modeldownload.json</span> file as follows (make a backup of your current file in
case):<br /><br />&nbsp; &nbsp; &nbsp; &nbsp; Windows path example:&nbsp;<span
style="color: #2980b9;">c:&bsol;&bsol;mystuff&bsol;&bsol;mydownloads&bsol;&bsol;myTTSmodel&bsol;&bsol;</span><span
style="color: #00ccff;"><em>{files in here}<br /></em></span><span style="color: #3366ff;">&nbsp; &nbsp;
&nbsp; &nbsp; base_path</span> would be "<span
style="color: #2980b9;">c:&bsol;&bsol;mystuff&bsol;&bsol;mydownloads"<br /></span><span
style="color: #3366ff;">&nbsp; &nbsp; &nbsp; &nbsp; model_path</span> would be:<span
style="color: #2980b9;">&nbsp;"myTTSmodel"</span></p>
<p style="padding-left: 30px;"><span style="color: #ff0000;">Note: </span>On Windows systems, you have to
specify a <span style="color: #e74c3c;"></span><span style="color: #e74c3c;">double backslash
&bsol;&bsol;</span> for each folder level in the path (as above)</p>
<p style="padding-left: 30px;">&nbsp; &nbsp; &nbsp; &nbsp; Linux path example:&nbsp;<span
style="color: #2980b9;">/home/myaccount/myTTSmodel/</span><span style="color: #00ccff;"><em>{files in
here}<br /></em></span><span style="color: #3366ff;">&nbsp; &nbsp; &nbsp; &nbsp; base_path</span>
would be "<span style="color: #2980b9;">/home/myaccount"<br /></span><span style="color: #3366ff;">&nbsp;
&nbsp; &nbsp; &nbsp; model_path</span> would be:<span
style="color: #2980b9;">&nbsp;"TTSmodel"<br /><br /></span>Once you restart Alltalk, it will check this
path for the files and output any details at the console.</p>
<p style="padding-left: 30px;">When you are happy it's' working correctly, you are welcome to go delete the
models folder stored at&nbsp;<span style="color: #3366ff;">/alltalk_tts/models</span><strong>.</strong></p>
<p style="padding-left: 30px;">If you wish to change the files that the modeldownloader is pulling at startup,
you can further edit the <span style="color: #3366ff;">modeldownload.json</span> and change the https
addresses within this files&nbsp;<span style="color: #3366ff;">files_to_download</span> section &nbsp;e.g.
</p>
<p style="padding-left: 30px;">"files_to_download": {<br /> &nbsp; &nbsp; &nbsp; &nbsp; "LICENSE.txt": "<span
style="color: #00ccff;">https://huggingface.co/coqui/XTTS-v2/resolve/v2.0.2/LICENSE.txt?download=true</span>",<br />
&nbsp; &nbsp; &nbsp; &nbsp; "README.md": "<span
style="color: #00ccff;">https://huggingface.co/coqui/XTTS-v2/resolve/v2.0.2/README.md?download=true</span>",<br />
&nbsp; &nbsp; &nbsp; &nbsp; "config.json": "<span
style="color: #00ccff;">https://huggingface.co/coqui/XTTS-v2/resolve/v2.0.2/config.json?download=true</span>",<br />
&nbsp; &nbsp; &nbsp; &nbsp; "model.pth": "<span
style="color: #00ccff;">https://huggingface.co/coqui/XTTS-v2/resolve/v2.0.2/model.pth?download=true</span>",<br />
&nbsp; &nbsp; &nbsp; &nbsp; "vocab.json": "<span
style="color: #00ccff;">https://huggingface.co/coqui/XTTS-v2/resolve/v2.0.2/vocab.json?download=true</span>"<br />&nbsp;}
</p>
<p style="padding-left: 30px;"><a href="#toc">Back to top of page<br /></a></p>
<h2 id="configuration-details"><strong>Configuration file settings</strong></h2>
<p><span style="color: #3366ff;">confignew.json</span> file:</p>
<table>
<tr>
<th>Key</th>
<th>Default Value</th>
<th>Explanation</th>
</tr>
<tr>
<td>"activate"</td>
<td>true</td>
<td>Sets activation state within Text-generation-webUI.</td>
</tr>
<tr>
<td>"autoplay"</td>
<td>true</td>
<td>Sets autoplay within Text generation webUI.</td>
</tr>
<tr>
<td>"branding"</td>
<td>"AllTalk"</td>
<td>Used to change the default name. Requires a space e.g. "Mybrand ".</td>
</tr>
<tr>
<td>"deepspeed_activate"</td>
<td>false</td>
<td>Sets DeepSpeed activation on startup.</td>
</tr>
<tr>
<td>"delete_output_wavs"</td>
<td>"Disabled"</td>
<td>Sets duration of outputs to delete.</td>
</tr>
<tr>
<td>"ip_address"</td>
<td>"127.0.0.1"</td>
<td>Sets default IP address.</td>
</tr>
<tr>
<td>"language"</td>
<td>"English"</td>
<td>Sets default language for Text-generation-webUI TTS.</td>
</tr>
<tr>
<td>"low_vram"</td>
<td>false</td>
<td>Sets default setting for LowVRAM mode.</td>
</tr>
<tr>
<td>"local_temperature"</td>
<td>"0.70"</td>
<td>Sets default model temp for API Local and XTTSv2 Local.</td>
</tr>
<tr>
<td>"local_repetition_penalty"</td>
<td>"10.0"</td>
<td>Sets default model repetition for API Local and XTTSv2 Local.</td>
</tr>
<tr>
<td>"tts_model_loaded"</td>
<td>true</td>
<td>AllTalk internal use only. Do not change.</td>
</tr>
<tr>
<td>"tts_model_name"</td>
<td>"tts_models/multilingual/multi-dataset/xtts_v2"</td>
<td>Sets default model that API TTS is looking for through the TTS service (separate to API Local and
XTTSv2 Local).</td>
</tr>
<tr>
<td>"narrator_enabled"</td>
<td>true</td>
<td>Sets default narrator on/off in Text-generation-webUI TTS.</td>
</tr>
<tr>
<td>"narrator_voice"</td>
<td>"female_02.wav"</td>
<td>Sets default wav to use for narrator in Text-generation-webUI TTS.</td>
</tr>
<tr>
<td>"port_number"</td>
<td>"7851"</td>
<td>Sets default port number for AllTalk.</td>
</tr>
<tr>
<td>"output_folder_wav"</td>
<td>"extensions/alltalk_tts/outputs/"</td>
<td>Sets default output path Text-generation-webUI should use for finding outputs.</td>
</tr>
<tr>
<td>"output_folder_wav_standalone"</td>
<td>"outputs/"</td>
<td>Sets default output path in standalone mode.</td>
</tr>
<tr>
<td>"remove_trailing_dots"</td>
<td>false</td>
<td>Sets trailing dot removal pre-generating TTS.</td>
</tr>
<tr>
<td>"show_text"</td>
<td>true</td>
<td>Sets if text should be displayed below audio in Text-generation-webUI.</td>
</tr>
<tr>
<td>"tts_method_api_local"</td>
<td>false</td>
<td>Sets API Local as the default model/method for TTS.</td>
</tr>
<tr>
<td>"tts_method_api_tts"</td>
<td>false</td>
<td>Sets API TTS as the default model/method for TTS.</td>
</tr>
<tr>
<td>"tts_method_xtts_local"</td>
<td>true</td>
<td>Sets XTTSv2 Local as the default model/method for TTS.</td>
</tr>
<tr>
<td>"voice"</td>
<td>"female_01.wav"</td>
<td>Sets default voice for TTS.</td>
</tr>
</table>
<p><span style="color: #3366ff;">modeldownload.json</span> file:</p>
<table id="modeldownload-table">
<tr>
<th>Key</th>
<th>Value</th>
<th>Description</th>
</tr>
<tr>
<td><span class="key">"base_path"</span></td>
<td>"models"</td>
<td>Sets local model base path for API Local and XTTSv2 Local.</td>
</tr>
<tr>
<td><span class="key">"model_path"</span></td>
<td>"xttsv2_2.0.2"</td>
<td>Sets local model folder for API Local and XTTSv2 Local below the base path.</td>
</tr>
<tr>
<td><span class="key">"files_to_download"</span></td>
<td>
<table>
<tr>
<th>File</th>
<th>Download URL</th>
</tr>
<tr>
<td>"LICENSE.txt"</td>
<td><span
style="color: #333399;">https://huggingface.co/coqui/XTTS-v2/resolve/v2.0.2/LICENSE.txt?download=true</span>
</td>
</tr>
<tr>
<td>"README.md"</td>
<td><span
style="color: #333399;">https://huggingface.co/coqui/XTTS-v2/resolve/v2.0.2/README.md?download=true</span>
</td>
</tr>
<tr>
<td>"config.json"</td>
<td><span
style="color: #333399;">https://huggingface.co/coqui/XTTS-v2/resolve/v2.0.2/config.json?download=true</span>
</td>
</tr>
<tr>
<td>"model.pth"</td>
<td><span
style="color: #333399;">https://huggingface.co/coqui/XTTS-v2/resolve/v2.0.2/model.pth?download=true</span>
</td>
</tr>
<tr>
<td>"vocab.json"</td>
<td><span
style="color: #333399;">https://huggingface.co/coqui/XTTS-v2/resolve/v2.0.2/vocab.json?download=true</span>
</td>
</tr>
</table>
</td>
<td>Sets the model files required to be downloaded into &bsol;base_path&bsol;model_path&bsol; and where
to download them from.</td>
</tr>
</table>
<h2 id="curl-commands"><strong>JSON calls &amp; CURL Commands</strong></h2>
<h3>Overview</h3>
<p style="margin-left: 40px; text-align: justify;">The Text-to-Speech (TTS) Generation API allows you to
generate speech from text input using various configuration options. This API supports both character and
narrator voices, providing flexibility for creating dynamic and engaging audio content.</p>
<h3>TTS Generation Endpoint</h3>
<ul>
<li><strong>URL</strong>: <span style="color: #3366ff;">http://127.0.0.1:7851/api/tts-generate</span></li>
<li><strong>Method</strong>: <span style="color: #3366ff;">POST</span></li>
<li><strong>Content-Type</strong>: <span style="color: #3366ff;">application/x-www-form-urlencoded</span>
</li>
</ul>
<h3>Example command line</h3>
<p style="padding-left: 30px;">Standard TTS speech Example (standard text) generating a time-stamped file</p>
<p style="padding-left: 30px;"><span style="color: #3366ff;">curl -X POST
"http://127.0.0.1:7851/api/tts-generate" -d "text_input=</span><span style="color: #ff9900;">All of this
is text spoken by the character. This is text not inside quotes, though that doesnt matter in the
slightest</span><span style="color: #3366ff;">" -d "text_filtering=</span><span
style="color: #339966;">standard</span><span style="color: #3366ff;">" -d
"character_voice_gen=</span><span style="color: #ff9900;">female_01.wav</span><span
style="color: #3366ff;">" -d "narrator_enabled=</span><span style="color: #339966;">false</span><span
style="color: #3366ff;">" -d "narrator_voice_gen=male_01.wav" -d "text_not_inside=character" -d
"language=en" -d "output_file_name=</span><span style="color: #ff9900;">myoutputfile</span><span
style="color: #3366ff;">" -d "output_file_timestamp=</span><span
style="color: #ff9900;">true</span><span style="color: #3366ff;">" -d "autoplay=true" -d
"autoplay_volume=0.8"</span></p>
<p style="padding-left: 30px;">Narrator Example (standard text)&nbsp;generating a time-stamped file</p>
<p style="padding-left: 30px;"><span style="color: #3366ff;">curl -X POST
"http://127.0.0.1:7851/api/tts-generate" -d "text_input=</span><span style="color: #ff9900;">*This is
text spoken by the narrator* &bsol;"This is text spoken by the character&bsol;". This is text not inside
quotes.</span><span style="color: #3366ff;">" -d "text_filtering=</span><span
style="color: #339966;">standard</span><span style="color: #3366ff;">" -d
"character_voice_gen=</span><span style="color: #ff9900;">female_01.wav</span><span
style="color: #3366ff;">" -d "narrator_enabled=</span><span style="color: #339966;">true</span><span
style="color: #3366ff;">" -d "narrator_voice_gen=</span><span
style="color: #339966;">male_01.wav</span><span style="color: #3366ff;">" -d
"text_not_inside=</span><span style="color: #339966;">character</span><span style="color: #3366ff;">" -d
"language=en" -d "output_file_name=</span><span style="color: #ff9900;">myoutputfile</span><span
style="color: #3366ff;">" -d "output_file_timestamp=</span><span
style="color: #ff9900;">true</span><span style="color: #3366ff;">" -d "autoplay=true" -d
"autoplay_volume=0.8"</span></p>
<p style="padding-left: 30px; text-align: justify;"><strong><span style="color: #ff0000;">Note</span></strong>
that if your text that needs to be generated contains double quotes you will need to escape them
with&nbsp;&bsol;" (Please see the narrator example).</p>
<h3>&nbsp;Request Parameters</h3>
<ul>
<li>
<p style="text-align: justify;"><strong>text_input</strong>: The text you want the TTS engine to
produce. Use escaped double quotes for character speech and asterisks for narrator speech if using
the narrator function. Example:</p>
<p>-<span style="color: #3366ff;">d "text_input=*This is text spoken by the narrator* &bsol;"This is
text spoken by the character&bsol;". This is text not inside quotes." </span></p>
</li>
<li>
<p><strong>text_filtering</strong>: Filter for text. Options:</p>
<ul>
<li style="text-align: justify;"><span style="color: #3366ff;">none</span>&nbsp;No filtering.
Whatever is sent will go over to the TTS engine as raw text, which may result in some odd sounds
with some special characters.</li>
<li><span style="color: #3366ff;">standard</span>&nbsp;Human-readable text and a basic level of
filtering, just to clean up some special characters.</li>
<li><span style="color: #3366ff;">html</span>&nbsp;HTML content. Where you are using HTML entity's
like &amp;quot;</li>
</ul>
<p><span style="color: #3366ff;">-d "text_filtering=none" </span><br /><span style="color: #3366ff;">-d
"text_filtering=standard" </span><br /><span style="color: #3366ff;">-d
"text_filtering=html"</span></p>
<p><strong>Example:</strong></p>
<ul>
<li>Standard Example: <span style="color: #3366ff;">*This is text spoken by the narrator* "This is
text spoken by the character" This is text not inside quotes.</span></li>
<li>HTML Example:<span style="color: #3366ff;"> *This is text spoken by the
narrator*&nbsp;&amp;quot;This is text spoken by the character&amp;quot;&nbsp;This is text
not inside quotes.</span></li>
<li>None will just pass whatever characters/text you send at it.</li>
</ul>
</li>
<li>
<p><strong>character_voice_gen</strong>: The WAV file name for the character's voice.</p>
<p><span style="color: #3366ff;">-d "character_voice_gen=female_01.wav" </span></p>
</li>
<li>
<p><strong>narrator_enabled</strong>: Enable or disable the narrator function. If true, minimum text
filtering is set to standard. Anything between double quotes is considered the character's speech,
and anything between asterisks is considered the narrator's speech.</p>
<p><span style="color: #3366ff;">-d "narrator_enabled=true"<br />-d "narrator_enabled=false"
</span><span style="font-family: Verdana, Arial, Helvetica, sans-serif;">&nbsp;</span></p>
</li>
<li>
<p><strong>narrator_voice_gen</strong>: The WAV file name for the narrator's voice.</p>
<p><span style="color: #3366ff;">-d "narrator_voice_gen=male_01.wav"</span></p>
</li>
<li>
<p><strong>text_not_inside</strong>: Specify the handling of lines not inside double quotes or
asterisks, for the narrator feature. Options:</p>
<ul>
<li><span style="color: #3366ff;">character</span>: Treat as character speech.</li>
<li><span style="color: #3366ff;">narrator</span>: Treat as narrator speech.</li>
</ul>
<p><span style="color: #3366ff;">-d "text_not_inside=character" </span><br /><span
style="color: #3366ff;">-d "text_not_inside=narrator"</span></p>
</li>
<li>
<p><strong>language</strong>: Choose the language for TTS. Options:</p>
<ul>
<li><span style="color: #3366ff;">ar</span>&nbsp;Arabic</li>
<li><span style="color: #3366ff;">zh-cn</span>&nbsp;Chinese (Simplified)</li>
<li><span style="color: #3366ff;">cs</span>&nbsp;Czech</li>
<li><span style="color: #3366ff;">nl</span> Dutch</li>
<li><span style="color: #3366ff;">en</span> English</li>
<li><span style="color: #3366ff;">fr</span> French</li>
<li><span style="color: #3366ff;">de</span> German</li>
<li><span style="color: #3366ff;">hu</span> Hungarian</li>
<li><span style="color: #3366ff;">it</span> Italian</li>
<li><span style="color: #3366ff;">ja</span> Japanese</li>
<li><span style="color: #3366ff;">ko</span> Korean</li>
<li><span style="color: #3366ff;">pl</span> Polish</li>
<li><span style="color: #3366ff;">pt</span> Portuguese</li>
<li><span style="color: #3366ff;">ru </span>Russian</li>
<li><span style="color: #3366ff;">es</span> Spanish</li>
<li><span style="color: #3366ff;">tr</span> Turkish</li>
</ul>
<p><span style="color: #3366ff;">-d "language=en"</span></p>
</li>
<li>
<p><strong>output_file_name</strong>: The name of the output file (excluding the .wav extension).</p>
<p><span style="color: #3366ff;">-d "output_file_name=myoutputfile" </span></p>
</li>
<li>
<p><strong>output_file_timestamp</strong>: Add a timestamp to the output file name. If true, each file
will have a unique timestamp; otherwise, the same file name will be overwritten each time you
generate TTS.</p>
<p><span style="color: #3366ff;">-d "output_file_timestamp=true" <br />-d "output_file_timestamp=false"
</span></p>
</li>
<li>
<p><strong>autoplay</strong>: Enable or
disable autoplay. Still needs to be specified in the JSON request.</p>
<p><span style="color: #3366ff;">-d "autoplay=true" <br />-d "autoplay=false"</span></p>
</li>
<li>
<p><strong>autoplay_volume</strong>: Set
the autoplay volume. Should be between 0.1 and 1.0.&nbsp;Still needs to be specified in the JSON
request.</p>
<p><span style="color: #3366ff;">-d "autoplay_volume=0.8" </span></p>
</li>
</ul>
<h3>TTS Generation Response</h3>
<p style="padding-left: 30px;">The API returns a JSON object with the following properties:</p>
<ul>
<li><span style="color: #3366ff;"><strong>status</strong></span>&nbsp;Indicates whether the generation was
successful (<span style="color: #3366ff;">generate-success</span>) or failed (<span
style="color: #3366ff;">generate-failure</span>).</li>
<li><span style="color: #3366ff;"><strong>output_file_path</strong></span>&nbsp;The on-disk location of the
generated WAV file.</li>
<li><span style="color: #3366ff;"><strong>output_file_url</strong></span>&nbsp;The HTTP location for
accessing the generated WAV file.</li>
</ul>
<p><strong>Example JSON TTS Generation Response:</strong></p>
<p style="padding-left: 30px;"><span style="color: #339966;">{"status": "generate-success", "output_file_path":
"C:\\text-generation-webui\\extensions\\alltalk_tts\\outputs\\myoutputfile_1703149973.wav",
"output_file_url": "http://127.0.0.1:7851/audio/myoutputfile_1703149973.wav"}</span></p>
<h4>Switching Model</h4>
<p style="padding-left: 30px;"><span style="color: #3366ff;">curl -X POST
"http://127.0.0.1:7851/api/reload?tts_method=API%20Local"</span><br /><span style="color: #3366ff;">
curl -X POST "http://127.0.0.1:7851/api/reload?tts_method=API%20TTS"</span><br /><span
style="color: #3366ff;"> curl -X POST
"http://127.0.0.1:7851/api/reload?tts_method=XTTSv2%20Local"</span></p>
<p style="padding-left: 30px;">Switch between the 3 models respectively.</p>
<p style="padding-left: 30px;">JSON return <span style="color: #339966;">{"status": "model-success"}</span></p>
<h4>Switch DeepSpeed</h4>
<p style="padding-left: 30px;"><span style="color: #3366ff;">curl -X POST
"http://127.0.0.1:7851/api/deepspeed?new_deepspeed_value=True"</span></p>
<p style="padding-left: 30px;">Replace True with False to disable DeepSpeed mode.</p>
<p style="padding-left: 30px;">JSON return <span style="color: #339966;">{"status": "deepspeed-success"}</span>
</p>
<h4>Switching Low VRAM</h4>
<p style="padding-left: 30px;"><span style="color: #3366ff;">curl -X POST
"http://127.0.0.1:7851/api/lowvramsetting?new_low_vram_value=True"</span></p>
<p style="padding-left: 30px;">Replace True with False to disable Low VRAM mode.</p>
<p style="padding-left: 30px;">JSON return <span style="color: #339966;">{"status": "lowvram-success"}</span>
</p>
<h4>Ready Endpoint</strong></h4>
<p>Check if the Text-to-Speech (TTS) service is ready to accept requests.</p>
<ul>
<li>URL: <span style="color: #3366ff;">http://127.0.0.1:7851/api/ready</span></li>
<li>Method: <span style="color: #3366ff;">GET</span></li>
<li>Response: <span style="color: #339966;">Ready</span></li>
</ul>
<p style="padding-left: 30px;"><span style="color: #3366ff;">curl -X GET
"http://127.0.0.1:7851/api/ready"</span></p>
<h4>Voices List Endpoint</strong></h4>
<p>Retrieve a list of available voices for generating speech.</p>
<ul>
<li>URL: <span style="color: #3366ff;">http://127.0.0.1:7851/api/voices</span></li>
<li>Method: <span style="color: #3366ff;">GET</span></li>
</ul>
<p style="padding-left: 30px;"><span style="color: #3366ff;">curl -X GET
"http://127.0.0.1:7851/api/voices"</span></p>
<p style="padding-left: 30px;">JSON return: <span style="color: #339966;">{"voices": ["voice1.wav",
"voice2.wav", "voice3.wav"]}</span></p>
<h4><strong>Preview Voice Endpoint</strong></h4>
<p>Generate a preview of a specified voice with hardcoded settings.</p>
<ul>
<li>URL: <span style="color: #3366ff;">http://127.0.0.1:7851/api/previewvoice/</span></li>
<li>Method: <span style="color: #3366ff;">POST</span></li>
<li>Content-Type: <span style="color: #3366ff;">application/x-www-form-urlencoded</span></li>
</ul>
<p style="padding-left: 30px;"><span style="color: #3366ff;">curl -X POST
"http://127.0.0.1:7851/api/previewvoice/" -F "voice=female_01.wav"</span></p>
<p style="padding-left: 30px;">Replace <span style="color: #3366ff;">female_01.wav</span> with the name of the
voice sample you want to hear.</p>
<p style="padding-left: 30px;">JSON return: <span style="color: #339966;">{"status": "generate-success",
"output_file_path": "/path/to/outputs/api_preview_voice.wav", "output_file_url":
"http://127.0.0.1:7851/audio/api_preview_voice.wav"}</span></p>
<p><a href="#toc">Back to top of page<br /></a></p>
<h2 id="debugging-and-tts-generation-information"><strong>Debugging and TTS Generation Information</strong></h2>
<p style="padding-left: 30px; text-align: justify;">Command line outputs are more verbose to assist in
understanding backend processes and debugging.</p>
<p style="padding-left: 30px; text-align: justify;">Its possible during startup you can get a warning message
such as: <br /><br /><span
style="color: black;">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[AllTalk
Startup]</span> <span style="color: #ff0000;">Warning</span> <span style="color: black;">TTS Subprocess
has NOT started up yet, Will keep trying for 80 seconds maximum</span> <br /><br />This is normal
behaviour if the subprocess is taking a while to start, however, if there is an issue starting the
subprocess, you may see multiples of this message and an it will time out after 80 seconds, resulting in the
TTS engine not starting. Its likely that you are not in the correct python environment or one that has a TTS
engine inside, if this happens, though the system will output a warning about that ahead of this message</p>
<p style="padding-left: 30px; text-align: justify;">Typically, the command line console will output any warning
or error messages. If you need to reset your default configuration, the settings are all listed above in the
configuration details.</p>
<p style="padding-left: 30px;"><a href="#toc">Back to top of page</a></p>
<h2 id="references"><strong>Thanks &amp; References</strong></h2>
<h4>Coqui TTS Engine</h4>
<ul>
<li><a href="https://coqui.ai/cpml.txt" target="_blank" rel="noopener">Coqui License</a></li>
<li><a href="https://github.com/coqui-ai/TTS" target="_blank" rel="noopener">Coqui TTS GitHub Repository</a>
</li>
</ul>
<h4>Extension coded by</h4>
<ul>
<li><a href="https://github.com/erew123" target="_blank" rel="noopener">Erew123 GitHub Profile</a></li>
</ul>
<h4>Thanks to &amp; Text generation webUI</h4>
<ul>
<li><a href="https://github.com/oobabooga/text-generation-webui" target="_blank" rel="noopener">Ooobabooga
GitHub Repository</a> (Portions of orginal Coquii_TTS extension)</li>
</ul>
<h4>Thanks to</h4>
<ul>
<li><a href="https://github.com/daswer123" target="_blank" rel="noopener">daswer123 GitHub Profile</a>
(Assistance with cuda to cpu moving)</li>
<li><a href="https://github.com/S95Sedan" target="_blank" rel="noopener">S95Sedan GitHub Profile</a>
(Editing the Microsoft DeepSpeed v11.x installation files so they work)</li>
<li><a href="https://github.com/kanttouchthis" target="_blank" rel="noopener">kanttouchthis GitHub
Profile</a> (Portions of orginal Coquii_TTS extension)</li>
<li><a href="https://github.com/Wuzzooy" target="_blank" rel="noopener">Wuzzooy GitHub Profile</a> (Trying
out the code while in development)</li>
</ul>
<p><a href="#toc">Back to top of page</a></p>
</body>
</html>