Documentation update

Updated a note in the documentation about correct formatting of RP character greeting cards

README.md

@@ -1,8 +1,8 @@
 # AllTalk TTS
 AllTalk is an updated version of the Coqui_tts extension for Text Generation web UI. Features include:
 
-- **Custom Startup Settings:** Adjust your standard startup settings. [Screenshot](https://github.com/erew123/alltalk_tts#screenshots)
+- **Custom Start-up Settings:** Adjust your default start-up settings. [Screenshot](https://github.com/erew123/alltalk_tts#screenshots)
-- **Nararator:** Use different voices for main character and narration. [Screenshot](https://github.com/erew123/alltalk_tts#screenshots)
+- **Narrarator:** Use different voices for main character and narration. [Example Narration](https://vocaroo.com/18fYWVxiQpk1)
 - **Low VRAM mode:** Improve generation performance if your VRAM is filled by your LLM. [Screenshot](https://github.com/erew123/alltalk_tts#screenshots)
 - **DeepSpeed:** When DeepSpeed is installed you can get a 3-4x performance boost generating TTS.
 - **Local/Custom models:** Use any of the XTTSv2 models (API Local and XTTSv2 Local).
@@ -11,6 +11,18 @@ AllTalk is an updated version of the Coqui_tts extension for Text Generation web
 - **Console output** Clear command line output for any warnings or issues.
 - **Standalone/3rd Party support via JSON calls** Can be used with 3rd party applications via JSON calls.
 
+#### Updates
+The latest build (13 Dec 2023) has had the entire text filtering engine and narration engine rebuilt from scratch. It's highly complicated how its actually working, but the end result it a much clearer TTS output and much better control over the narrator option and how to handle text that isnt within quotes or asterisks. Its a highly recommened update, for the improved quality it gives to the TTS output, if nothing else.
+
+Should you want the older version of the narrator engine+text filtering, I will leave this older copy [here](https://github.com/erew123/alltalk_tts/releases/tag/v1-old-narrator)
+
+#### The one thing I cant easily work around
+With a RP chat with your AI, on your character card `parameters menu` > `character tab` > `greeting` make sure that anything in there that is the narrator is in asterisks and anything spoken is in double quotes, then hit the `save` (disk) button. Greeting paragraphs/sentences are handled differently from how the AI sends text and so its difficut to account for them both.
+
+I could force a delimeter in at this stage, but I know it would/may affect things further down the line in the chat and I need a good think about that before just making a change. This issue **only** affects the greeting card/start of conversation and the "example" card that comes with text-generation-webui suffers this issue (if you want to try it for yourself). So you would put double quotes around like this (from the example card):
+
+"Hey! I'm so excited to finally meet you. I've heard so many great things about you and I'm eager to pick your brain about computers. I'm sure you have a wealth of knowledge that I can learn from."
+
 ## Installation on Text generation web UI
 This has been tested on the current Dec 2023 release of Text generation webUI. If you have not updated it for a while, you may wish to update Text generation webUI, instructions [here](https://github.com/oobabooga/text-generation-webui#getting-updates)
 
@@ -56,5 +68,5 @@ Once the extension is loaded, please find all documentation and settings on the
 To start AllTalk every Text generation webUI loads, edit the Text generation webUI `CMD_FLAGS.txt` file in the main `text-generation-webui` folder and add `--extensions alltalk_tts`.
 
 ## Screenshots
-|![image](https://github.com/erew123/alltalk_tts/assets/35898566/dfb972dd-817b-45f8-8878-3d3f82ac1ecc) | ![image](https://github.com/erew123/alltalk_tts/assets/35898566/3497d656-9729-4cb7-8d0d-6367078794ee) |
+|![image](https://github.com/erew123/alltalk_tts/assets/35898566/a4d983ab-f9e1-42dd-94ee-a85043f74ab2) | ![image](https://github.com/erew123/alltalk_tts/assets/35898566/3497d656-9729-4cb7-8d0d-6367078794ee) |
 |:---:|:---:|

script.py

@@ -48,7 +48,6 @@ except ModuleNotFoundError:
 # IMPORT - Attempt Importing DeepSpeed (required for displaying Deepspeed checkbox in gradio)
 try:
     import deepspeed
-
     deepspeed_installed = True
 except ImportError:
     deepspeed_installed = False
@@ -155,8 +154,8 @@ elif params["tts_method_api_local"] == True:
 elif params["tts_method_xtts_local"] == True:
     gr_modelchoice = "XTTSv2 Local"
 
-gr_narrator_enabled = str(params["narrator_enabled"]).lower()
+# Set the default for Narrated text without asterisk or quotes to be Narrator
-
+non_quoted_text_is = True
 
 ######################
 #### GRADIO STUFF ####
@@ -289,6 +288,21 @@ def send_reload_request(tts_method):
         if json_response.get("status") == "model-success":
             # Update model_loaded to True if the reload was successful
             params["model_loaded"] = True
+            # Update local script parameters based on the tts_method
+            if tts_method == "API TTS":
+                params["tts_method_api_local"] = False
+                params["tts_method_xtts_local"] = False
+                params["tts_method_api_tts"] = True
+                params["deepspeed_activate"] = False
+            elif tts_method == "API Local":
+                params["tts_method_api_tts"] = False
+                params["tts_method_xtts_local"] = False
+                params["tts_method_api_local"] = True
+                params["deepspeed_activate"] = False
+            elif tts_method == "XTTSv2 Local":
+                params["tts_method_api_tts"] = False
+                params["tts_method_api_local"] = False
+                params["tts_method_xtts_local"] = True
         return json_response
     except requests.exceptions.RequestException as e:
         # Handle the HTTP request error
@@ -402,11 +416,11 @@ def combine(audio_files, output_folder, state):
             audio = np.concatenate((audio, audio_data))
 
     # Save the combined audio to a file with a specified sample rate
-    output_file_path = os.path.join(
+    if "character_menu" in state:
-        output_folder, f'{state["character_menu"]}_{int(time.time())}_combined.wav'
+        output_file_path = os.path.join(output_folder, f'{state["character_menu"]}_{int(time.time())}_combined.wav')
-    )
+    else:
+        output_file_path = os.path.join(output_folder, f'TTSOUT_{int(time.time())}_combined.wav')
     sf.write(output_file_path, audio, samplerate=sample_rate)
-    print(f"[{params['branding']}TTSGen] Narrated audio generated")
     # Clean up unnecessary files
     for audio_file in audio_files:
         os.remove(audio_file)
@@ -463,7 +477,6 @@ def voice_preview(string):
 #################################
 # STANDARD VOICE - Generate TTS Function
 def output_modifier(string, state):
-    # DEBUG print("THE ORIGINAL STRING IS:", string,"\n")
     if not params["activate"]:
         return string
     original_string = string
@@ -477,65 +490,90 @@ def output_modifier(string, state):
     if process_lock.acquire(blocking=False):
         try:
             if params["narrator_enabled"]:
-                processed_string = original_string
+                # Do Some basic stripping and remove any double CR's
-                processed_string = processed_string.replace("***", "*")
+                processed_string = (
-                processed_string = processed_string.replace("**", "*")
+                    original_string
-                processed_string = processed_string.replace(".*", "*")
+                    .replace("***", "*")
-                processed_string = processed_string.replace(".'", "'")
+                    .replace("**", "*")
-                processed_string = processed_string.replace(".'", "'")
+                    .replace("\n\n", "\n")
-                # Remove new lines
+                )
-                processed_string = processed_string.replace("\n", " ")
+                # Clean up two asterisks (narrators) being made between a newline, making it one sentence.
-                #DEBUG print("PROCESSED STRING IS NOW:", processed_string)
+                processed_string = re.sub(r'\.\*\n\*', '. ', processed_string)
-                # Split the processed string into lines
+                # Clean up a few other bits.
-                lines = processed_string.split("\n")
+                processed_string = (
+                    processed_string
+                    .replace("'", "'")
+                    .replace("\n", " ")
+                    # Add special characters to the quote so that we can use it to identify things later after its been split
+                    .replace('&quot;', '&quot;<')
+                    # Capture new conversations which wont have things like &quote in them
+                    .replace('"', '&quot;<')
+                )
+                #capturing another outlier in inital character paragraph
+                processed_string = processed_string.replace('&quot;<. *', '&quot;< *"')
+                processed_string = processed_string.replace('< *"', '< *')
+                # Set up a tracking of the individual wav files.
                 audio_files_all_paragraphs = []
-                is_narrator = False
+                # Split the line using &quot; and ".* " (so end of sentences, leaving special characters added to the start of all OTHER sentences, bar possibly the first one if its starting with a *
-                for line in lines:
+                parts = re.split(r'&quot;|\.\*', processed_string)
-                    # Decode HTML entities
+                audio_files_paragraph = []
-                    decoded_text = html.unescape(line)
+                for i, part in enumerate(parts):
-                    # Initialize variables to track narrator and voice parts
+                    # Skip parts that are too short
-                    is_narrator = "*" in line or is_narrator
+                    if len(part.strip()) <= 1:
-                    # Split the line using double quotes
+                        continue
-                    parts = re.split(r'"', decoded_text)
+                    # Figure out which type of line it is, then replace characters as necessary to avoid TTS trying to pronunce them, htmlunescape after. 
-                    audio_files_paragraph = []
+                    # Character will always be a < with a letter immediately after it
-                    for i, part in enumerate(parts):
+                    if '<' in part and '< *' not in part and '<*' not in part and '<  *' not in part and '< ' not in part and '<  ' not in part:
-                        # Skip parts that are too short
+                        cleaned_part = html.unescape(part.replace('<', ''))
-                        if len(part.strip()) <= 1:
+                        voice_to_use = params["voice"]
-                            continue
+                    #Narrator will always be am * or < with an * a position or two after it.
-                        # Determine if it's a narrator or voice part within double quotes
+                    elif '<*' in part or '< *' in part or '<  *' in part or '*' in part:
-                        is_narrator_part = is_narrator if i % 2 == 0 else False
+                        cleaned_part = html.unescape(part.replace('<*', '').replace('< *', '').replace('<  *', '').replace('*', ''))
-                        voice_to_use = (
+                        voice_to_use = params["narrator_voice"]
-                            params["narrator_voice"] if is_narrator_part else params["voice"]
+                    #If the other two dont capture it, aka, the AI gave no * or &quot; on the line, use non_quoted_text_is aka user interface, user can choose Char or Narrator
-                        )
+                    elif non_quoted_text_is:
-                        # DEBUG print(f"THE STRING TO BE USED: {voice_to_use} {part.strip()}")
+                        cleaned_part = html.unescape(part.replace('< ', '').replace('<  ', '').replace('<  ', ''))
-                        # Process the part
+                        voice_to_use = params["voice"]
-                        output_file = Path(
+                    else:
-                            f'{params["output_folder_wav"]}/{state["character_menu"]}_{int(time.time())}_{i}.wav'
+                        cleaned_part = html.unescape(part.replace('< ', '').replace('<  ', '').replace('<  ', ''))
-                        )
+                        voice_to_use = params["narrator_voice"]
-                        output_file_str = output_file.as_posix()
+                    # Check if character name exists and if not, just call it TTSOUT_
-                        generate_response = send_generate_request(
+                    if "character_menu" in state:
-                            part, voice_to_use, language_code, output_file_str
+                        output_file = Path(f'{params["output_folder_wav"]}/{state["character_menu"]}_{int(time.time())}_{i}.wav')
-                        )
+                    else:
-                        audio_path = generate_response.get("data", {}).get("audio_path")
+                        output_file = Path(f'{params["output_folder_wav"]}/TTSOUT_{int(time.time())}_{i}.wav')
-                        audio_files_paragraph.append(audio_path)
+                    # Generate that TTS and output to a file
-                    is_narrator_part = is_narrator if i % 2 == 0 else False
+                    output_file_str = output_file.as_posix()
-                    voice_to_use = (
+                    generate_response = send_generate_request(
-                        params["narrator_voice"] if is_narrator_part else params["voice"]
+                        cleaned_part, voice_to_use, language_code, output_file_str
                     )
-                    # Accumulate audio files within the paragraph
+                    audio_path = generate_response.get("data", {}).get("audio_path")
-                    audio_files_all_paragraphs.extend(audio_files_paragraph)
+                    audio_files_paragraph.append(audio_path)
+                # Accumulate audio files within the paragraph
+                audio_files_all_paragraphs.extend(audio_files_paragraph)
                 # Combine audio files across paragraphs
                 final_output_file = combine(
                     audio_files_all_paragraphs, params["output_folder_wav"], state
                 )
             else:
-                output_file = Path(
+                processed_string = (
-                    f'{params["output_folder_wav"]}/{state["character_menu"]}_{int(time.time())}.wav'
+                    original_string
+                    .replace("***", "")
+                    .replace("**", "")
+                    .replace("*", "")
+                    .replace("\n\n", "\n")
+                    .replace("&#x27;", "'")
                 )
+                cleaned_part = html.unescape(processed_string)                
+                # Process the part and give it a non-character name if being used vai API or standalone.
+                if "character_menu" in state:
+                    output_file = Path(f'{params["output_folder_wav"]}/{state["character_menu"]}_{int(time.time())}.wav')
+                else:
+                    output_file = Path(f'{params["output_folder_wav"]}/TTSOUT_{int(time.time())}.wav')
                 output_file_str = output_file.as_posix()
                 output_file = get_output_filename(state)
                 generate_response = send_generate_request(
-                    string, params["voice"], language_code, output_file_str
+                    cleaned_part, params["voice"], language_code, output_file_str
                 )
                 audio_path = generate_response.get("data", {}).get("audio_path")
                 final_output_file = audio_path
@@ -620,6 +658,12 @@ def update_narrator_enabled(value):
     elif value == "Disabled":
         params["narrator_enabled"] = False
 
+def update_non_quoted_text_is(value):
+    global non_quoted_text_is
+    if value == "Narrator":
+        non_quoted_text_is = False
+    elif value == "Char":
+        non_quoted_text_is = True
 
 def input_modifier(string, state):
     if not params["activate"]:
@@ -702,11 +746,16 @@ def ui():
                     },
                     "refresh-button",
                 )
-            narrator_enabled_gr = gr.Radio(
+                narrator_enabled_gr = gr.Radio(
-                choices={"Enabled": "true", "Disabled": "false"},
+                    choices={"Enabled": "true", "Disabled": "false"},
-                label="Narrator Activation",
+                    label="Narrator Activation",
-                value="Enabled" if gr_narrator_enabled == "true" else "Disabled",
+                    value="Enabled" if params.get("narrator_enabled") else "Disabled",
-            )
+                )
+                non_quoted_text_is_gr = gr.Radio(
+                    choices={"Char": "true", "Narrator": "false"},
+                    label='Text NOT inside of * or " is',
+                    value="Char" if non_quoted_text_is else "Narrator",
+                )
 
         with gr.Row():
             low_vram = gr.Checkbox(
@@ -807,9 +856,8 @@ def ui():
 
     # Narrator selection actions
     narrator_enabled_gr.change(update_narrator_enabled, narrator_enabled_gr, None)
-    narrator_voice_gr.change(
+    non_quoted_text_is_gr.change(update_non_quoted_text_is, non_quoted_text_is_gr, None)
-        lambda x: params.update({"narrator_voice": x}), narrator_voice_gr, None
+    narrator_voice_gr.change(lambda x: params.update({"narrator_voice": x}), narrator_voice_gr, None)
-    )
 
     # Play preview
     preview_text.submit(voice_preview, preview_text, preview_audio)

tts_server.py

@@ -1,5 +1,4 @@
 import json
-import random
 import time
 import os
 from pathlib import Path
@@ -15,13 +14,12 @@ from fastapi import (
     FastAPI,
     Form,
     Request,
-    HTTPException,
-    BackgroundTasks,
     Response,
     Depends,
 )
 from fastapi.responses import JSONResponse, HTMLResponse, RedirectResponse
 from fastapi.templating import Jinja2Templates
+from fastapi.staticfiles import StaticFiles
 from contextlib import asynccontextmanager
 
 ###########################
@@ -511,7 +509,6 @@ def list_files(directory):
     ]
     return files
 
-
 #############################
 #### JSON CONFIG UPDATER ####
 #############################
@@ -541,7 +538,8 @@ async def get_settings(request: Request):
             "wav_files": wav_files,
         },
     )
-
+# Define an endpoint to serve static files
+app.mount("/static", StaticFiles(directory=str(this_dir / "templates")), name="static")
 
 @app.post("/update-settings")
 async def update_settings(
@@ -716,6 +714,7 @@ simple_webpage = """
         <li><a href="#getting-started">Getting Started with AllTalk TTS</a></li>
         <li><a href="#server-information">Server Information</a></li>
         <li><a href="#using-voice-samples">Using Voice Samples</a></li>
+        <li><a href="#text-not-inside">Text Not inside function</a></li>
         <li><a href="#local-model-temperature-and-repetition-settings">Local Model temperature and repetition settings</a></li>
         <li><a href="#where-are-the-outputs-stored">Automatic output wav file deletion</a></li>
         <li><a href="#low-vram-option-overview">Low VRAM Overview</a></li>
@@ -733,6 +732,8 @@ simple_webpage = """
     <h2 id="getting-started">Getting Started</h2>
     <p>AllTalk is a web interface, based around the Coqui TTS speech generation system. To generate TTS, you can use the provided interface or interact with the server using CURL commands. Below are some details and examples:</p>
     <p><b>Note:</b> When loading up a new character in Text generation webUI it may look like nothing is happening for 20-30 seconds. Its actually processing the introduction section of the text and once that is completed, it will appear. You can see the activity occuring in the console window. Refreshing the page multiple times will try force the TTS engine to keep re-generating the text, so please just wait and check the console if needed. </p>
+    <p><b>IMPORTANT Note:</b> Ensure that your RP character card has asterisks around anything for the narration and double quotes around anything spoken. There is a complication ONLY with the greeting card so, ensuring it has the correct use of quotes and asterisks will help make sure that the greeting card sounds correct. I will aim to address this issue in a future update.</p>
+    <p>Details on how to edit your greeting card, and where I'm at with solving this, can be found here <a href="https://github.com/erew123/alltalk_tts/tree/main#the-one-thing-i-cant-easily-work-around" target="_blank">erew123's Github</a> and you can look here at an example of how text should be formatted with quotes and asterisks <a href="#text-not-inside">Text Not inside function</a></p>
 
     <h3 id="server-information">Server Information</h3>
     <ul>
@@ -751,7 +752,7 @@ simple_webpage = """
     <p>This extension will download the 2.0.2 model to <b>/extensions/alltalk_tts/models/</b></p>
     <p>This TTS engine will also download the latest available model and store it wherever it normally stores it for your OS (Windows/Linux/Mac).</p>
     <h4>How do I create a new voice sample?</h4>
-    <p>To create a new voice sample you need to make a <b>wav</b> file that is <b>22050Hz</b>, <b>Mono</b>, <b>16 bit</b> and between <b>6 to 12 seconds long</b>, though 7 to 9 seconds is usually good.</p>
+    <p>To create a new voice sample you need to make a <b>wav</b> file that is <b>22050Hz</b>, <b>Mono</b>, <b>16 bit</b> and between <b>6 to 30 seconds long</b>, though 8 to 10 seconds is usually good enough. The model <b>can handle up to a 30 second sample</b>, however Ive not noticed any improvemnt in voice output from a much shorter clip.</p>
     <p>You want to find a nice clear selection of audio, so lets say you wanted to clone your favourite celebrity. You may go looking for an interview where they are talking. Pay close attention to the audio you are listening to and trying to sample, are there noises in the backgroud, hiss on the soundtrack, a low humm, some quiet music playing or something? The better quality the audio the better the result. Dont forget, the AI that processes the sounds can hear everything, all those liitle noises, and it will use them in the voice its trying to recreate. </p>
     <p>Try make your clip one of nice flowing speech, like the included example files. No big pauses, gaps or other sounds. Preferably one that the person you are trying to copy will show a little vocal range and emotion in their voice. Also, try to avoid a clip starting or ending with breathy sounds (breathing in/out etc).</p>
     <h4>Generating the sample!</h4>
@@ -768,6 +769,15 @@ simple_webpage = """
     </ul>
     <p><a href="#toc">Back to top of page</a></p>
 
+    <h3 id="text-not-inside"><strong>Text Not inside function</strong></h3>
+    <p>When using the Narrator function, most AI models should be using asterisks or double quotes to differentiate between the Narrator or the Character, however, many models sometimes switch between using asterisks and double quotes or nothing at all. This leaves a bit of a mess because sometimes that text is narration and sometimes its the character talking and there's no clear way to know where to split sentences. There is no 100&percnt; solution at the moment.</p>
+    <p>Most models usually lean more way than the other as to it is narration or the character talking when it does this. So you can use the "<b>Text NOT inside of * or &quot; is</b>" function to decide what the TTS engine should do in situations like this. It's an either/or situation. It will either use the Character voice or the narrator voice when text is not inside asterisks or double quotes.</p>
+
+    <div style="text-align: center;">
+        <img src="/static/textnotinside.jpg" alt="When the AI doesnt use an asterisk or a quote">
+    </div>
+
+    <p><a href="#toc">Back to top of page</a></p>
 
     <h3 id="local-model-temperature-and-repetition-settings"><strong>Local Model Temperature and Repetition Settings</strong></h3>
     <p><strong>Caution:</strong> It is recommended not to modify these settings unless you fully comprehend their effects. A general overview is provided below for reference.</p>