diff --git a/system/admin.html b/system/admin.html deleted file mode 100644 index 7a988dc..0000000 --- a/system/admin.html +++ /dev/null @@ -1,2615 +0,0 @@ - - - - - - - - AllTalk TTS for Text generation webUI - - - - -

AllTalk TTS

- -
-
-
-

Page Index

- -
-
-

Links

- - - - -

AllTalk Server Status

- -
-
-
- -

🛠️ About this project

-

AllTalk is a labour of love, developed and supported in my personal free time. As such, my ability to respond to - support requests is limited. I prioritize issues based on their impact and the number of users affected. I - appreciate your understanding and patience. If your inquiry isn't covered by the documentation or existing - discussions, and it's not related to a bug or feature request, I'll do my best to assist as time allows. -

-
-

💖 Showing Your Support

-

If AllTalk has been helpful to you, consider showing your support through a donation on my Ko-fi page. Your support is greatly appreciated and helps ensure the - continued development and improvement of AllTalk.

-
- - -

🗣️ Demo/Test TTS

-
-
- - - - - - - - - -
- - -
- - - - - - - - {{ output_file_path }} - -
-
- - - - -
- -

🔊 Using Voice Samples

-

Where are the sample voices stored?

-

Voice samples are stored in /alltalk_tts/voices/ - and should be named using the following format name.wav

- -

🔊 Where are the outputs stored & Automatic output wav file deletion.

-

Voice outputs are stored in /alltalk_tts/outputs/

-

You can configure automatic maintenance deletion of old wav - files by setting Del WAV's older than in the settings above.

-

When Disabled your output - wav files will be left untouched. When set to a setting 1 Day or greater, - your output wav files older than that time period will be automatically deleted on start-up of AllTalk.

- -

🔊 Where are the models stored?

-

This extension will download the 2.0.2 model to /alltalk_tts/models/

-

This TTS engine will also download the latest available model - and store it wherever your OS normally stores it (Windows/Linux/Mac).

- -

🔊 How do I create a new voice sample?

-

To create a new voice sample, you need to make a wav file that - is - 22050Hz, Mono, 16 bit and between 6 to 30 seconds long, though 8 to 10 seconds is usually - good enough. The model can handle up to 30 second samples, however I've not noticed any improvement in voice - output from much longer clips. -

-

You want to find a nice clear selection of audio, so lets say - you wanted to clone your favourite celebrity. You may go looking for an interview where they are talking. Pay - close attention to the audio you are listening to and trying to sample. Are there noises in the background, hiss - on the soundtrack, a low humm, some quiet music playing or something? The better quality the audio the better - the final TTS result. Don't forget, the AI that processes the sounds can hear everything in your sample and it - will use them in the voice its trying to recreate.

-

Try make your clip one of nice flowing speech, like the included - example files. No big pauses, gaps or other sounds. Preferably a sample that the person you are trying to copy - will show a little vocal range and emotion in their voice. Also, try to avoid a clip starting or ending with - breathy sounds (breathing in/out etc).

- -

🔊 Editing your sample!

-

So, you’ve downloaded your favourite celebrity interview off - YouTube, from here you need to chop it down to 6 to 30 seconds in length and resample it.

-

If you need to clean it up, do audio processing, volume level - changes etc, do this before down-sampling.

Using the latest version of Audacity select/highlight your 6 to 30 second clip and:

                  Tracks - > Resample to 22050Hz then
                  Tracks > - Mix > Stereo to - Mono then
              -     File > Export Audio saving it as a WAV of 22050Hz. -

-

Save your generated wav file in the /alltalk_tts/voices/ folder.

-

Its worth mentioning that using AI generated audio clips and bad quality samples may - introduce unwanted sounds into TTS generation.

- -

🔊 Why doesnt it sound like XXX Person?

-

Maybe you might be interested in trying Finetuning of the model. Otherwise, - the reasons - can be that you:

- -

Some samples just never seem to work correctly, so maybe try a - different sample. Always remember though, this is an AI model attempting to re-create a voice, so you will never - get a 100% match.

-

Back to top of page

- -
-

🧮 Low VRAM

-

The Low VRAM option is a crucial feature designed to enhance - performance under constrained (VRAM) conditions, as the TTS models require 2GB-3GB of VRAM to run effectively. - This feature strategically manages the relocation of the Text-to-Speech (TTS) model between your system's Random - Access Memory (RAM) and VRAM, moving it between the two on the fly. Obviously, this is very useful for people - who have smaller graphics cards or people whos LLM has filled their VRAM.

-

When you don't have enough VRAM free after loading your LLM - model - into your VRAM (Normal Mode example below), you can see that with so little working space, your GPU will have to - swap in and out bits of the TTS model, which causes horrible slowdown.

-

Note: An Nvidia Graphics - card is required for the LowVRAM option to work, as you will just be using system RAM otherwise. 

- -

🧮 How It Works

-

The Low VRAM mode intelligently orchestrates the relocation of - the entire TTS model and stores the TTS model in your system RAM. When the TTS engine requires VRAM for - processing, the entire model seamlessly moves into VRAM, causing your LLM to unload/displace some layers, - ensuring optimal performance of the TTS engine.

-

Post-TTS processing, the model moves back to system RAM, freeing - up VRAM space for your Language Model (LLM) to load back in the missing layers. This adds about 1-2 seconds to - both text generation by the LLM and the TTS engine.

-

By transferring the entire model between RAM and VRAM, the Low - VRAM option avoids fragmentation, ensuring the TTS model remains cohesive and has all the working space it needs - in your GPU, without having to just work on small bits of the TTS model at a time (which causes terrible slow - down).

-

This creates a TTS generation performance Boost for Low VRAM - Users and is particularly beneficial for users with less than 2GB of free VRAM after loading their LLM, - delivering a substantial 5-10x improvement in TTS generation speed.

-

🧮 Visual explanation

-
How Low VRAM Works
- -

Back to top of page

-
- - -

🏰 SillyTavern Support

-

Important note for Text-generation-webui users

-

You HAVE to disable - Enable TTS within the Text-generation-webui AllTalk interface, - otherwise Text-generation-webui will also generate TTS due to the way it sends out text. You can do this - each time you start up Text-generation-webui or set it in the start-up settings at the top of this page. -

-

🏰 Quick Tips

- -

🏰 TTS Generation Methods in SilllyTavern

-

You have 2 types of audio generation options, Streaming and Standard.

-

The Streaming Audio Generation method is designed for speed and - is best suited for situations where you just want quick audio playback. This method, however, is limited to - using just one voice per TTS generation request. This means a limitation of the Streaming method is the - inability to utilize the AllTalk narrator function, making it a straightforward but less nuanced option.

-

On the other hand, the Standard Audio Generation method provides - a richer auditory experience. It's slightly slower than the Streaming method but compensates for this with its - ability to split text into multiple voices. This functionality is particularly useful in scenarios where - differentiating between character dialogues and narration can enhance the storytelling and delivery. The - inclusion of the AllTalk narrator functionality in the Standard method allows for a more layered and immersive - experience, making it ideal for content where depth and variety in voice narration add significant value.

-

In summary, the choice between Streaming and Standard methods in - AllTalk TTS depends on what you want. Streaming is great for quick and simple audio generation, while Standard - is preferable for a more dynamic and engaging audio experience.

- -

Changing model or DeepSpeed or Low VRAM each take about 15 - seconds so you should only change one at - a time and wait for Ready before changing the next setting. To set these - options long term you can apply the settings at the top of this page.

-

🏰 AllTalk Narrator

-

Only available on the Standard Audo Generation method.

- -

🏰 Usage Notes:

- -

🏰 Troubleshooting:

- -

Back to top of page

-
-

🟩 Good to know

- -

🟩 A note on Character Cards, Greeting Messages & - Text-not-inside

-
- Click to expand - -

Messages intended for the Narrator should be enclosed in asterisks * and those for the character - inside quotation marks ". However, AI systems often deviate from these rules, resulting in - text that is neither in quotes nor asterisks. Sometimes, text may appear with only a single asterisk, and AI - models may vary their formatting mid-conversation. For example, they might use asterisks initially and then - switch to unmarked text. A properly formatted line should look like this:

-

"Hey! I'm so excited to finally meet you. I've heard so many great things about you - and I'm eager to pick your brain about computers." *She walked across the - room and picked up her cup of coffee*

-

Most narrator/character systems switch voices upon encountering an asterisk or quotation marks, which is - somewhat effective. AllTalk has undergone several revisions in its sentence splitting and identification - methods. While some irregularities and AI deviations in message formatting are inevitable, any line - beginning or ending with an asterisk should now be recognized as Narrator dialogue. Lines enclosed in double - quotes are identified as Character dialogue. For any other text, you can choose how AllTalk handles it: - whether it should be interpreted as Character or Narrator dialogue (most AI systems tend to lean more - towards one format when generating text not enclosed in quotes or asterisks).

-

With improvements to the splitter/processor, I'm confident it's functioning well. You can monitor - what AllTalk identifies as Narrator lines on the command line and adjust its behavior if needed (Text Not - Inside - Function).

-
When the AI doesnt use an asterisk or a quote
-
- - -

🟩 Changing AllTalks IP address & - Accessing AllTalk over your Network

-
- Click to expand - -

AllTalk is coded to start on 127.0.0.1, meaning that it will ONLY be accessable to the local computer it is - running on. If you want to make AllTalk available to other systems on your network, you will need to change - its IP address to match the IP address of your network card/computers current IP address. There are 2x ways - to change the IP address:

-
    -
  1. Start AllTalk and within its web interface and you can edit the IP address on the "AllTalk Startup - Settings".
    2. -
  2. You can edit the confignew.jsonfile in a text editor and change - "ip_address": "127.0.0.1", to the IP address of your choosing. -
    3. -
-

So, for example, if your computer's network card was on IP address 192.168.0.20, you would change - AllTalk's setting to 192.168.1.20 and then restart AllTalk. You will need to ensure - your machine stays on this IP address each time it is restarted, by setting your machine to have a static IP - address.

-
- -

🟩 Text-geneneration-webui - & Stable-Diffusion Plugin - Load Order & stripped text

-
- Click to expand - -

The Stable Diffusion plugin for Text-generation-webui strips out some of the text, which is - passed to Stable Diffusion for image/scene generation. Because this text is stripped, its important to - consider the load order of the plugins to get the desired result you want. Lets assume the AI has just - generated the following message - *He walks into the room with a smile on his face and says* Hello how are you?. Depending on the - load order will change what text reaches AllTalk for generation e.g. -

-

SD Plugin loaded before AllTalk - Only Hi how are you? is sent to AllTalk, with - the *He walks into the room with a smile on his face and says* being sent over to SD for image - generation. Narration of the scene is not possible.

- AllTalk loaded before SD Plugin - - *He walks into the room with a smile on his face and says* Hello how are you? is sent to - AllTalk with the *He walks into the room with a smile on his face and says* being sent over to - SD for image generation.

- The load order can be changed within Text-generation-webui's settings.yaml file or - cmd_flags.txt (depending on how you are managing your extensions).

- image -

-
- -

🟩 I want to know more about the XTTS AI model used -

-
- Click to expand - -

Currently the XTTS model is the main model used by AllTalk for TTS generation. If you want to know more - details about the XTTS model, its capabilties or its technical features you can look at resources such as: -

- -
-

Back to top of page

-
-

🟪 Updating

-

Maintaining the latest version of your setup ensures access to new features and improvements. Below are the steps - to update your installation, whether you're using Text-Generation-webui or running as a Standalone - Application.

-
- UPDATING - Text-Generation-webui - -

The update process closely mirrors the installation steps. Follow these to ensure your setup remains current: -

-
    -
  1. -

    Open a Command Prompt/Terminal:

    -
      -
    • Navigate to your Text-Generation-webui folder with:
        -
      • cd text-generation-webui
        • -
      -
      • -
    -
    2. -
  2. -

    Start the Python Environment:

    -
      -
    • Activate the Python environment tailored for your operating system. Use the appropriate command - from below based on your OS:
        -
      • Windows: cmd_windows.bat
        • -
      • Linux: ./cmd_linux.sh
        • -
      • macOS: cmd_macos.sh
        • -
      • WSL (Windows Subsystem for Linux): cmd_wsl.bat

        • -
      -
      • -
    -
    -

    If you're unfamiliar with Python environments and wish to learn more, consider reviewing - Understanding Python Environments Simplified in the Help section. -

    -
    -
    3. -
  3. -

    Navigate to the AllTalk TTS Folder:

    -
      -
    • Move into your extensions and then the alltalk_tts directory:
        -
      • cd extensions/alltalk_tts
        • -
      -
      • -
    -
    4. -
  4. -

    Update the Repository:

    -
      -
    • Fetch the latest updates from the repository with:
        -
      • git pull
        • -
      -
      • -
    -
    5. -
  5. -

    Install Updated Requirements:

    -
      -
    • Depending on your machine's OS, install the required dependencies using pip:
        -
      • For Windows Machines:
          -
        • pip install -r system\requirements\requirements_textgen.txt
          • -
        -
        • -
      • For Linux/Mac:
          -
        • pip install -r system/requirements/requirements_textgen.txt
          • -
        -
        • -
      -
      • -
    -
    6. -
-
- -
- UPDATING - Standalone Application - -

For Standalone Application users, here's how to update your setup:

-
    -
  1. -

    Open a Command Prompt/Terminal:

    -
      -
    • Navigate to your AllTalk folder with:
        -
      • cd alltalk_tts
        • -
      -
      • -
    -
    2. -
  2. -

    Access the Python Environment:

    -
      -
    • In a command prompt or terminal window, navigate to your alltalk_tts directory and - start the Python environment:
        -
      • Windows:
          -
        • start_environment.bat
          • -
        -
        • -
      • Linux/macOS:
          -
        • ./start_environment.sh
          • -
        -
        • -
      -
      • -
    -
    3. -
-
-

If you're unfamiliar with Python environments and wish to learn more, consider reviewing - Understanding Python Environments Simplified in the Help section. -

-
-
    -
  1. -

    Pull the Latest Updates:

    -
      -
    • Retrieve the latest changes from the repository with:
        -
      • git pull
        • -
      -
      • -
    -
    2. -
  2. -

    Install Updated Requirements:

    -
      -
    • Depending on your machine's OS, install the required dependencies using pip:
        -
      • For Windows Machines:
          -
        • pip install -r system\requirements\requirements_standalone.txt
          • -
        -
        • -
      • For Linux/Mac:
          -
        • pip install -r system/requirements/requirements_standalone.txt
          • -
        -
        • -
      -
      • -
    -
    3. -
-
- -

🟪 Resolving Update Issues

-

If you encounter problems during or after an update, following these steps can help resolve the issue by - refreshing your installation while preserving your data:

-
- RESOLVING - Text-Generation-webui - -

The process involves renaming your existing alltalk_tts directory, setting up a fresh instance, - and then migrating your data:

-
    -
  1. -

    Rename Existing Directory:

    -
      -
    • First, rename your current alltalk_tts folder to back it up, e.g., - alltalk_tts.old. This preserves any existing data. -
      • -
    -
    2. -
  2. -

    Open a Console/Terminal:

    -
      -
    • -

      Navigate to the Text-generation-webui directory and start the Python environment appropriate - for your operating system:

      -
        -
      • cd text-generation-webui
        • -
      -
      • -
    • -

      Then use one of the following commands based on your OS:

      -
        -
      • Windows: cmd_windows.bat
        • -
      • Linux: ./cmd_linux.sh
        • -
      • macOS: cmd_macos.sh
        • -
      • WSL (Windows Subsystem for Linux): cmd_wsl.bat

        • -
      -
      -

      If you're not familiar with Python environments, see Understanding Python - Environments Simplified in the Help section for more info.

      -
      -
      • -
    -
    3. -
  3. -

    Clone the AllTalk TTS Repository:

    -
      -
    • Move into the extensions directory and clone a fresh copy of - alltalk_tts:
        -
      • cd extensions
        • -
      • git clone https://github.com/erew123/alltalk_tts
        • -
      -
      • -
    -
    4. -
  4. -

    Install Requirements:

    -
      -
    • Navigate to the newly cloned alltalk_tts directory and install the necessary - dependencies for your system:
        -
      • cd alltalk_tts
        • -
      -
      • -
    • Depending on your machine's OS, install the required dependencies using pip:
        -
      • For Windows Machines:
          -
        • pip install -r system\requirements\requirements_textgen.txt
          • -
        -
        • -
      • For Linux/Mac:
          -
        • pip install -r system/requirements/requirements_textgen.txt
          • -
        -
        • -
      -
      • -
    -
    5. -
  5. -

    Migrate Your Data:

    -
      -
    • Before starting the application, transfer the models, voices, and - outputs folders from alltalk_tts.old to the new - alltalk_tts directory. This action preserves your voice history and prevents the - need to re-download the model. -
      • -
    -
    6. -
-

You're now ready to launch Text-generation-webui. Note that you may need to reapply any previously saved - configuration changes through the configuration page.

-
    -
  1. Final Step:
    2. -
- -
-

Back to top of page

-
-

🔵🟢 DeepSpeed Installation Options

-

DeepSpeed requires an Nvidia Graphics card

-

DeepSpeed provides a 2x-3x speed boost for Text-to-Speech and AI - tasks. It's all about making AI and TTS happen faster and more efficiently.

-

🔵 Linux Installation

-

DeepSpeed requires access to the Nvidia CUDA Development Toolkit to compile on a Linux system. - It's important to note that this toolkit is distinct and unrealted to your graphics card driver or the CUDA - version the Python environment uses.

-
- Linux DeepSpeed - Text-generation-webui - -

DeepSpeed Installation for Text generation webUI

-
    -
  1. -

    Nvidia CUDA Development Toolkit Installation:

    -
      -
    • The toolkit is crucial for DeepSpeed to compile/build for your version of Linux and requires - around 3GB's of disk space.
      • -
    • Install using your package manager (Recommended) e.g. CUDA Toolkit - 11.8 or download directly from Nvidia CUDA Toolkit Archive - (choose 11.8 or 12.1 for Linux).
      • -
    -
    2. -
  2. -

    Open a Terminal Console:

    -
      -
    • After Nvidia CUDA Development Toolkit installation, access your terminal console.
      • -
    -
    3. -
  3. -

    Install libaio-dev:

    -
      -
    • -

      Use your Linux distribution's package manager.

      -
      • -
    • -
        -
      • sudo apt install libaio-dev for Debian-based systems
        • -
      • sudo yum install libaio-devel for RPM-based systems.
        • -
      -
      • -
    -
    4. -
  4. -

    Navigate to Text generation webUI Folder:

    -
      -
    • Change directory to your Text generation webUI folder with - cd text-generation-webui. -
      • -
    -
    5. -
  5. -

    Activate Text generation webUI Custom Conda Environment:

    -
      -
    • Run ./cmd_linux.sh to start the environment.

      • -
    -
    -

    If you're unfamiliar with Python environments and wish to learn more, consider reviewing - Understanding Python Environments Simplified in the Help section. -

    -
    -
    6. -
  6. -

    Set CUDA_HOME Environment Variable:

    -
      -
    • DeepSpeed locates the Nvidia toolkit using the CUDA_HOME environment variable.
      • -
    • You will only set this temporarily as Text generation webUI sets up its own CUDA_HOME - environment each time you use ./cmd_linux.sh or ./start_linux.sh
      • -
    -
    7. -
  7. -

    Temporarily Configuring CUDA_HOME:

    -
      -
    • -

      When the Text generation webUI Python environment is active (step 5), set - CUDA_HOME. -

      -
      • -
    • -
        -
      • export CUDA_HOME=/usr/local/cuda
        • -
      • export PATH=${CUDA_HOME}/bin:${PATH}
        • -
      • export LD_LIBRARY_PATH=${CUDA_HOME}/lib64:$LD_LIBRARY_PATH
        • -
      -
      • -
    • -

      You can confirm the path is set correctly and working by running the command - nvcc --version should confirm - Cuda compilation tools, release 11.8.. -

      -
      • -
    • -

      Incorrect path settings may lead to errors. If you encounter path issues or receive errors - like [Errno 2] No such file or directory when you run the next step, confirm - the path correctness or adjust as necessary.

      -
      • -
    -
    8. -
  8. -

    DeepSpeed Installation:

    -
      -
    • Install DeepSpeed using pip install deepspeed.
      • -
    -
    9. -
  9. -

    Troubleshooting:

    -
      -
    • Troubleshooting steps for DeepSpeed installation can be located down below.
      • -
    • NOTE: You DO NOT need to set Text-generation-webUI's - --deepspeed setting for AllTalk to be able to use DeepSpeed. These are two - completely separate things and incorrectly setting that on Text-generation-webUI may cause other - complications. -
-
- Linux DeepSpeed - Standalone Installation - - - - -

DeepSpeed Installation for Standalone AllTalk

-
    -
  1. -

    Nvidia CUDA Development Toolkit Installation:

    -
      -
    • The toolkit is crucial for DeepSpeed to compile/build for your version of Linux and requires - around 3GB's of disk space.
      • -
    • Install using your package manager (Recommended) e.g. CUDA Toolkit - 11.8 or download directly from Nvidia CUDA Toolkit Archive - (choose 11.8 or 12.1 for Linux).
      • -
    -
    2. -
  2. -

    Open a Terminal Console:

    -
      -
    • After Nvidia CUDA Development Toolkit installation, access your terminal console.
      • -
    -
    3. -
  3. -

    Install libaio-dev:

    -
      -
    • -

      Use your Linux distribution's package manager.

      -
      • -
    • -
        -
      • sudo apt install libaio-dev for Debian-based systems
        • -
      • sudo yum install libaio-devel for RPM-based systems.
        • -
      -
      • -
    -
    4. -
  4. -

    Navigate to AllTalk TTS Folder:

    -
      -
    • Change directory to your AllTalk TTS folder with cd alltalk_tts.
      • -
    -
    5. -
  5. -

    Activate AllTalk Custom Conda Environment:

    -
      -
    • Run ./start_environment.sh to start the AllTalk Python environment.
      • -
    • This command will start the custom Python environment that was installed with - ./atsetup.sh.

      -
      • -
    -
    -

    If you're unfamiliar with Python environments and wish to learn more, consider reviewing - Understanding Python Environments Simplified in the Help section. -

    -
    -
    6. -
  6. -

    Set CUDA_HOME Environment Variable:

    -
      -
    • The DeepSpeed installation routine locates the Nvidia toolkit using the CUDA_HOME - environment variable. This can be set temporarily for a session or permanently, depending on - other requirements you may have for other Python/System environments.
      • -
    • For temporary use, proceed to step 8. For a permanent solution, see Conda's - manual on setting environment variables.
      • -
    -
    7. -
  7. -

    (Optional) Permanent CUDA_HOME Setup:

    -
      -
    • If you choose to set CUDA_HOME permanently, follow the instructions in the provided - Conda manual link above.
      • -
    -
    8. -
  8. -

    Configuring CUDA_HOME:

    -
      -
    • -

      When your Python environment is active (step 5), set CUDA_HOME. -

      -
      • -
    • -
        -
      • export CUDA_HOME=/usr/local/cuda
        • -
      • export PATH=${CUDA_HOME}/bin:${PATH}
        • -
      • export LD_LIBRARY_PATH=${CUDA_HOME}/lib64:$LD_LIBRARY_PATH
        • -
      -
      • -
    • -

      You can confirm the path is set correctly and working by running the command - nvcc --version should confirm - Cuda compilation tools, release 11.8.. -

      -
      • -
    • -

      Incorrect path settings may lead to errors. If you encounter path issues or receive errors - like [Errno 2] No such file or directory when you run the next step, confirm - the path correctness or adjust as necessary.

      -
      • -
    -
    9. -
  9. -

    DeepSpeed Installation:

    -
      -
    • Install DeepSpeed using pip install deepspeed.
      • -
    -
    10. -
  10. -

    Starting AllTalk TTS WebUI:

    -
      -
    • Launch the AllTalk TTS interface with ./start_alltalk.sh and enable DeepSpeed.
      • -
    -
    11. -
-

Troubleshooting

- -
- -

🟢 Windows Installation

-

You have 2x options for how to setup DeepSpeed on Windows. Pre-compiled wheel files for specific Python, CUDA and - Pytorch builds, or manually compiling DeepSpeed.

-
- Windows DeepSpeed - Pre-Compiled Wheels (Quick and Easy) - -

DeepSpeed Installation with Pre-compiled Wheels

-
    -
  1. -

    Introduction to Pre-compiled Wheels:

    -
      -
    • The atsetup.bat utility simplifies the installation of DeepSpeed by automatically - downloading and installing pre-compiled wheel files. These files are tailored for - specific versions of Python, CUDA, and PyTorch, ensuring compatibility with - both the Standalone Installation and a standard build of - Text-generation-webui. -
      • -
    -
    2. -
  2. -

    Manual Installation of Pre-compiled Wheels:

    -
      -
    • If needed, pre-compiled DeepSpeed wheel files that I have built are available on the Releases Page. You can manually - install or uninstall these wheels using the following commands:
        -
      • Installation: pip install {deep-speed-wheel-file-name-here}
        • -
      • Uninstallation: pip uninstall deepspeed
        • -
      -
      • -
    -
    3. -
  3. -

    Using atsetup.bat for Simplified Management:

    -
      -
    • For those running the Standalone Installation or a standard build of Text-generation-webui, the - atsetup.bat utility offers the simplest and most efficient way to manage DeepSpeed - installations on Windows. -
      • -
    -
    4. -
-
- -
- Windows DeepSpeed - Manual Compilation - -

Manual DeepSpeed Wheel Compilation

-
    -
  1. -

    Preparation for Manual Compilation:

    -
      -
    • Manual compilation of DeepSpeed wheels is an advanced process that requires:
        -
      • 1-2 hours of your time for initial setup and compilation.
        • -
      • 6-10GB of disk space on your computer.
        • -
      • A solid technical understanding of Windows environments and Python.
        • -
      -
      • -
    -
    2. -
  2. -

    Understanding Wheel Compatibility:

    -
      -
    • A compiled DeepSpeed wheel is uniquely tied to the specific versions of Python, PyTorch, and - CUDA used during its compilation. If any of these versions are changed, you will need to compile - a new DeepSpeed wheel to ensure compatibility.
      • -
    -
    3. -
  3. -

    Compiling DeepSpee Resources:

    -
      -
    • Myself and @S95Sedan have worked to simplify the - compilation process. @S95Sedan has notably improved - the process for later versions of DeepSpeed, ensuring ease of build on Windows.
      • -
    • Because @S95Sedan is now maintaining the instructions - for compiling DeepSpeed on Windows, please visit @S95Sedan's
      DeepSpeed GitHub page.
      • -
    -
    4. -
-
-

🔵🟢 DeepSpeed Performance Example

-
DeepSpeed on vs off
-

Back to top of page

-
-

🆘 Support Requests, Troubleshooting & Feature - requests

-

I'm thrilled to see the enthusiasm and engagement with AllTalk! Your feedback and questions are invaluable, - helping to make this project even better. To ensure everyone gets the help they need efficiently, please - consider the following before submitting a support request:

-

Consult the Documentation: A comprehensive guide and FAQ sections (below) are available to help - you navigate AllTalk. Many common questions and troubleshooting steps are covered here.

-

Search Past Discussions: Your issue or question might already have been addressed in the - discussions area or closed - issues. Please use the search function to see if there's an existing solution or advice that applies - to your situation.

-

Bug Reports: If you've encountered what you believe is a bug, please first check the Updates & Bug Fixes List to see if it's - a known issue or one that's already been resolved. If not, I encourage you to report it by raising a bug - report in the Issues section, providing as much - detail as possible to help identify and fix the issue.

-

Feature Requests: The current Feature request list can be found here. I love hearing your ideas for - new features! While I can't promise to implement every suggestion, I do consider all feedback carefully. - Please share your thoughts in the Discussions - area or via a Feature Request in the Issues - section.

- -

Back to top of page

-
-

🟨 Help with problems

-

     🔄 Minor updates/bug - fixes list can be found here

-

      Please check Github for the most up to date help list here

-
- 🟨 How to make a diagnostics report file - -

If you are on a Windows machine or a Linux machine, you should be able to use the atsetup.bat or - ./atsetup.sh utility to create a diagnositcs file. If you are unable to use the - atsetup utility, please follow the instructions below. -

-
- Manually making a diagnostics report file - -
    -
  1. Open a command prompt window and start the Python environment. Depending on your setup - (Text-generation-webui or Standalone AllTalk), the steps to start the Python environment vary:
    -
    2. -
- -
    -
  1. -

    Run the diagnostics and select the requirements file name you installed AllTalk with:

    -
      -
    • python diagnostics.py
      • -
    -
    2. -
  2. -

    You will have an on screen output showing your environment setttings, file versions request vs - whats installed and details of your graphics card (if Nvidia). This will also create a file - called diagnostics.log in the alltalk_tts folder, that you can upload - if you need to create a support ticket on here.

    -
    3. -
-
Diagnostics

-
-
-

-

Installation and Setup Issues

-
- 🟨 Understanding Python Environments Simplified - -

Think of Python environments like different rooms in your house, each designed for a specific purpose. Just - as you wouldn't cook in the bathroom or sleep in the kitchen, different Python applications need their - own "spaces" or environments because they have unique requirements. Sometimes, these requirements - can clash with those of other applications (imagine trying to cook a meal in a bathroom!). To avoid this, - you can create separate Python environments.

-

Why Separate Environments?

-

Separate environments, like separate rooms, keep everything organized and prevent conflicts. For instance, - one Python application might need a specific version of a library or dependency, while another requires a - different version. Just as you wouldn't store kitchen utensils in the bathroom, you wouldn't want - these conflicting requirements to interfere with each other. Each environment is tailored and customized for - its application, ensuring it has everything it needs without disrupting others.

-

How It Works in Practice:

-

Standalone AllTalk Installation: When you install AllTalk standalone, it's akin to - adding a new room to your house specifically designed for your AllTalk activities. The setup process, using - the atsetup utility, constructs this custom "room" (Python environment - alltalk_environment) with all the necessary tools and furnishings (libraries and dependencies) - that AllTalk needs to function smoothly, without meddling with the rest of your "house" (computer - system). The AllTalk environment is started each time you run start_alltalk or - start_environment within the AllTalk folder. -

-

Text-generation-webui Installation: Similarly, installing Text-generation-webui is like - setting up another specialized room. Upon installation, it automatically creates its own tailored - environment, equipped with everything required for text generation, ensuring a seamless and conflict-free - operation. The Text-generation-webui environment is started each time you run - start_*your-os-version* or cmd_*your-os-version* within the Text-generation-webui - folder. -

-

Managing Environments:

-

Just as you might renovate a room or bring in new furniture, you can also update or modify Python - environments as needed. Tools like Conda or venv make it easy to manage these environments, allowing you to - create, duplicate, activate, or delete them much like how you might manage different rooms in your house for - comfort and functionality.

-

Once you're in the right environment, by activating it, installing or updating dependencies (the tools - and furniture of your Python application) is straightforward. Using pip, a package installer for Python, you - can easily add what you need. For example, to install all required dependencies listed in a requirements.txt - file, you'd use:

-

pip install -r requirements.txt

-

This command tells pip to read the list of required packages and versions from the requirements.txt file and - install them in the current environment, ensuring your application has everything it needs to operate. - It's like having a shopping list for outfitting a room and ensuring you have all the right items - delivered and set up.

-

Remember, just as it's important to use the right tools for tasks in different rooms of your house, - it's crucial to manage your Python environments and dependencies properly to ensure your applications - run as intended.

-

How do I know if I am in a Python environment?:

-

When a Python environment starts up, it changes the command prompt to show the Python environment that it - currently running within that terminal/console.

-
Python Environment

-
- -
- 🟨 Windows & Python requirements for compiling packages (ERROR: Could not build wheels for - TTS) - -

ERROR: Microsoft Visual C++ 14.0 or greater is required or - ERROR: Could not build wheels for TTS. or - ModuleNotFoundError: No module named 'TTS -

-

Python requires that you install C++ development tools on Windows. This is detailed on the Python site here. You would need to install - MSVCv142 - VS 2019 C++ x64/x86 build tools and Windows 10/11 SDK from the C++ - Build tools section. -

-

You can get hold of the Community edition here the during installation, selecting - C++ Build tools and then MSVCv142 - VS 2019 C++ x64/x86 build tools and - Windows 10/11 SDK. -

-

image

-
-
- 🟨 Standalone Install - start_{youros}.xx opens and closes instantly and AllTalk doesnt start - -

This is more than likely caused by having a - in your folder path e.g. - c:\myfiles\alltalk_tts-main. In this circumstance you would be best renaming the folder to - remove the - from its name e.g. c:\myfiles\alltalk_tts, delete the - alltalk_environment folder and start_alltalk.bat or start_alltalk.sh - and then re-run atsetup to re-create the environment and startup files. -

-
-
- 🟨 I think AllTalks requirements file has installed something another extension doesn't like - -

Ive paid very close attention to not impact what Text-generation-webui is requesting on a - factory install. This is one of the requirements of submitting an extension to Text-generation-webui. If you - want to look at a comparison of a factory fresh text-generation-webui installed packages (with cuda 12.1, - though AllTalk's requirements were set on cuda 11.8) you can find that comparison here. This comparison shows that AllTalk is - requesting the same package version numbers as Text-generation-webui or even lower version numbers (meaning - AllTalk will not update them to a later version). What other extensions do, I cant really account for that. -

-

I will note that the TTS engine downgrades Pandas data validator to 1.5.3 though its unlikely to cause any - issues. You can upgrade it back to text-generation-webui default (december 2023) with - pip install pandas==2.1.4 when inside of the python environment. I have noticed no ill effects - from it being a lower or higher version, as far as AllTalk goes. This is also the same behaviour as the - Coqui_tts extension that comes with Text-generation-webui. -

-

Other people are reporting issues with extensions not starting with errors about Pydantic e.g. - pydantic.errors.PydanticImportError: BaseSettings` has been moved to the pydantic-settings package. See https://docs.pydantic.dev/2.5/migration/#basesettings-has-moved-to-pydantic-settings for more details. -

-

Im not sure if the Pydantic version has been recently updated by the Text-generation-webui installer, but - this is nothing to do with AllTalk. The other extension you are having an issue with, need to be updated to - work with Pydantic 2.5.x. AllTalk was updated in mid december to work with 2.5.x. I am not specifically - condoning doing this, as it may have other knock on effects, but within the text-gen Python environment, you - can use pip install pydantic==2.5.0 or pip install pydantic==1.10.13 to change the - version of Pydantic installed.

-
-
- 🟨 I am having problems getting AllTalk to start after changing settings or making a custom setup/model - setup. - -

I would suggest following Problems - Updating and if you still have issues after that, you can raise an issue here

-
- -

Networking and Access Issues

-
- 🟨 I cannot access AllTalk from another machine on my Network - -

You will need to change the IP address within AllTalk's settings from being 127.0.0.1, which only allows - access from the local machine its installed on. To do this, please see Changing - AllTalks IP address & Accessing AllTalk over your Network at the top of this page.

-

You may also need to allow access through your firewall or Antivirus package to AllTalk.

-
- -
- 🟨 I am running a Headless system and need to change the IP Address manually as I cannot reach the - config page - -

To do this you can edit the confignew.json file within the alltalk_tts folder. You - would look for "ip_address": "127.0.0.1", and change the - 127.0.0.1 to your chosen IP address,then save the file and start AllTalk.

-

-

When doing this, be careful not to impact the formatting of the JSON file. Worst case, you can re-download a - fresh copy of confignew.json from this website and that will put you back to a factory setting. -

-
- -

Configuration and Usage Issues

-
- 🟨 I activated DeepSpeed in the settings page, but I didnt install DeepSpeed yet and now I have issues - starting up - -

You can either follow the Problems - Updating and fresh install your config. Or you can edit the confignew.json file within - the alltalk_tts folder. You would look for '"deepspeed_activate": true,' and - change the word true to false `"deepspeed_activate": false,' ,then save the file and try - starting again.

-

If you want to use DeepSpeed, you need an Nvidia Graphics card and to install DeepSpeed on your system. - Instructions are here -

-
- -
- 🟨 I am having problems updating/some other issue where it wont start up/Im sure this is a bug - - -

Please see Problems Updating. If that - doesnt help you can raise an ticket here. It - would be handy to have any log files from the console where your error is being shown. I can only losely - support custom built Python environments and give general pointers. Please create a - diagnostics.log report file to submit with a support request. -

-

Also, is your text-generation-webui up to date? instructions - here

-
- -
- 🟨 I see some red "asyncio" messages - -

As far as I am aware, these are to do with the chrome browser the gradio text-generation-webui in some way. I - raised an issue about this on the text-generation-webui here where you can see that - AllTalk is not loaded and the messages persist. Either way, this is more a warning than an actual issue, so - shouldnt affect any functionality of either AllTalk or text-generation-webui, they are more just an - annoyance.

-
- -

Performance and Compatibility Issues

-
- 🟨 Warning TTS Subprocess has NOT started up yet, Will keep trying for 120 seconds maximum. Please - wait. It times out after 120 seconds.
- When the subprocess is starting 2x things are occurring:
- -

A) Its trying to load the voice model into your graphics card VRAM (assuming you have a - Nvidia Graphics card, otherwise its your system RAM)
- B) Its trying to start up the mini-webserver and send the "ready" signal back to - the main process. -

-

Before giving other possibilities a go, some people with old machines are finding their - startup times are very slow 2-3 minutes. Ive extended the allowed time within the script - from 1 minute to 2 minutes. If you have an older machine and wish to try extending this - further, you can do so by editing script.py and changing startup_wait_time = 120 - (120 seconds, aka 2 minutes) at the top of the script.py file, to a larger value e.g - startup_wait_time = 240 (240 seconds aka 4 minutes). -

-

Note: If you need to create a support ticket, please create a diagnostics.log - report file to submit with a support request. Details on doing this are above.

-

Other possibilities for this issue are:

-
    -
  1. -

    You are starting AllTalk in both your CMD FLAG.txt and settings.yaml file. - The CMD FLAG.txt you would have manually edited and the settings.yaml is - the one you change and save in the session tab of text-generation-webui and you can - Save UI defaults to settings.yaml. Please only have one of those two starting up - AllTalk. -

    -
    2. -
  2. -

    You are not starting text-generation-webui with its normal Python environment. Please start it with - start_{your OS version} as detailed here - (start_windows.bat,./start_linux.sh, start_macos.sh or - start_wsl.bat) OR (cmd_windows.bat, ./cmd_linux.sh, - cmd_macos.sh or cmd_wsl.bat and then python server.py). -

    -
    3. -
  3. -

    You have installed the wrong version of DeepSpeed on your system, for the wrong version of - Python/Text-generation-webui. You can go to your text-generation-webui folder in a terminal/command - prompt and run the correct cmd version for your OS e.g. (cmd_windows.bat, - ./cmd_linux.sh, cmd_macos.sh or cmd_wsl.bat) and then you can - type pip uninstall deepspeed then try loading it again. If that works, please see here - for the correct instructions for installing DeepSpeed here. -

    -
    4. -
  4. -

    You have an old version of text-generation-webui (pre Dec 2023) I have not tested on older versions - of text-generation-webui, so cannot confirm viability on older versions. For instructions on - updating the text-generation-webui, please look here - (update_linux.sh, update_windows.bat, update_macos.sh, or - update_wsl.bat). -

    -
    5. -
  5. -

    You already have something running on port 7851 on your computer, so the mini-webserver cant start on - that port. You can change this port number by editing the confignew.json file and - changing "port_number": "7851" to - "port_number": "7602" or any port number you wish that isn’t - reserved. Only change the number and save the file, do not change the formatting of the document. - This will at least discount that you have something else clashing on the same port number. -

    -
    6. -
  6. -

    You have antivirus/firewalling that is blocking that port from being accessed. If you had to do - something to allow text-generation-webui through your antivirus/firewall, you will have to do that - for this too.

    -
    7. -
  7. -

    You have quite old graphics drivers and may need to update them.

    -
    8. -
  8. -

    Something within text-generation-webui is not playing nicely for some reason. You can go to your - text-generation-webui folder in a terminal/command prompt and run the correct cmd version for your - OS e.g. (cmd_windows.bat, ./cmd_linux.sh, cmd_macos.sh or - cmd_wsl.bat) and then you can type python extensions\alltalk_tts\script.py - and see if AllTalk starts up correctly. If it does then something else is interfering. -

    -
    9. -
  9. -

    Something else is already loaded into your VRAM or there is a crashed python process. Either check - your task manager for erroneous Python processes or restart your machine and try again.

    -
    10. -
  10. -

    You are running DeepSpeed on a Linux machine and although you are starting with - ./start_linux.sh AllTalk is failing there on starting. This is because - text-generation-webui will overwrite some environment variables when it loads its python - environment. To see if this is the problem, from a terminal go into your text-generation-webui - folder and ./cmd_linux.sh then set your environment variable again e.g. - export CUDA_HOME=/usr/local/cuda (this may vary depending on your OS, but this is the - standard one for Linux, and assuming you have installed the CUDA toolkit), then - python server.py and see if it starts up. If you want to edit the environment - permanently you can do so, I have not managed to write full instructions yet, but here is the conda - guide here. -

    -
    11. -
  11. -

    You have built yourself a custom Python environment and something is funky with it. This is very hard - to diagnose as its not a standard environment. You may want to updating text-generation-webui and re - installing its requirements file (whichever one you use that comes down with text-generation-webui). -

    -
- - -
- 🟨 I have multiple GPU's and I have problems running Finetuning - -

Finetuning pulls in various other scripts and some of those scripts can have issues with multiple Nvidia - GPU's being present. Until the people that created those other scripts fix up their code, there is a - workaround to temporarily tell your system to only use the 1x of your Nvidia GPU's. To do this:

- -

Rebooting your system will also unset this. The setting is only applied temporarily.

-

Depending on which of your Nvidia GPU's is the more powerful one, you can change the 0 to - 1 or whichever of your GPU's is the most powerful. -

-
- -
- 🟨 Firefox - Streaming Audio doesnt work on Firefox - -

This is a long standing issue with Firefox and one I am unable to resolve. The solution is to use another web - browser if you want to use Streaming audio. For details of my prior invesitigation please look at this ticket

-
- -

Application Specific Issues

-
- 🟨 SillyTavern - I changed my IP address and now SillyTavern wont connect with AllTalk - -

SillyTavern checks the IP address when loading extensions, saving the IP to its configuration only if the - check - succeeds. For whatever reason, SillyTavern's checks dont always allow changing its IP address a second - time.

- -

To manually change the IP address:

-
    -
  1. Navigate to the SillyTavern Public folder located at /sillytavern/public/.
    2. -
  2. Open the settings.json file.
    3. -
  3. Look for the AllTalk section and find the provider_endpoint entry.
    4. -
  4. Replace localhost with your desired IP address, for example, 192.168.1.64. -
    5. -
-

image

-
- -

TTS Generation Issues & Questions

-
- 🟨 XTTS - Does the XTTS AI Model Support Emotion Control or Singing? - -

No, the XTTS AI model does not currently support direct control over emotions or singing capabilities. While - XTTS infuses generated speech with a degree of emotional intonation based on the context of the text, users - cannot explicitly control this aspect. It's worth noting that regenerating the same line of TTS may - yield slightly different emotional inflections, but there is no way to directly control it with XTTS.

-
-
- 🟨 XTTS - Skips, repeats or pronunciation Issues - -

Firstly, it's important to clarify that the development and maintenance of the XTTS AI models and core - scripts are handled by Coqui, with additional - scripts and libraries from entities like huggingface among many other Python scripts - and libraries used by AllTalk.

-

AllTalk is designed to be a straightforward interface that simplifies setup and interaction with AI TTS - models like XTTS. Currently, AllTalk supports the XTTS model, with plans to include more models in the - future. Please understand that the deep inner workings of XTTS, including reasons why it may skip, repeat, - or mispronounce, along with 3rd party scripts and libraries utilized, are ultimately outside my control.

-

Although I ensure the text processed through AllTalk is accurately relayed to the XTTS model speech - generation process, and I have aimed to mitigate as many issues as much as possible; skips, repeats and bad - pronounciation can still occur.

-

Certain aspects I have not been able to investigate due to my own time limitations, are:

- -

From my experience and anecdotally gained knowledge:

- -

So for example, the female_01.wav file that is provided with AllTalk is a studio quality voice - sample, which the XTTS model was trained on. Typically you will find it unlikely that anomolies occur with - TTS generation when using this voice sample. Hence good quality samples and finetuning, generally improve - results with XTTS.

-

If you wish to try out the XTTS version 2.0.3 model and see if it works better, you can download it from here, replacing all the files within your - /alltalk_tts/models/xttsv2_2.0.2 folder. This is on my list to both test version 2.0.3 more, - but also build a more flexible TTS models downloader, that will not only accomdating other XTTS models, but - also other TTS engines. If you try the XTTS version 2.0.3 model and gleen any insights, please let me know. -

-
-

Back to top of page

-
-

⚫ Finetuning a model

-

If you have a voice that the model doesnt quite reproduce correctly, or indeed you just want to improve the - reproduced voice, then finetuning is a way to train your "XTTSv2 local" model (stored in - /alltalk_tts/models/xxxxx/) on a specific voice. For this you will need:

- -

⚫ How will this work/How complicated is it?

-

Everything has been done to make this as simple as possible. At its simplest, you can literally just download a - large chunk of audio from an interview, and tell the finetuning to strip through it, find spoken parts and build - your dataset. You can literally click 4 buttons, then copy a few files and you are done. At it's more - complicated end you will clean up the audio a little beforehand, but its still only 4x buttons and copying a few - files.

-

⚫ The audio you will use

-

I would suggest that if its in an interview format, you cut out the interviewer speaking in audacity or your - chosen audio editing package. You dont have to worry about being perfect with your cuts, the finetuning Step 1 - will go and find spoken audio and cut it out for you. Is there is music over the spoken parts, for best quality - you would cut out those parts, though its not 100% necessary. As always, try to avoid bad quality audio with - noises in it (humming sounds, hiss etc). You can try something like Audioenhancer to try clean up noisier audio. There is no need to - down-sample any of the audio, all of that is handled for you. Just give the finetuning some good quality audio - to work with.

-

⚫ Can I Finetune a model more than once on - more than one voice

-

Yes you can. You would do these as multiple finetuning's, but its absolutely possible and fine to do. - Finetuning the XTTS model does not restrict it to only being able to reproduce that 1x voice you trained it on. - Finetuning is generally nuding the model in a direction to learn the ability to sound a bit more like a voice - its not heard before.

-

⚫ A note about anonymous training - Telemetry information & disabling it

-

Portions of Coqui's TTS trainer scripts gather anonymous training information which you can disable. Their - statement on this is listed here. If you start - AllTalk Finetuning with start_finetuning.bat or ./start_finetuning.sh telemetry will - be disabled. If you manually want to disable it, please expand the below:

-
- Manually disable telemetry
- -

Before starting finetuning, run the following in your terminal/command prompt:

- -

Before you start finetune.py. You will now be able to finetune offline and no anonymous training - data will be sent.

-
- -

⚫ Prerequisites for Fine-tuning - with Nvidia CUDA Development Toolkit 11.8

-

All the requirements for Finetuning will be installed by using the atsetup utility and installing your correct - requirements (Standalone or for Text-generation-webui). The legacy manual instructions are stored below, however - these shouldnt be required.

-
- Legacy manual instructions for installing Nvidia CUDA Development Toolkit 11.8
- - To perform fine-tuning, a specific portion of the **Nvidia CUDA Development Toolkit v11.8** must be installed. - This is crucial for step 1 of fine-tuning. The objective is to minimize the installation footprint by installing - only the essential components.
- - The **Nvidia CUDA Development Toolkit v11.8** operates independently from your graphics card drivers and the - CUDA version utilized by your Python environment.
- - This installation process aims to keep the download and install size as minimal as possible, however a full - install of the tookit requires 3GB's of disk space.
- - When running Finetuning it will require upto 20GB's of temporary disk space, so please ensure you have this - space available and preferably use a SSD or NVME drive. - -
    -
  1. -

    Download the Toolkit:

    -
      -
    • Obtain the network install version of the Nvidia CUDA Development Toolkit 11.8 - from Nvidia's - Archive.
      • -
    -
    2. -
  2. -

    Run the Installer:

    -
      -
    • Choose Custom (Advanced) installation.
      • -
    • Deselect all options initially.
      • -
    • Select the following components:
        -
      • CUDA > Development > Compiler > - nvcc -
        • -
      • CUDA > Development > Libraries > - CUBLAS (both development and runtime) -
        • -
      -
      • -
    -
    3. -
  3. -

    Configure Environment Search Path:

    -
      -
    • -

      It's essential that nvcc and CUDA 11.8 library files are discoverable in - your environment's search path. Adjustments can be reverted post-fine-tuning if desired. -

      -

      For Windows:

      -
        -
      • Edit the Path environment variable to include - C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.8\bin. -
        • -
      • Add CUDA_HOME and set its path to - C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.8. -
        • -
      -

      For Linux:

      -
        -
      • The path may vary by Linux distribution. Here's a generic setup:
          -
        • -

          export CUDA_HOME=/usr/local/cuda

          -
          • -
        • -

          export PATH=${CUDA_HOME}/bin:${PATH}

          -
          • -
        • -

          export LD_LIBRARY_PATH=${CUDA_HOME}/lib64:$LD_LIBRARY_PATH

          -
          • -
        • -

          Consider adding these to your ~/.bashrc for permanence, or apply - temporarily for the current session by running the above commands each time - you start your Python environment.

          -
          • -
        -
        • -
      -

      Note: If using Text-generation-webui, its best to set these temporarily.

      -
      • -
    -
    4. -
  4. -

    Verify Installation:

    -
      -
    • Open a new terminal/command prompt to refresh the search paths.
      • -
    • In a terminal or command prompt, execute nvcc --version.
      • -
    • Success is indicated by a response of Cuda compilation tools, release 11.8. - Specifically, ensure it is version 11.8.
      • -
    -
    5. -
  5. -

    Troubleshooting:

    -
      -
    • If the correct version isn't reported, recheck your environment path settings for accuracy - and potential conflicts with other CUDA versions. -
- - - - -

Additional Note on Torch and Torchaudio:

- -

⚫ Starting Fine-tuning

-

NOTE: Ensure AllTalk has been launched at least once after any updates to download necessary - files for fine-tuning.

-
    -
  1. -

    Close Resource-Intensive Applications:

    - -
    2. -
  2. -

    Organize Voice Samples:

    - -
    3. -
-

Depending on your setup (Text-generation-webui or Standalone AllTalk), the steps to start the Python environment - vary:

- -
    -
  1. -

    Pre-Flight Checklist:

    - -
    2. -
  2. -

    Post Fine-tuning Actions:

    - -
    3. -
-

These steps guide you through the initial preparations, starting the Python environment based on your setup, and - the fine-tuning process itself. Ensure all prerequisites are met to facilitate a smooth fine-tuning experience. -

-

⚫ How many Epochs etc is the right amount?

-

In finetuning the suggested/recommended amount of epochs, batch size, evaluation percent etc is already set. - However, there is no absolutely correct answer to what the settings should be, it all depends on what you are - doing.

- -

There are no absolute correct settings, as there are too many variables, ranging from the amount of samples you - are using (5 minutes worth? 4 hours worth? etc), if they are similar samples to what the AI model already - understands, so on and so forth. Coqui whom originally trained the model usually say something along the lines - of, once you’ve trained it X amount, if it sounds good then you are done and if it doesn’t, train it more.

-

⚫ Evaluation Data Percentage

-

In the process of finetuning, it's crucial to balance the data used for training the model against the data - reserved for evaluating its performance. Typically, a portion of the dataset is set aside as an 'evaluation - set' to assess the model's capabilities in dealing with unseen data. On Step 1 of finetuning you have - the option to adjust this evaluation data percentage, offering more control over your model training - process.

- Why Adjust the Evaluation Percentage?

- Adjusting the evaluation percentage can be beneficial in scenarios with limited voice samples. - When dealing with a smaller dataset, allocating a slightly larger portion to training could enhance the - model's ability to learn from these scarce samples. Conversely, with abundant data, a higher evaluation - percentage might be more appropriate to rigorously test the model's performance. There are currently no - absolutely optimal split percentages as it varies by dataset. -

- -

⚫ Using a Finetuned model in Text-generation-webui

-

At the end of the finetune process, you will have an option to - Compact and move model to /trainedmodel/ this will compact the raw training file and move it to - /model/trainedmodel/. When AllTalk starts up within Text-generation-webui, if it finds a model in - this location a new loader will appear in the interface for XTTSv2 FT and you can use this to load - your finetuned model.

Be careful not to train a new model from the base model, then - overwrite your current /model/trainedmodel/ if you want a seperately trained - model. This is why there is an OPTION B to move your just trained model to - /models/lastfinetuned/. -

-

⚫ Training one model with multiple voices

-

At the end of the finetune process, you will have an option to - Compact and move model to /trainedmodel/ this will compact the raw training file and move it to - /model/trainedmodel/. This model will become available when you start up finetuning. You will have - a choice to train the Base Model or the Existing finetuned model (which is the one in - /model/trainedmodel/). So you can use this to keep further training this model with additional - voices, then copying it back to /model/trainedmodel/ at the end of training. -

-

⚫ Do I need to keep the raw training data/model?

-

If you've compacted and moved your model, its highly unlikely you would want to keep that data, however the - choice is there to keep it if you wish. It will be between 5-10GB in size, so most people will want to delete - it.

-

Back to top of page

-
-

⬜ AllTalk TTS Generator

- - -

AllTalk TTS Generator is the solution for converting large volumes of text into speech using the voice of your - choice. Whether you're creating audio content or just want to hear text read aloud, the TTS Generator is - equipped to handle it all efficiently. Please see here for a quick demo

The link to open the TTS generator can be - found on the built-in Settings and Documentation page.

DeepSpeed is - highly recommended to speed up generation. Low VRAM would be best turned off - and your LLM model unloaded from your GPU VRAM (unload your model). No Playback will reduce - memory overhead on very large generations (15,000 words or more). Splitting Export to Wav into - smaller groups will also reduce memory overhead at the point of exporting your wav files (so good for low memory - systems). -

-

⬜ Estimated Throughput

-

This will vary by system for a multitude of reasons, however, while generating a 58,000 word document to TTS, - with DeepSpeed enabled, LowVram disabled, splitting size 2 and on an Nvidia RTX 4070, throughput was around - 1,000 words per minute. Meaning, this took 1 hour to generate the TTS. Exporting to combined wavs took about 2-3 - minutes total.

-

⬜ Quick Start

- -

⬜ Customization and Preferences

- -

⬜ Interface and Accessibility

- -

⬜ TTS Generation Modes

- -

⬜ Playback and List Management

- -

⬜ Exporting Your Audio

- -

⬜ Exporting Subtitles (SRT file)

- -

⬜ Analyzing generated TTS for errors

- -

As such, there is a % Accuracy setting. This uses a couple of methods to try find things that are - similar e.g. taking the their and there example from above, it would identify that - they both sound the same, so even if the text says their and the AI listening to the generated TTS - interprets the word as there, it will realise that both sound the same/are similar so there is no - need to flag that as an error. However, there are limits to this and some things may slip through or get picked - up when you would prefer them not to be flagged.

-

The higher the accuracy you choose, the more things it will flag up, however you may get more unwanted - detections. The lower the less detections. Based on my few tests, accuracy settings between 96 to 98 seem to - generally give the best results. However, I would highly recommend you test out a small 10-20 line text and test - out the Analyze TTS button to get a feel for how it responds to different settings, as well as - things it flags up.

-

You will be able to see the ID's and Text (orignal and as interpreted) by looking at the terminal/command - prompt window.

-

The Analyze TTS feature uses the Whisper Larger-v2 AI engine, which will download on first use if necessary. This - will require about 2.5GB's of disk space and could take a few minutes to download, depending on your - internet connection.

-

⬜ Tricks to get the model to say things correctly

-

Sometimes the AI model won’t say something the way that you want it to. It could be because it’s a new word, an - acronym or just something it’s not good at for whatever reason. There are some tricks you can use to improve the - chances of it saying something correctly.

-

Adding pauses
- You can use semi-colons ";" and colons ":" to create a pause, similar to a period - "." which can be helpful with some splitting issues.

-

Acronyms
- Not all acronyms are going to be pronounced correctly. Let’s work with the word ChatGPT. We know it - is pronounced "Chat G P T" but when presented to the model, it doesn’t know how to break - it down correctly. So, there are a few ways we could get it to break out "Chat" and the G P and T. - e.g.

-

Chat G P T. - Chat G,P,T. - Chat G.P.T. - Chat G-P-T. - Chat gee pee tea -

-

All bar the last one are using ways within the English language to split out "Chat" into one word being - pronounced and then split the G, P and T into individual letters. The final example, which is to use Phonetics - will sound perfectly fine, but clearly would look wrong as far as human readable text goes. The phonetics method - is very useful in edge cases where pronunciation difficult.

-

⬜ Notes on Usage

- -

Back to top of page

-
-

🟠 API Suite and JSON-CURL

-

🟠Overview

-

The Text-to-Speech (TTS) Generation API allows you to generate speech from text input using various configuration - options. This API supports both character and narrator voices, providing flexibility for creating dynamic and - engaging audio content.

-

🟠 Ready Endpoint

-

Check if the Text-to-Speech (TTS) service is ready to accept requests.

- -

🟠 Voices List Endpoint

-

Retrieve a list of available voices for generating speech.

- -

🟠 Current Settings Endpoint

-

Retrieve a list of available voices for generating speech.

- -

🟠 Preview Voice Endpoint

-

Generate a preview of a specified voice with hardcoded settings.

- -

🟠 Switching Model Endpoint

- -

🟠 Switch DeepSpeed Endpoint

- -

🟠 Switching Low VRAM Endpoint

- -

🟠 TTS Generation Endpoint (Standard Generation)

-

Streaming endpoint details are further down the page.

- -

🟠 Example command lines (Standard Generation)

-

Standard TTS generation supports Narration and will generate a wav file/blob. Standard TTS speech Example - (standard text) generating a time-stamped file

-

curl -X POST "http://127.0.0.1:7851/api/tts-generate" -d "text_input=All of this is text spoken by the character. This is text not inside quotes, though that doesnt matter in the slightest" -d "text_filtering=standard" -d "character_voice_gen=female_01.wav" -d "narrator_enabled=false" -d "narrator_voice_gen=male_01.wav" -d "text_not_inside=character" -d "language=en" -d "output_file_name=myoutputfile" -d "output_file_timestamp=true" -d "autoplay=true" -d "autoplay_volume=0.8"
-

-

Narrator Example (standard text) generating a time-stamped file

-

curl -X POST "http://127.0.0.1:7851/api/tts-generate" -d "text_input=*This is text spoken by the narrator* \"This is text spoken by the character\". This is text not inside quotes." -d "text_filtering=standard" -d "character_voice_gen=female_01.wav" -d "narrator_enabled=true" -d "narrator_voice_gen=male_01.wav" -d "text_not_inside=character" -d "language=en" -d "output_file_name=myoutputfile" -d "output_file_timestamp=true" -d "autoplay=true" -d "autoplay_volume=0.8"
-

-

Note that if your text that needs to be generated contains double quotes you will need to escape them with - \" (Please see the narrator example). -

-

🟠 Request Parameters

-

🟠 text_input: The text you want the TTS engine to produce. Use escaped double quotes for - character speech and asterisks for narrator speech if using the narrator function. Example:

-

-d "text_input=*This is text spoken by the narrator* \"This is text spoken by the character\". This is text not inside quotes." -

-

🟠 text_filtering: Filter for text. Options:

- -

-d "text_filtering=none"
- -d "text_filtering=standard"
- -d "text_filtering=html"
-

-

Example:

- -

🟠 character_voice_gen: The WAV file name for the character's voice.

-

-d "character_voice_gen=female_01.wav"

-

🟠 narrator_enabled: Enable or disable the narrator function. If true, minimum text filtering is - set to standard. Anything between double quotes is considered the character's speech, and anything between - asterisks is considered the narrator's speech.

-

-d "narrator_enabled=true"
- -d "narrator_enabled=false" -

-

🟠 narrator_voice_gen: The WAV file name for the narrator's voice.

-

-d "narrator_voice_gen=male_01.wav"

-

🟠 text_not_inside: Specify the handling of lines not inside double quotes or asterisks, for the - narrator feature. Options:

- -

-d "text_not_inside=character"
- -d "text_not_inside=narrator" -

-

🟠 language: Choose the language for TTS. Options:

-

ar Arabic
- zh-cn Chinese (Simplified)
- cs Czech
- nl Dutch
- en English
- fr French
- de German
- hu Hungarian
- it Italian
- ja Japanese
- ko Korean
- pl Polish
- pt Portuguese
- ru Russian
- es Spanish
- tr Turkish
-

-

-d "language=en"

-

🟠 output_file_name: The name of the output file (excluding the .wav extension).

-

-d "output_file_name=myoutputfile"

-

🟠 output_file_timestamp: Add a timestamp to the output file name. If true, each file will have - a unique timestamp; otherwise, the same file name will be overwritten each time you generate TTS.

-

-d "output_file_timestamp=true"
- -d "output_file_timestamp=false" -

-

🟠 autoplay: Enable or disable playing the generated TTS to your standard sound output device at - time of TTS generation.

-

-d "autoplay=true"
- -d "autoplay=false" -

-

🟠 autoplay_volume: Set the autoplay volume. Should be between 0.1 and 1.0. Needs to be - specified in the JSON request even if autoplay is false.

-

-d "autoplay_volume=0.8"

-

🟠 TTS Generation Response

-

The API returns a JSON object with the following properties:

- -

Example JSON TTS Generation Response:

-

{"status":"generate-success","output_file_path":"C:\\text-generation-webui\\extensions\\alltalk_tts\\outputs\\myoutputfile_1704141936.wav", "output_file_url":"http://127.0.0.1:7851/audio/myoutputfile_1704141936.wav", "output_cache_url":"http://127.0.0.1:7851/audiocache/myoutputfile_1704141936.wav"} -

-

🟠 TTS Generation Endpoint (Streaming Generation)

-

Streaming TTS generation does NOT support Narration and will generate an audio stream. Streaming TTS speech - JavaScript Example:

- - 
// Example parameters
-        const text = "Here is some text";
-        const voice = "female_01.wav";
-        const language = "en";
-        const outputFile = "stream_output.wav";
-        // Encode the text for URL
-        const encodedText = encodeURIComponent(text);
-        // Create the streaming URL
-        const streamingUrl = `http://localhost:7851/api/tts-generate-streaming?text=${encodedText}&voice=${voice}&language=${language}&output_file=${outputFile}`;
-        // Create and play the audio element
-        const audioElement = new Audio(streamingUrl);
-        audioElement.play(); // Play the audio stream directly
-
- -

Back to top of page

-
-

Thanks & References

-

Coqui TTS Engine

- - -

Extension coded by

- - -

Thanks to & Text generation webUI

- - -

Thanks to

- - -

Back to top of page

- - - - diff --git a/system/at_admin/3c6fde566d6a.jpg b/system/at_admin/3c6fde566d6a.jpg deleted file mode 100644 index c618633..0000000 Binary files a/system/at_admin/3c6fde566d6a.jpg and /dev/null differ diff --git a/system/at_admin/4abed1ca.jpg b/system/at_admin/4abed1ca.jpg deleted file mode 100644 index 852f2fd..0000000 Binary files a/system/at_admin/4abed1ca.jpg and /dev/null differ diff --git a/system/at_admin/at_settings.html b/system/at_admin/at_settings.html deleted file mode 100644 index ddf67a8..0000000 --- a/system/at_admin/at_settings.html +++ /dev/null @@ -1,260 +0,0 @@ - - - - - - - AllTalk Startup Settings - - - - -

AllTalk Startup Settings


-
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - {{ - modeldownload_model_path }} - - - - - - - - - - - - - - - - - -
- - -
- -
- - -
- -
- - -
- -
-

WARNING: Changing RED items or setting DeepSpeed to True on a system without DeepSpeed installed, - may cause errors.

- - - - \ No newline at end of file diff --git a/system/at_admin/atandsdplugin.jpg b/system/at_admin/atandsdplugin.jpg deleted file mode 100644 index d6c5e79..0000000 Binary files a/system/at_admin/atandsdplugin.jpg and /dev/null differ diff --git a/system/at_admin/deepspeedexample.jpg b/system/at_admin/deepspeedexample.jpg deleted file mode 100644 index b193b49..0000000 Binary files a/system/at_admin/deepspeedexample.jpg and /dev/null differ diff --git a/system/at_admin/favicon.ico b/system/at_admin/favicon.ico deleted file mode 100644 index 1e0d7ff..0000000 Binary files a/system/at_admin/favicon.ico and /dev/null differ diff --git a/system/at_admin/lowvrammode.png b/system/at_admin/lowvrammode.png deleted file mode 100644 index 4fef834..0000000 Binary files a/system/at_admin/lowvrammode.png and /dev/null differ diff --git a/system/at_admin/pythonenvironment.jpg b/system/at_admin/pythonenvironment.jpg deleted file mode 100644 index 1cb3de2..0000000 Binary files a/system/at_admin/pythonenvironment.jpg and /dev/null differ diff --git a/system/at_admin/pythonrequirementswindows.jpg b/system/at_admin/pythonrequirementswindows.jpg deleted file mode 100644 index c57e4d7..0000000 Binary files a/system/at_admin/pythonrequirementswindows.jpg and /dev/null differ diff --git a/system/at_admin/textnotinside.jpg b/system/at_admin/textnotinside.jpg deleted file mode 100644 index 74523d2..0000000 Binary files a/system/at_admin/textnotinside.jpg and /dev/null differ diff --git a/system/at_sounds/apilocal.wav b/system/at_sounds/apilocal.wav deleted file mode 100644 index 1e7d818..0000000 Binary files a/system/at_sounds/apilocal.wav and /dev/null differ diff --git a/system/at_sounds/apitts.wav b/system/at_sounds/apitts.wav deleted file mode 100644 index ef28fb2..0000000 Binary files a/system/at_sounds/apitts.wav and /dev/null differ diff --git a/system/at_sounds/changingmodel.wav b/system/at_sounds/changingmodel.wav deleted file mode 100644 index 31b2f20..0000000 Binary files a/system/at_sounds/changingmodel.wav and /dev/null differ diff --git a/system/at_sounds/deepspeeddisabled.wav b/system/at_sounds/deepspeeddisabled.wav deleted file mode 100644 index e276b47..0000000 Binary files a/system/at_sounds/deepspeeddisabled.wav and /dev/null differ diff --git a/system/at_sounds/deepspeedenabled.wav b/system/at_sounds/deepspeedenabled.wav deleted file mode 100644 index e711b0f..0000000 Binary files a/system/at_sounds/deepspeedenabled.wav and /dev/null differ diff --git a/system/at_sounds/lowvramdisabled.wav b/system/at_sounds/lowvramdisabled.wav deleted file mode 100644 index ac6fef9..0000000 Binary files a/system/at_sounds/lowvramdisabled.wav and /dev/null differ diff --git a/system/at_sounds/lowvramenabled.wav b/system/at_sounds/lowvramenabled.wav deleted file mode 100644 index e165bb5..0000000 Binary files a/system/at_sounds/lowvramenabled.wav and /dev/null differ diff --git a/system/at_sounds/xttsfinetuned.wav b/system/at_sounds/xttsfinetuned.wav deleted file mode 100644 index 6bddcfd..0000000 Binary files a/system/at_sounds/xttsfinetuned.wav and /dev/null differ diff --git a/system/at_sounds/xttslocal.wav b/system/at_sounds/xttslocal.wav deleted file mode 100644 index 0129168..0000000 Binary files a/system/at_sounds/xttslocal.wav and /dev/null differ diff --git a/system/config/at_configdowngrade.json b/system/config/at_configdowngrade.json deleted file mode 100644 index e02aef2..0000000 --- a/system/config/at_configdowngrade.json +++ /dev/null @@ -1,2 +0,0 @@ -{ -} \ No newline at end of file diff --git a/system/config/at_configupdate.json b/system/config/at_configupdate.json deleted file mode 100644 index e02aef2..0000000 --- a/system/config/at_configupdate.json +++ /dev/null @@ -1,2 +0,0 @@ -{ -} \ No newline at end of file diff --git a/system/config/harvard_sentences.txt b/system/config/harvard_sentences.txt deleted file mode 100644 index 1947d9c..0000000 --- a/system/config/harvard_sentences.txt +++ /dev/null @@ -1,720 +0,0 @@ -The birch canoe slid on the smooth planks. -Glue the sheet to the dark blue background. -It's easy to tell the depth of a well. -These days a chicken leg is a rare dish. -Rice is often served in round bowls. -The juice of lemons makes fine punch. -The box was thrown beside the parked truck. -The hogs were fed chopped corn and garbage. -Four hours of steady work faced us. -A large size in stockings is hard to sell. -The boy was there when the sun rose. -A rod is used to catch pink salmon. -The source of the huge river is the clear spring. -Kick the ball straight and follow through. -Help the woman get back to her feet. -A pot of tea helps to pass the evening. -Smoky fires lack flame and heat. -The soft cushion broke the man's fall. -The salt breeze came across from the sea. -The girl at the booth sold fifty bonds. -The small pup gnawed a hole in the sock. -The fish twisted and turned on the bent hook. -Press the pants and sew a button on the vest. -The swan dive was far short of perfect. -The beauty of the view stunned the young boy. -Two blue fish swam in the tank. -Her purse was full of useless trash. -The colt reared and threw the tall rider. -It snowed, rained, and hailed the same morning. -Read verse out loud for pleasure. -Hoist the load to your left shoulder. -Take the winding path to reach the lake. -Note closely the size of the gas tank. -Wipe the grease off his dirty face. -Mend the coat before you go out. -The wrist was badly strained and hung limp. -The stray cat gave birth to kittens. -The young girl gave no clear response. -The meal was cooked before the bell rang. -What joy there is in living. -A king ruled the state in the early days. -The ship was torn apart on the sharp reef. -Sickness kept him home the third week. -The wide road shimmered in the hot sun. -The lazy cow lay in the cool grass. -Lift the square stone over the fence. -The rope will bind the seven books at once. -Hop over the fence and plunge in. -The friendly gang left the drug store. -Mesh wire keeps chicks inside. -The frosty air passed through the coat. -The crooked maze failed to fool the mouse. -Adding fast leads to wrong sums. -The show was a flop from the very start. -A saw is a tool used for making boards. -The wagon moved on well oiled wheels. -March the soldiers past the next hill. -A cup of sugar makes sweet fudge. -Place a rosebush near the porch steps. -Both lost their lives in the raging storm. -We talked of the side show in the circus. -Use a pencil to write the first draft. -He ran half way to the hardware store. -The clock struck to mark the third period. -A small creek cut across the field. -Cars and busses stalled in snow drifts. -The set of china hit the floor with a crash. -This is a grand season for hikes on the road. -The dune rose from the edge of the water. -Those words were the cue for the actor to leave. -A yacht slid around the point into the bay. -The two met while playing on the sand. -The ink stain dried on the finished page. -The walled town was seized without a fight. -The lease ran out in sixteen weeks. -A tame squirrel makes a nice pet. -The horn of the car woke the sleeping cop. -The heart beat strongly and with firm strokes. -The pearl was worn in a thin silver ring. -The fruit peel was cut in thick slices. -The Navy attacked the big task force. -See the cat glaring at the scared mouse. -There are more than two factors here. -The hat brim was wide and too droopy. -The lawyer tried to lose his case. -The grass curled around the fence post. -Cut the pie into large parts. -Men strive but seldom get rich. -Always close the barn door tight. -He lay prone and hardly moved a limb. -The slush lay deep along the street. -A wisp of cloud hung in the blue air. -A pound of sugar costs more than eggs. -The fin was sharp and cut the clear water. -The play seems dull and quite stupid. -Bail the boat to stop it from sinking. -The term ended in late June that year. -A tusk is used to make costly gifts. -Ten pins were set in order. -The bill was paid every third week. -Oak is strong and also gives shade. -Cats and dogs each hate the other. -The pipe began to rust while new. -Open the crate but don't break the glass. -Add the sum to the product of these three. -Thieves who rob friends deserve jail. -The ripe taste of cheese improves with age. -Act on these orders with great speed. -The hog crawled under the high fence. -Move the vat over the hot fire. -The bark of the pine tree was shiny and dark. -Leaves turn brown and yellow in the fall. -The pennant waved when the wind blew. -Split the log with a quick, sharp blow. -Burn peat after the logs give out. -He ordered peach pie with ice cream. -Weave the carpet on the right hand side. -Hemp is a weed found in parts of the tropics. -A lame back kept his score low. -We find joy in the simplest things. -Type out three lists of orders. -The harder he tried the less he got done. -The boss ran the show with a watchful eye. -The cup cracked and spilled its contents. -Paste can cleanse the most dirty brass. -The slang word for raw whiskey is booze. -It caught its hind paw in a rusty trap. -The wharf could be seen at the farther shore. -Feel the heat of the weak dying flame. -The tiny girl took off her hat. -A cramp is no small danger on a swim. -He said the same phrase thirty times. -Pluck the bright rose without leaves. -Two plus seven is less than ten. -The glow deepened in the eyes of the sweet girl. -Bring your problems to the wise chief. -Write a fond note to the friend you cherish. -Clothes and lodging are free to new men. -We frown when events take a bad turn. -Port is a strong wine with a smoky taste. -The young kid jumped the rusty gate. -Guess the results from the first scores. -A salt pickle tastes fine with ham. -The just claim got the right verdict. -These thistles bend in a high wind. -Pure bred poodles have curls. -The tree top waved in a graceful way. -The spot on the blotter was made by green ink. -Mud was spattered on the front of his white shirt. -The cigar burned a hole in the desk top. -The empty flask stood on the tin tray. -A speedy man can beat this track mark. -He broke a new shoelace that day. -The coffee stand is too high for the couch. -The urge to write short stories is rare. -The pencils have all been used. -The pirates seized the crew of the lost ship. -We tried to replace the coin but failed. -She sewed the torn coat quite neatly. -The sofa cushion is red and of light weight. -The jacket hung on the back of the wide chair. -At that high level the air is pure. -Drop the two when you add the figures. -A filing case is now hard to buy. -An abrupt start does not win the prize. -Wood is best for making toys and blocks. -The office paint was a dull, sad tan. -He knew the skill of the great young actress. -A rag will soak up spilled water. -A shower of dirt fell from the hot pipes. -Steam hissed from the broken valve. -The child almost hurt the small dog. -There was a sound of dry leaves outside. -The sky that morning was clear and bright blue. -Torn scraps littered the stone floor. -Sunday is the best part of the week. -The doctor cured him with these pills. -The new girl was fired today at noon. -They felt gay when the ship arrived in port. -Add the store's account to the last cent. -Acid burns holes in wool cloth. -Fairy tales should be fun to write. -Eight miles of woodland burned to waste. -The third act was dull and tired the players. -A young child should not suffer fright. -Add the column and put the sum here. -We admire and love a good cook. -There the flood mark is ten inches. -He carved a head from the round block of marble. -She has a smart way of wearing clothes. -The fruit of a fig tree is apple-shaped. -Corn cobs can be used to kindle a fire. -Where were they when the noise started. -The paper box is full of thumb tacks. -Sell your gift to a buyer at a good gain. -The tongs lay beside the ice pail. -The petals fall with the next puff of wind. -Bring your best compass to the third class. -They could laugh although they were sad. -Farmers came in to thresh the oat crop. -The brown house was on fire to the attic. -The lure is used to catch trout and flounder. -Float the soap on top of the bath water. -A blue crane is a tall wading bird. -A fresh start will work such wonders. -The club rented the rink for the fifth night. -After the dance, they went straight home. -The hostess taught the new maid to serve. -He wrote his last novel there at the inn. -Even the worst will beat his low score. -The cement had dried when he moved it. -The loss of the second ship was hard to take. -The fly made its way along the wall. -Do that with a wooden stick. -Live wires should be kept covered. -The large house had hot water taps. -It is hard to erase blue or red ink. -Write at once or you may forget it. -The doorknob was made of bright clean brass. -The wreck occurred by the bank on Main Street. -A pencil with black lead writes best. -Coax a young calf to drink from a bucket. -Schools for ladies teach charm and grace. -The lamp shone with a steady green flame. -They took the axe and the saw to the forest. -The ancient coin was quite dull and worn. -The shaky barn fell with a loud crash. -Jazz and swing fans like fast music. -Rake the rubbish up and then burn it. -Slash the gold cloth into fine ribbons. -Try to have the court decide the case. -They are pushed back each time they attack. -He broke his ties with groups of former friends. -They floated on the raft to sun their white backs. -The map had an X that meant nothing. -Whitings are small fish caught in nets. -Some ads serve to cheat buyers. -Jerk the rope and the bell rings weakly. -A waxed floor makes us lose balance. -Madam, this is the best brand of corn. -On the islands the sea breeze is soft and mild. -The play began as soon as we sat down. -This will lead the world to more sound and fury. -Add salt before you fry the egg. -The rush for funds reached its peak Tuesday. -The birch looked stark white and lonesome. -The box is held by a bright red snapper. -To make pure ice, you freeze water. -The first worm gets snapped early. -Jump the fence and hurry up the bank. -Yell and clap as the curtain slides back. -They are men who walk the middle of the road. -Both brothers wear the same size. -In some form or other we need fun. -The prince ordered his head chopped off. -The houses are built of red clay bricks. -Ducks fly north but lack a compass. -Fruit flavors are used in fizz drinks. -These pills do less good than others. -Canned pears lack full flavor. -The dark pot hung in the front closet. -Carry the pail to the wall and spill it there. -The train brought our hero to the big town. -We are sure that one war is enough. -Gray paint stretched for miles around. -The rude laugh filled the empty room. -High seats are best for football fans. -Tea served from the brown jug is tasty. -A dash of pepper spoils beef stew. -A zestful food is the hot-cross bun. -The horse trotted around the field at a brisk pace. -Find the twin who stole the pearl necklace. -Cut the cord that binds the box tightly. -The red tape bound the smuggled food. -Look in the corner to find the tan shirt. -The cold drizzle will halt the bond drive. -Nine men were hired to dig the ruins. -The junk yard had a mouldy smell. -The flint sputtered and lit a pine torch. -Soak the cloth and drown the sharp odor. -The shelves were bare of both jam or crackers. -A joy to every child is the swan boat. -All sat frozen and watched the screen. -A cloud of dust stung his tender eyes. -To reach the end he needs much courage. -Shape the clay gently into block form. -A ridge on a smooth surface is a bump or flaw. -Hedge apples may stain your hands green. -Quench your thirst, then eat the crackers. -Tight curls get limp on rainy days. -The mute muffled the high tones of the horn. -The gold ring fits only a pierced ear. -The old pan was covered with hard fudge. -Watch the log float in the wide river. -The node on the stalk of wheat grew daily. -The heap of fallen leaves was set on fire. -Write fast if you want to finish early. -His shirt was clean but one button was gone. -The barrel of beer was a brew of malt and hops. -Tin cans are absent from store shelves. -Slide the box into that empty space. -The plant grew large and green in the window. -The beam dropped down on the workmen's head. -Pink clouds floated with the breeze. -She danced like a swan, tall and graceful. -The tube was blown and the tire flat and useless. -It is late morning on the old wall clock. -Let's all join as we sing the last chorus. -The last switch cannot be turned off. -The fight will end in just six minutes. -The store walls were lined with colored frocks. -The peace league met to discuss their plans. -The rise to fame of a person takes luck. -Paper is scarce, so write with much care. -The quick fox jumped on the sleeping cat. -The nozzle of the fire hose was bright brass. -Screw the round cap on as tight as needed. -Time brings us many changes. -The purple tie was ten years old. -Men think and plan and sometimes act. -Fill the ink jar with sticky glue. -He smoke a big pipe with strong contents. -We need grain to keep our mules healthy. -Pack the records in a neat thin case. -The crunch of feet in the snow was the only sound. -The copper bowl shone in the sun's rays. -Boards will warp unless kept dry. -The plush chair leaned against the wall. -Glass will clink when struck by metal. -Bathe and relax in the cool green grass. -Nine rows of soldiers stood in line. -The beach is dry and shallow at low tide. -The idea is to sew both edges straight. -The kitten chased the dog down the street. -Pages bound in cloth make a book. -Try to trace the fine lines of the painting. -Women form less than half of the group. -The zones merge in the central part of town. -A gem in the rough needs work to polish. -Code is used when secrets are sent. -Most of the news is easy for us to hear. -He used the lathe to make brass objects. -The vane on top of the pole revolved in the wind. -Mince pie is a dish served to children. -The clan gathered on each dull night. -Let it burn, it gives us warmth and comfort. -A castle built from sand fails to endure. -A child's wit saved the day for us. -Tack the strip of carpet to the worn floor. -Next Tuesday we must vote. -Pour the stew from the pot into the plate. -Each penny shone like new. -The man went to the woods to gather sticks. -The dirt piles were lines along the road. -The logs fell and tumbled into the clear stream. -Just hoist it up and take it away. -A ripe plum is fit for a king's palate. -Our plans right now are hazy. -Brass rings are sold by these natives. -It takes a good trap to capture a bear. -Feed the white mouse some flower seeds. -The thaw came early and freed the stream. -He took the lead and kept it the whole distance. -The key you designed will fit the lock. -Plead to the council to free the poor thief. -Better hash is made of rare beef. -This plank was made for walking on. -The lake sparkled in the red hot sun. -He crawled with care along the ledge. -Tend the sheep while the dog wanders. -It takes a lot of help to finish these. -Mark the spot with a sign painted red. -Take two shares as a fair profit. -The fur of cats goes by many names. -North winds bring colds and fevers. -He asks no person to vouch for him. -Go now and come here later. -A sash of gold silk will trim her dress. -Soap can wash most dirt away. -That move means the game is over. -He wrote down a long list of items. -A siege will crack the strong defense. -Grape juice and water mix well. -Roads are paved with sticky tar. -Fake stones shine but cost little. -The drip of the rain made a pleasant sound. -Smoke poured out of every crack. -Serve the hot rum to the tired heroes. -Much of the story makes good sense. -The sun came up to light the eastern sky. -Heave the line over the port side. -A lathe cuts and trims any wood. -It's a dense crowd in two distinct ways. -His hip struck the knee of the next player. -The stale smell of old beer lingers. -The desk was firm on the shaky floor. -It takes heat to bring out the odor. -Beef is scarcer than some lamb. -Raise the sail and steer the ship northward. -A cone costs five cents on Mondays. -A pod is what peas always grow in. -Jerk the dart from the cork target. -No cement will hold hard wood. -We now have a new base for shipping. -A list of names is carved around the base. -The sheep were led home by a dog. -Three for a dime, the young peddler cried. -The sense of smell is better than that of touch. -No hardship seemed to keep him sad. -Grace makes up for lack of beauty. -Nudge gently but wake her now. -The news struck doubt into restless minds. -Once we stood beside the shore. -A chink in the wall allowed a draft to blow. -Fasten two pins on each side. -A cold dip restores health and zest. -He takes the oath of office each March. -The sand drifts over the sill of the old house. -The point of the steel pen was bent and twisted. -There is a lag between thought and act. -Seed is needed to plant the spring corn. -Draw the chart with heavy black lines. -The boy owed his pal thirty cents. -The chap slipped into the crowd and was lost. -Hats are worn to tea and not to dinner. -The ramp led up to the wide highway. -Beat the dust from the rug onto the lawn. -Say it slowly but make it ring clear. -The straw nest housed five robins. -Screen the porch with woven straw mats. -This horse will nose his way to the finish. -The dry wax protects the deep scratch. -He picked up the dice for a second roll. -These coins will be needed to pay his debt. -The nag pulled the frail cart along. -Twist the valve and release hot steam. -The vamp of the shoe had a gold buckle. -The smell of burned rags itches my nose. -New pants lack cuffs and pockets. -The marsh will freeze when cold enough. -They slice the sausage thin with a knife. -The bloom of the rose lasts a few days. -A gray mare walked before the colt. -Breakfast buns are fine with a hot drink. -Bottles hold four kinds of rum. -The man wore a feather in his felt hat. -He wheeled the bike past the winding road. -Drop the ashes on the worn old rug. -The desk and both chairs were painted tan. -Throw out the used paper cup and plate. -A clean neck means a neat collar. -The couch cover and hall drapes were blue. -The stems of the tall glasses cracked and broke. -The wall phone rang loud and often. -The clothes dried on a thin wooden rack. -Turn on the lantern which gives us light. -The cleat sank deeply into the soft turf. -The bills were mailed promptly on the tenth of the month. -To have is better than to wait and hope. -The price is fair for a good antique clock. -The music played on while they talked. -Dispense with a vest on a day like this. -The bunch of grapes was pressed into wine. -He sent the figs, but kept the ripe cherries. -The hinge on the door creaked with old age. -The screen before the fire kept in the sparks. -Fly by night, and you waste little time. -Thick glasses helped him read the print. -Birth and death mark the limits of life. -The chair looked strong but had no bottom. -The kite flew wildly in the high wind. -A fur muff is stylish once more. -The tin box held priceless stones. -We need an end of all such matter. -The case was puzzling to the old and wise. -The bright lanterns were gay on the dark lawn. -We don't get much money but we have fun. -The youth drove with zest, but little skill. -Five years he lived with a shaggy dog. -A fence cuts through the corner lot. -The way to save money is not to spend much. -Shut the hatch before the waves push it in. -The odor of spring makes young hearts jump. -Crack the walnut with your sharp side teeth. -He offered proof in the form of a large chart. -Send the stuff in a thick paper bag. -A quart of milk is water for the most part. -They told wild tales to frighten him. -The three story house was built of stone. -In the rear of the ground floor was a large passage. -A man in a blue sweater sat at the desk. -Oats are a food eaten by horse and man. -Their eyelids droop for want of sleep. -A sip of tea revives his tired friend. -There are many ways to do these things. -Tuck the sheet under the edge of the mat. -A force equal to that would move the earth. -We like to see clear weather. -The work of the tailor is seen on each side. -Take a chance and win a china doll. -Shake the dust from your shoes, stranger. -She was kind to sick old people. -The square wooden crate was packed to be shipped. -The dusty bench stood by the stone wall. -We dress to suit the weather of most days. -Smile when you say nasty words. -A bowl of rice is free with chicken stew. -The water in this well is a source of good health. -Take shelter in this tent, but keep still. -That guy is the writer of a few banned books. -The little tales they tell are false. -The door was barred, locked, and bolted as well. -Ripe pears are fit for a queen's table. -A big wet stain was on the round carpet. -The kite dipped and swayed, but stayed aloft. -The pleasant hours fly by much too soon. -The room was crowded with a wild mob. -This strong arm shall shield your honor. -She blushed when he gave her a white orchid. -The beetle droned in the hot June sun. -Press the pedal with your left foot. -Neat plans fail without luck. -The black trunk fell from the landing. -The bank pressed for payment of the debt. -The theft of the pearl pin was kept secret. -Shake hands with this friendly child. -The vast space stretched into the far distance. -A rich farm is rare in this sandy waste. -His wide grin earned many friends. -Flax makes a fine brand of paper. -Hurdle the pit with the aid of a long pole. -A strong bid may scare your partner stiff. -Even a just cause needs power to win. -Peep under the tent and see the clowns. -The leaf drifts along with a slow spin. -Cheap clothes are flashy but don't last. -A thing of small note can cause despair. -Flood the mails with requests for this book. -A thick coat of black paint covered all. -The pencil was cut to be sharp at both ends. -Those last words were a strong statement. -He wrote his name boldly at the top of the sheet. -Dill pickles are sour but taste fine. -Down that road is the way to the grain farmer. -Either mud or dust are found at all times. -The best method is to fix it in place with clips. -If you mumble your speech will be lost. -At night the alarm roused him from a deep sleep. -Read just what the meter says. -Fill your pack with bright trinkets for the poor. -The small red neon lamp went out. -Clams are small, round, soft, and tasty. -The fan whirled its round blades softly. -The line where the edges join was clean. -Breathe deep and smell the piny air. -It matters not if he reads these words or those. -A brown leather bag hung from its strap. -A toad and a frog are hard to tell apart. -A white silk jacket goes with any shoes. -A break in the dam almost caused a flood. -Paint the sockets in the wall dull green. -The child crawled into the dense grass. -Bribes fail where honest men work. -Trample the spark, else the flames will spread. -The hilt of the sword was carved with fine designs. -A round hole was drilled through the thin board. -Footprints showed the path he took up the beach. -She was waiting at my front lawn. -A vent near the edge brought in fresh air. -Prod the old mule with a crooked stick. -It is a band of steel three inches wide. -The pipe ran almost the length of the ditch. -It was hidden from sight by a mass of leaves and shrubs. -The weight of the package was seen on the high scale. -Wake and rise, and step into the green outdoors. -The green light in the brown box flickered. -The brass tube circled the high wall. -The lobes of her ears were pierced to hold rings. -Hold the hammer near the end to drive the nail. -Next Sunday is the twelfth of the month. -Every word and phrase he speaks is true. -He put his last cartridge into the gun and fired. -They took their kids from the public school. -Drive the screw straight into the wood. -Keep the hatch tight and the watch constant. -Sever the twine with a quick snip of the knife. -Paper will dry out when wet. -Slide the catch back and open the desk. -Help the weak to preserve their strength. -A sullen smile gets few friends. -Stop whistling and watch the boys march. -Jerk the cord, and out tumbles the gold. -Slide the tray across the glass top. -The cloud moved in a stately way and was gone. -Light maple makes for a swell room. -Set the piece here and say nothing. -Dull stories make her laugh. -A stiff cord will do to fasten your shoe. -Get the trust fund to the bank early. -Choose between the high road and the low. -A plea for funds seems to come again. -He lent his coat to the tall gaunt stranger. -There is a strong chance it will happen once more. -The duke left the park in a silver coach. -Greet the new guests and leave quickly. -When the frost has come it is time for turkey. -Sweet words work better than fierce. -A thin stripe runs down the middle. -A six comes up more often than a ten. -Lush fern grow on the lofty rocks. -The ram scared the school children off. -The team with the best timing looks good. -The farmer swapped his horse for a brown ox. -Sit on the perch and tell the others what to do. -A steep trail is painful for our feet. -The early phase of life moves fast. -Green moss grows on the northern side. -Tea in thin china has a sweet taste. -Pitch the straw through the door of the stable. -The latch on the back gate needed a nail. -The goose was brought straight from the old market. -The sink is the thing in which we pile dishes. -A whiff of it will cure the most stubborn cold. -The facts don't always show who is right. -She flaps her cape as she parades the street. -The loss of the cruiser was a blow to the fleet. -Loop the braid to the left and then over. -Plead with the lawyer to drop the lost cause. -Calves thrive on tender spring grass. -Post no bills on this office wall. -Tear a thin sheet from the yellow pad. -A cruise in warm waters in a sleek yacht is fun. -A streak of color ran down the left edge. -It was done before the boy could see it. -Crouch before you jump or miss the mark. -Pack the kits and don't forget the salt. -The square peg will settle in the round hole. -Fine soap saves tender skin. -Poached eggs and tea must suffice. -Bad nerves are jangled by a door slam. -Ship maps are different from those for planes. -Dimes showered down from all sides. -They sang the same tunes at each party. -The sky in the west is tinged with orange red. -The pods of peas ferment in bare fields. -The horse balked and threw the tall rider. -The hitch between the horse and cart broke. -Pile the coal high in the shed corner. -A gold vase is both rare and costly. -The knife was hung inside its bright sheath. -The rarest spice comes from the far East. -The roof should be tilted at a sharp slant. -A smatter of French is worse than none. -The mule trod the treadmill day and night. -The aim of the contest is to raise a great fund. -To send it now in large amounts is bad. -There is a fine hard tang in salty air. -Cod is the main business of the north shore. -The slab was hewn from heavy blocks of slate. -Dunk the stale biscuits into strong drink. -Hang tinsel from both branches. -Cap the jar with a tight brass cover. -The poor boy missed the boat again. -Be sure to set the lamp firmly in the hole. -Pick a card and slip it under the pack. -A round mat will cover the dull spot. -The first part of the plan needs changing. -A good book informs of what we ought to know. -The mail comes in three batches per day. -You cannot brew tea in a cold pot. -Dots of light betrayed the black cat. -Put the chart on the mantel and tack it down. -The night shift men rate extra pay. -The red paper brightened the dim stage. -See the player scoot to third base. -Slide the bill between the two leaves. -Many hands help get the job done. -We don't like to admit our small faults. -No doubt about the way the wind blows. -Dig deep in the earth for pirate's gold. -The steady drip is worse than a drenching rain. -A flat pack takes less luggage space. -Green ice frosted the punch bowl. -A stuffed chair slipped from the moving van. -The stitch will serve but needs to be shortened. -A thin book fits in the side pocket. -The gloss on top made it unfit to read. -The hail pattered on the burnt brown grass. -Seven seals were stamped on great sheets. -Our troops are set to strike heavy blows. -The store was jammed before the sale could start. -It was a bad error on the part of the new judge. -One step more and the board will collapse. -Take the match and strike it against your shoe. -The pot boiled, but the contents failed to jell. -The baby puts his right foot in his mouth. -The bombs left most of the town in ruins. -Stop and stare at the hard working man. -The streets are narrow and full of sharp turns. -The pup jerked the leash as he saw a feline shape. -Open your book to the first page. -Fish evade the net and swim off. -Dip the pail once and let it settle. -Will you please answer that phone. -The big red apple fell to the ground. -The curtain rose and the show was on. -The young prince became heir to the throne. -He sent the boy on a short errand. -Leave now and you will arrive on time. -The corner store was robbed last night. -A gold ring will please most any girl. -The long journey home took a year. -She saw a cat in the neighbor's house. -A pink shell was found on the sandy beach. -Small children came to see him. -The grass and bushes were wet with dew. -The blind man counted his old coins. -A severe storm tore down the barn. -She called his name many times. -When you hear the bell, come quickly. \ No newline at end of file diff --git a/system/config/languages.json b/system/config/languages.json deleted file mode 100644 index 3b780e8..0000000 --- a/system/config/languages.json +++ /dev/null @@ -1,18 +0,0 @@ -{ - "Arabic": "ar", - "Chinese": "zh-cn", - "Czech": "cs", - "Dutch": "nl", - "English": "en", - "French": "fr", - "German": "de", - "Hungarian": "hu", - "Italian": "it", - "Japanese": "ja", - "Korean": "ko", - "Polish": "pl", - "Portuguese": "pt", - "Russian": "ru", - "Spanish": "es", - "Turkish": "tr" -} \ No newline at end of file diff --git a/system/favicon.ico b/system/favicon.ico deleted file mode 100644 index 1e0d7ff..0000000 Binary files a/system/favicon.ico and /dev/null differ diff --git a/system/ft_tokenizer/tokenizer.py b/system/ft_tokenizer/tokenizer.py deleted file mode 100644 index 4642cc1..0000000 --- a/system/ft_tokenizer/tokenizer.py +++ /dev/null @@ -1,869 +0,0 @@ -import os -import re -import textwrap -from functools import cached_property - -import pypinyin -import torch -from hangul_romanize import Transliter -from hangul_romanize.rule import academic -from num2words import num2words -from spacy.lang.ar import Arabic -from spacy.lang.en import English -from spacy.lang.es import Spanish -from spacy.lang.ja import Japanese -from spacy.lang.zh import Chinese -from tokenizers import Tokenizer - -from TTS.tts.layers.xtts.zh_num2words import TextNorm as zh_num2words - - -def get_spacy_lang(lang): - if lang == "zh": - return Chinese() - elif lang == "ja": - return Japanese() - elif lang == "ar": - return Arabic() - elif lang == "es": - return Spanish() - else: - # For most languages, Enlish does the job - return English() - - -def split_sentence(text, lang, text_split_length=250): - """Preprocess the input text""" - text_splits = [] - if text_split_length is not None and len(text) >= text_split_length: - text_splits.append("") - nlp = get_spacy_lang(lang) - nlp.add_pipe("sentencizer") - doc = nlp(text) - for sentence in doc.sents: - if len(text_splits[-1]) + len(str(sentence)) <= text_split_length: - # if the last sentence + the current sentence is less than the text_split_length - # then add the current sentence to the last sentence - text_splits[-1] += " " + str(sentence) - text_splits[-1] = text_splits[-1].lstrip() - elif len(str(sentence)) > text_split_length: - # if the current sentence is greater than the text_split_length - for line in textwrap.wrap( - str(sentence), - width=text_split_length, - drop_whitespace=True, - break_on_hyphens=False, - tabsize=1, - ): - text_splits.append(str(line)) - else: - text_splits.append(str(sentence)) - - if len(text_splits) > 1: - if text_splits[0] == "": - del text_splits[0] - else: - text_splits = [text.lstrip()] - - return text_splits - - -_whitespace_re = re.compile(r"\s+") - -# List of (regular expression, replacement) pairs for abbreviations: -_abbreviations = { - "en": [ - (re.compile("\\b%s\\." % x[0], re.IGNORECASE), x[1]) - for x in [ - ("mrs", "misess"), - ("mr", "mister"), - ("dr", "doctor"), - ("st", "saint"), - ("co", "company"), - ("jr", "junior"), - ("maj", "major"), - ("gen", "general"), - ("drs", "doctors"), - ("rev", "reverend"), - ("lt", "lieutenant"), - ("hon", "honorable"), - ("sgt", "sergeant"), - ("capt", "captain"), - ("esq", "esquire"), - ("ltd", "limited"), - ("col", "colonel"), - ("ft", "fort"), - ] - ], - "es": [ - (re.compile("\\b%s\\." % x[0], re.IGNORECASE), x[1]) - for x in [ - ("sra", "señora"), - ("sr", "señor"), - ("dr", "doctor"), - ("dra", "doctora"), - ("st", "santo"), - ("co", "compañía"), - ("jr", "junior"), - ("ltd", "limitada"), - ] - ], - "fr": [ - (re.compile("\\b%s\\." % x[0], re.IGNORECASE), x[1]) - for x in [ - ("mme", "madame"), - ("mr", "monsieur"), - ("dr", "docteur"), - ("st", "saint"), - ("co", "compagnie"), - ("jr", "junior"), - ("ltd", "limitée"), - ] - ], - "de": [ - (re.compile("\\b%s\\." % x[0], re.IGNORECASE), x[1]) - for x in [ - ("fr", "frau"), - ("dr", "doktor"), - ("st", "sankt"), - ("co", "firma"), - ("jr", "junior"), - ] - ], - "pt": [ - (re.compile("\\b%s\\." % x[0], re.IGNORECASE), x[1]) - for x in [ - ("sra", "senhora"), - ("sr", "senhor"), - ("dr", "doutor"), - ("dra", "doutora"), - ("st", "santo"), - ("co", "companhia"), - ("jr", "júnior"), - ("ltd", "limitada"), - ] - ], - "it": [ - (re.compile("\\b%s\\." % x[0], re.IGNORECASE), x[1]) - for x in [ - # ("sig.ra", "signora"), - ("sig", "signore"), - ("dr", "dottore"), - ("st", "santo"), - ("co", "compagnia"), - ("jr", "junior"), - ("ltd", "limitata"), - ] - ], - "pl": [ - (re.compile("\\b%s\\." % x[0], re.IGNORECASE), x[1]) - for x in [ - ("p", "pani"), - ("m", "pan"), - ("dr", "doktor"), - ("sw", "święty"), - ("jr", "junior"), - ] - ], - "ar": [ - (re.compile("\\b%s\\." % x[0], re.IGNORECASE), x[1]) - for x in [ - # There are not many common abbreviations in Arabic as in English. - ] - ], - "zh": [ - (re.compile("\\b%s\\." % x[0], re.IGNORECASE), x[1]) - for x in [ - # Chinese doesn't typically use abbreviations in the same way as Latin-based scripts. - ] - ], - "cs": [ - (re.compile("\\b%s\\." % x[0], re.IGNORECASE), x[1]) - for x in [ - ("dr", "doktor"), # doctor - ("ing", "inženýr"), # engineer - ("p", "pan"), # Could also map to pani for woman but no easy way to do it - # Other abbreviations would be specialized and not as common. - ] - ], - "ru": [ - (re.compile("\\b%s\\b" % x[0], re.IGNORECASE), x[1]) - for x in [ - ("г-жа", "госпожа"), # Mrs. - ("г-н", "господин"), # Mr. - ("д-р", "доктор"), # doctor - # Other abbreviations are less common or specialized. - ] - ], - "nl": [ - (re.compile("\\b%s\\." % x[0], re.IGNORECASE), x[1]) - for x in [ - ("dhr", "de heer"), # Mr. - ("mevr", "mevrouw"), # Mrs. - ("dr", "dokter"), # doctor - ("jhr", "jonkheer"), # young lord or nobleman - # Dutch uses more abbreviations, but these are the most common ones. - ] - ], - "tr": [ - (re.compile("\\b%s\\." % x[0], re.IGNORECASE), x[1]) - for x in [ - ("b", "bay"), # Mr. - ("byk", "büyük"), # büyük - ("dr", "doktor"), # doctor - # Add other Turkish abbreviations here if needed. - ] - ], - "hu": [ - (re.compile("\\b%s\\." % x[0], re.IGNORECASE), x[1]) - for x in [ - ("dr", "doktor"), # doctor - ("b", "bácsi"), # Mr. - ("nőv", "nővér"), # nurse - # Add other Hungarian abbreviations here if needed. - ] - ], - "ko": [ - (re.compile("\\b%s\\." % x[0], re.IGNORECASE), x[1]) - for x in [ - # Korean doesn't typically use abbreviations in the same way as Latin-based scripts. - ] - ], - "ja": [ - (re.compile("\\b%s\\b" % x[0]), x[1]) - for x in [ - ("氏", "さん"), # Mr. - ("夫人", "おんなのひと"), # Mrs. - ("博士", "はかせ"), # Doctor or PhD - ("株", "株式会社"), # Corporation - ("有", "有限会社"), # Limited company - ("大学", "だいがく"), # University - ("先生", "せんせい"), # Teacher/Professor/Master - ("君", "くん") # Used at the end of boys' names to express familiarity or affection. - ] - ], -} - - -def expand_abbreviations_multilingual(text, lang="en"): - for regex, replacement in _abbreviations[lang]: - text = re.sub(regex, replacement, text) - return text - - -_symbols_multilingual = { - "en": [ - (re.compile(r"%s" % re.escape(x[0]), re.IGNORECASE), x[1]) - for x in [ - ("&", " and "), - ("@", " at "), - ("%", " percent "), - ("#", " hash "), - ("$", " dollar "), - ("£", " pound "), - ("°", " degree "), - ] - ], - "es": [ - (re.compile(r"%s" % re.escape(x[0]), re.IGNORECASE), x[1]) - for x in [ - ("&", " y "), - ("@", " arroba "), - ("%", " por ciento "), - ("#", " numeral "), - ("$", " dolar "), - ("£", " libra "), - ("°", " grados "), - ] - ], - "fr": [ - (re.compile(r"%s" % re.escape(x[0]), re.IGNORECASE), x[1]) - for x in [ - ("&", " et "), - ("@", " arobase "), - ("%", " pour cent "), - ("#", " dièse "), - ("$", " dollar "), - ("£", " livre "), - ("°", " degrés "), - ] - ], - "de": [ - (re.compile(r"%s" % re.escape(x[0]), re.IGNORECASE), x[1]) - for x in [ - ("&", " und "), - ("@", " at "), - ("%", " prozent "), - ("#", " raute "), - ("$", " dollar "), - ("£", " pfund "), - ("°", " grad "), - ] - ], - "pt": [ - (re.compile(r"%s" % re.escape(x[0]), re.IGNORECASE), x[1]) - for x in [ - ("&", " e "), - ("@", " arroba "), - ("%", " por cento "), - ("#", " cardinal "), - ("$", " dólar "), - ("£", " libra "), - ("°", " graus "), - ] - ], - "it": [ - (re.compile(r"%s" % re.escape(x[0]), re.IGNORECASE), x[1]) - for x in [ - ("&", " e "), - ("@", " chiocciola "), - ("%", " per cento "), - ("#", " cancelletto "), - ("$", " dollaro "), - ("£", " sterlina "), - ("°", " gradi "), - ] - ], - "pl": [ - (re.compile(r"%s" % re.escape(x[0]), re.IGNORECASE), x[1]) - for x in [ - ("&", " i "), - ("@", " małpa "), - ("%", " procent "), - ("#", " krzyżyk "), - ("$", " dolar "), - ("£", " funt "), - ("°", " stopnie "), - ] - ], - "ar": [ - # Arabic - (re.compile(r"%s" % re.escape(x[0]), re.IGNORECASE), x[1]) - for x in [ - ("&", " و "), - ("@", " على "), - ("%", " في المئة "), - ("#", " رقم "), - ("$", " دولار "), - ("£", " جنيه "), - ("°", " درجة "), - ] - ], - "zh": [ - # Chinese - (re.compile(r"%s" % re.escape(x[0]), re.IGNORECASE), x[1]) - for x in [ - ("&", " 和 "), - ("@", " 在 "), - ("%", " 百分之 "), - ("#", " 号 "), - ("$", " 美元 "), - ("£", " 英镑 "), - ("°", " 度 "), - ] - ], - "cs": [ - # Czech - (re.compile(r"%s" % re.escape(x[0]), re.IGNORECASE), x[1]) - for x in [ - ("&", " a "), - ("@", " na "), - ("%", " procento "), - ("#", " křížek "), - ("$", " dolar "), - ("£", " libra "), - ("°", " stupně "), - ] - ], - "ru": [ - # Russian - (re.compile(r"%s" % re.escape(x[0]), re.IGNORECASE), x[1]) - for x in [ - ("&", " и "), - ("@", " собака "), - ("%", " процентов "), - ("#", " номер "), - ("$", " доллар "), - ("£", " фунт "), - ("°", " градус "), - ] - ], - "nl": [ - # Dutch - (re.compile(r"%s" % re.escape(x[0]), re.IGNORECASE), x[1]) - for x in [ - ("&", " en "), - ("@", " bij "), - ("%", " procent "), - ("#", " hekje "), - ("$", " dollar "), - ("£", " pond "), - ("°", " graden "), - ] - ], - "tr": [ - (re.compile(r"%s" % re.escape(x[0]), re.IGNORECASE), x[1]) - for x in [ - ("&", " ve "), - ("@", " at "), - ("%", " yüzde "), - ("#", " diyez "), - ("$", " dolar "), - ("£", " sterlin "), - ("°", " derece "), - ] - ], - "hu": [ - (re.compile(r"%s" % re.escape(x[0]), re.IGNORECASE), x[1]) - for x in [ - ("&", " és "), - ("@", " kukac "), - ("%", " százalék "), - ("#", " kettőskereszt "), - ("$", " dollár "), - ("£", " font "), - ("°", " fok "), - ] - ], - "ko": [ - # Korean - (re.compile(r"%s" % re.escape(x[0]), re.IGNORECASE), x[1]) - for x in [ - ("&", " 그리고 "), - ("@", " 에 "), - ("%", " 퍼센트 "), - ("#", " 번호 "), - ("$", " 달러 "), - ("£", " 파운드 "), - ("°", " 도 "), - ] - ], - "ja": [ - (re.compile(r"%s" % re.escape(x[0])), x[1]) - for x in [ - ("&", " と "), - ("@", " アットマーク "), - ("%", " パーセント "), - ("#", " ナンバー "), - ("$", " ドル "), - ("£", " ポンド "), - ("°", " 度"), - ] - ], -} - - -def expand_symbols_multilingual(text, lang="en"): - for regex, replacement in _symbols_multilingual[lang]: - text = re.sub(regex, replacement, text) - text = text.replace(" ", " ") # Ensure there are no double spaces - return text.strip() - - -_ordinal_re = { - "en": re.compile(r"([0-9]+)(st|nd|rd|th)"), - "es": re.compile(r"([0-9]+)(º|ª|er|o|a|os|as)"), - "fr": re.compile(r"([0-9]+)(º|ª|er|re|e|ème)"), - "de": re.compile(r"([0-9]+)(st|nd|rd|th|º|ª|\.(?=\s|$))"), - "pt": re.compile(r"([0-9]+)(º|ª|o|a|os|as)"), - "it": re.compile(r"([0-9]+)(º|°|ª|o|a|i|e)"), - "pl": re.compile(r"([0-9]+)(º|ª|st|nd|rd|th)"), - "ar": re.compile(r"([0-9]+)(ون|ين|ث|ر|ى)"), - "cs": re.compile(r"([0-9]+)\.(?=\s|$)"), # In Czech, a dot is often used after the number to indicate ordinals. - "ru": re.compile(r"([0-9]+)(-й|-я|-е|-ое|-ье|-го)"), - "nl": re.compile(r"([0-9]+)(de|ste|e)"), - "tr": re.compile(r"([0-9]+)(\.|inci|nci|uncu|üncü|\.)"), - "hu": re.compile(r"([0-9]+)(\.|adik|edik|odik|edik|ödik|ödike|ik)"), - "ko": re.compile(r"([0-9]+)(번째|번|차|째)"), - "ja": re.compile(r"([0-9]+)(番|回|つ|目|等|位)") -} -_number_re = re.compile(r"[0-9]+") -_currency_re = { - "USD": re.compile(r"((\$[0-9\.\,]*[0-9]+)|([0-9\.\,]*[0-9]+\$))"), - "GBP": re.compile(r"((£[0-9\.\,]*[0-9]+)|([0-9\.\,]*[0-9]+£))"), - "EUR": re.compile(r"(([0-9\.\,]*[0-9]+€)|((€[0-9\.\,]*[0-9]+)))"), -} - -_comma_number_re = re.compile(r"\b\d{1,3}(,\d{3})*(\.\d+)?\b") -_dot_number_re = re.compile(r"\b\d{1,3}(.\d{3})*(\,\d+)?\b") -_decimal_number_re = re.compile(r"([0-9]+[.,][0-9]+)") - - -def _remove_commas(m): - text = m.group(0) - if "," in text: - text = text.replace(",", "") - return text - - -def _remove_dots(m): - text = m.group(0) - if "." in text: - text = text.replace(".", "") - return text - - -def _expand_decimal_point(m, lang="en"): - amount = m.group(1).replace(",", ".") - return num2words(float(amount), lang=lang if lang != "cs" else "cz") - - -def _expand_currency(m, lang="en", currency="USD"): - amount = float((re.sub(r"[^\d.]", "", m.group(0).replace(",", ".")))) - full_amount = num2words(amount, to="currency", currency=currency, lang=lang if lang != "cs" else "cz") - - and_equivalents = { - "en": ", ", - "es": " con ", - "fr": " et ", - "de": " und ", - "pt": " e ", - "it": " e ", - "pl": ", ", - "cs": ", ", - "ru": ", ", - "nl": ", ", - "ar": ", ", - "tr": ", ", - "hu": ", ", - "ko": ", ", - } - - if amount.is_integer(): - last_and = full_amount.rfind(and_equivalents[lang]) - if last_and != -1: - full_amount = full_amount[:last_and] - - return full_amount - - -def _expand_ordinal(m, lang="en"): - return num2words(int(m.group(1)), ordinal=True, lang=lang if lang != "cs" else "cz") - - -def _expand_number(m, lang="en"): - return num2words(int(m.group(0)), lang=lang if lang != "cs" else "cz") - - -def expand_numbers_multilingual(text, lang="en"): - if lang == "zh": - text = zh_num2words()(text) - else: - if lang in ["en", "ru"]: - text = re.sub(_comma_number_re, _remove_commas, text) - else: - text = re.sub(_dot_number_re, _remove_dots, text) - try: - text = re.sub(_currency_re["GBP"], lambda m: _expand_currency(m, lang, "GBP"), text) - text = re.sub(_currency_re["USD"], lambda m: _expand_currency(m, lang, "USD"), text) - text = re.sub(_currency_re["EUR"], lambda m: _expand_currency(m, lang, "EUR"), text) - except: - pass - if lang != "tr": - text = re.sub(_decimal_number_re, lambda m: _expand_decimal_point(m, lang), text) - text = re.sub(_ordinal_re[lang], lambda m: _expand_ordinal(m, lang), text) - text = re.sub(_number_re, lambda m: _expand_number(m, lang), text) - return text - - -def lowercase(text): - return text.lower() - - -def collapse_whitespace(text): - return re.sub(_whitespace_re, " ", text) - - -def multilingual_cleaners(text, lang): - text = text.replace('"', "") - if lang == "tr": - text = text.replace("İ", "i") - text = text.replace("Ö", "ö") - text = text.replace("Ü", "ü") - text = lowercase(text) - text = expand_numbers_multilingual(text, lang) - text = expand_abbreviations_multilingual(text, lang) - text = expand_symbols_multilingual(text, lang=lang) - text = collapse_whitespace(text) - return text - - -def basic_cleaners(text): - """Basic pipeline that lowercases and collapses whitespace without transliteration.""" - text = lowercase(text) - text = collapse_whitespace(text) - return text - - -def chinese_transliterate(text): - return "".join( - [p[0] for p in pypinyin.pinyin(text, style=pypinyin.Style.TONE3, heteronym=False, neutral_tone_with_five=True)] - ) - - -def japanese_cleaners(text, katsu): - text = katsu.romaji(text) - text = lowercase(text) - return text - - -def korean_transliterate(text): - r = Transliter(academic) - return r.translit(text) - - -DEFAULT_VOCAB_FILE = os.path.join(os.path.dirname(os.path.realpath(__file__)), "../data/tokenizer.json") - - -class VoiceBpeTokenizer: - def __init__(self, vocab_file=None): - self.tokenizer = None - if vocab_file is not None: - self.tokenizer = Tokenizer.from_file(vocab_file) - self.char_limits = { - "en": 250, - "de": 253, - "fr": 273, - "es": 239, - "it": 213, - "pt": 203, - "pl": 224, - "zh": 82, - "ar": 166, - "cs": 186, - "ru": 182, - "nl": 251, - "tr": 226, - "ja": 71, - "hu": 224, - "ko": 95, - } - - @cached_property - def katsu(self): - import cutlet - - return cutlet.Cutlet() - - def check_input_length(self, txt, lang): - lang = lang.split("-")[0] # remove the region - limit = self.char_limits.get(lang, 250) - if len(txt) > limit: - print( - f"[!] Warning: The text length exceeds the character limit of {limit} for language '{lang}', this might cause truncated audio." - ) - - def preprocess_text(self, txt, lang): - if lang in {"ar", "cs", "de", "en", "es", "fr", "hu", "it", "nl", "pl", "pt", "ru", "tr", "zh", "ko"}: - txt = multilingual_cleaners(txt, lang) - if lang == "zh": - txt = chinese_transliterate(txt) - if lang == "ko": - txt = korean_transliterate(txt) - elif lang == "ja": - txt = japanese_cleaners(txt, self.katsu) - elif lang == "hi": - # @manmay will implement this - txt = basic_cleaners(txt) - else: - raise NotImplementedError(f"Language '{lang}' is not supported.") - return txt - - def encode(self, txt, lang): - lang = lang.split("-")[0] # remove the region - self.check_input_length(txt, lang) - txt = self.preprocess_text(txt, lang) - lang = "zh-cn" if lang == "zh" else lang - txt = f"[{lang}]{txt}" - txt = txt.replace(" ", "[SPACE]") - return self.tokenizer.encode(txt).ids - - def decode(self, seq): - if isinstance(seq, torch.Tensor): - seq = seq.cpu().numpy() - txt = self.tokenizer.decode(seq, skip_special_tokens=False).replace(" ", "") - txt = txt.replace("[SPACE]", " ") - txt = txt.replace("[STOP]", "") - txt = txt.replace("[UNK]", "") - return txt - - def __len__(self): - return self.tokenizer.get_vocab_size() - - def get_number_tokens(self): - return max(self.tokenizer.get_vocab().values()) + 1 - - -def test_expand_numbers_multilingual(): - test_cases = [ - # English - ("In 12.5 seconds.", "In twelve point five seconds.", "en"), - ("There were 50 soldiers.", "There were fifty soldiers.", "en"), - ("This is a 1st test", "This is a first test", "en"), - ("That will be $20 sir.", "That will be twenty dollars sir.", "en"), - ("That will be 20€ sir.", "That will be twenty euro sir.", "en"), - ("That will be 20.15€ sir.", "That will be twenty euro, fifteen cents sir.", "en"), - ("That's 100,000.5.", "That's one hundred thousand point five.", "en"), - # French - ("En 12,5 secondes.", "En douze virgule cinq secondes.", "fr"), - ("Il y avait 50 soldats.", "Il y avait cinquante soldats.", "fr"), - ("Ceci est un 1er test", "Ceci est un premier test", "fr"), - ("Cela vous fera $20 monsieur.", "Cela vous fera vingt dollars monsieur.", "fr"), - ("Cela vous fera 20€ monsieur.", "Cela vous fera vingt euros monsieur.", "fr"), - ("Cela vous fera 20,15€ monsieur.", "Cela vous fera vingt euros et quinze centimes monsieur.", "fr"), - ("Ce sera 100.000,5.", "Ce sera cent mille virgule cinq.", "fr"), - # German - ("In 12,5 Sekunden.", "In zwölf Komma fünf Sekunden.", "de"), - ("Es gab 50 Soldaten.", "Es gab fünfzig Soldaten.", "de"), - ("Dies ist ein 1. Test", "Dies ist ein erste Test", "de"), # Issue with gender - ("Das macht $20 Herr.", "Das macht zwanzig Dollar Herr.", "de"), - ("Das macht 20€ Herr.", "Das macht zwanzig Euro Herr.", "de"), - ("Das macht 20,15€ Herr.", "Das macht zwanzig Euro und fünfzehn Cent Herr.", "de"), - # Spanish - ("En 12,5 segundos.", "En doce punto cinco segundos.", "es"), - ("Había 50 soldados.", "Había cincuenta soldados.", "es"), - ("Este es un 1er test", "Este es un primero test", "es"), - ("Eso le costará $20 señor.", "Eso le costará veinte dólares señor.", "es"), - ("Eso le costará 20€ señor.", "Eso le costará veinte euros señor.", "es"), - ("Eso le costará 20,15€ señor.", "Eso le costará veinte euros con quince céntimos señor.", "es"), - # Italian - ("In 12,5 secondi.", "In dodici virgola cinque secondi.", "it"), - ("C'erano 50 soldati.", "C'erano cinquanta soldati.", "it"), - ("Questo è un 1° test", "Questo è un primo test", "it"), - ("Ti costerà $20 signore.", "Ti costerà venti dollari signore.", "it"), - ("Ti costerà 20€ signore.", "Ti costerà venti euro signore.", "it"), - ("Ti costerà 20,15€ signore.", "Ti costerà venti euro e quindici centesimi signore.", "it"), - # Portuguese - ("Em 12,5 segundos.", "Em doze vírgula cinco segundos.", "pt"), - ("Havia 50 soldados.", "Havia cinquenta soldados.", "pt"), - ("Este é um 1º teste", "Este é um primeiro teste", "pt"), - ("Isso custará $20 senhor.", "Isso custará vinte dólares senhor.", "pt"), - ("Isso custará 20€ senhor.", "Isso custará vinte euros senhor.", "pt"), - ( - "Isso custará 20,15€ senhor.", - "Isso custará vinte euros e quinze cêntimos senhor.", - "pt", - ), # "cêntimos" should be "centavos" num2words issue - # Polish - ("W 12,5 sekundy.", "W dwanaście przecinek pięć sekundy.", "pl"), - ("Było 50 żołnierzy.", "Było pięćdziesiąt żołnierzy.", "pl"), - ("To będzie kosztować 20€ panie.", "To będzie kosztować dwadzieścia euro panie.", "pl"), - ("To będzie kosztować 20,15€ panie.", "To będzie kosztować dwadzieścia euro, piętnaście centów panie.", "pl"), - # Arabic - ("في الـ 12,5 ثانية.", "في الـ اثنا عشر , خمسون ثانية.", "ar"), - ("كان هناك 50 جنديًا.", "كان هناك خمسون جنديًا.", "ar"), - # ("ستكون النتيجة $20 يا سيد.", 'ستكون النتيجة عشرون دولار يا سيد.', 'ar'), # $ and € are mising from num2words - # ("ستكون النتيجة 20€ يا سيد.", 'ستكون النتيجة عشرون يورو يا سيد.', 'ar'), - # Czech - ("Za 12,5 vteřiny.", "Za dvanáct celá pět vteřiny.", "cs"), - ("Bylo tam 50 vojáků.", "Bylo tam padesát vojáků.", "cs"), - ("To bude stát 20€ pane.", "To bude stát dvacet euro pane.", "cs"), - ("To bude 20.15€ pane.", "To bude dvacet euro, patnáct centů pane.", "cs"), - # Russian - ("Через 12.5 секунды.", "Через двенадцать запятая пять секунды.", "ru"), - ("Там было 50 солдат.", "Там было пятьдесят солдат.", "ru"), - ("Это будет 20.15€ сэр.", "Это будет двадцать евро, пятнадцать центов сэр.", "ru"), - ("Это будет стоить 20€ господин.", "Это будет стоить двадцать евро господин.", "ru"), - # Dutch - ("In 12,5 seconden.", "In twaalf komma vijf seconden.", "nl"), - ("Er waren 50 soldaten.", "Er waren vijftig soldaten.", "nl"), - ("Dat wordt dan $20 meneer.", "Dat wordt dan twintig dollar meneer.", "nl"), - ("Dat wordt dan 20€ meneer.", "Dat wordt dan twintig euro meneer.", "nl"), - # Chinese (Simplified) - ("在12.5秒内", "在十二点五秒内", "zh"), - ("有50名士兵", "有五十名士兵", "zh"), - # ("那将是$20先生", '那将是二十美元先生', 'zh'), currency doesn't work - # ("那将是20€先生", '那将是二十欧元先生', 'zh'), - # Turkish - # ("12,5 saniye içinde.", 'On iki virgül beş saniye içinde.', 'tr'), # decimal doesn't work for TR - ("50 asker vardı.", "elli asker vardı.", "tr"), - ("Bu 1. test", "Bu birinci test", "tr"), - # ("Bu 100.000,5.", 'Bu yüz bin virgül beş.', 'tr'), - # Hungarian - ("12,5 másodperc alatt.", "tizenkettő egész öt tized másodperc alatt.", "hu"), - ("50 katona volt.", "ötven katona volt.", "hu"), - ("Ez az 1. teszt", "Ez az első teszt", "hu"), - # Korean - ("12.5 초 안에.", "십이 점 다섯 초 안에.", "ko"), - ("50 명의 병사가 있었다.", "오십 명의 병사가 있었다.", "ko"), - ("이것은 1 번째 테스트입니다", "이것은 첫 번째 테스트입니다", "ko"), - ] - for a, b, lang in test_cases: - out = expand_numbers_multilingual(a, lang=lang) - assert out == b, f"'{out}' vs '{b}'" - - -def test_abbreviations_multilingual(): - test_cases = [ - # English - ("Hello Mr. Smith.", "Hello mister Smith.", "en"), - ("Dr. Jones is here.", "doctor Jones is here.", "en"), - # Spanish - ("Hola Sr. Garcia.", "Hola señor Garcia.", "es"), - ("La Dra. Martinez es muy buena.", "La doctora Martinez es muy buena.", "es"), - # French - ("Bonjour Mr. Dupond.", "Bonjour monsieur Dupond.", "fr"), - ("Mme. Moreau est absente aujourd'hui.", "madame Moreau est absente aujourd'hui.", "fr"), - # German - ("Frau Dr. Müller ist sehr klug.", "Frau doktor Müller ist sehr klug.", "de"), - # Portuguese - ("Olá Sr. Silva.", "Olá senhor Silva.", "pt"), - ("Dra. Costa, você está disponível?", "doutora Costa, você está disponível?", "pt"), - # Italian - ("Buongiorno, Sig. Rossi.", "Buongiorno, signore Rossi.", "it"), - # ("Sig.ra Bianchi, posso aiutarti?", 'signora Bianchi, posso aiutarti?', 'it'), # Issue with matching that pattern - # Polish - ("Dzień dobry, P. Kowalski.", "Dzień dobry, pani Kowalski.", "pl"), - ("M. Nowak, czy mogę zadać pytanie?", "pan Nowak, czy mogę zadać pytanie?", "pl"), - # Czech - ("P. Novák", "pan Novák", "cs"), - ("Dr. Vojtěch", "doktor Vojtěch", "cs"), - # Dutch - ("Dhr. Jansen", "de heer Jansen", "nl"), - ("Mevr. de Vries", "mevrouw de Vries", "nl"), - # Russian - ("Здравствуйте Г-н Иванов.", "Здравствуйте господин Иванов.", "ru"), - ("Д-р Смирнов здесь, чтобы увидеть вас.", "доктор Смирнов здесь, чтобы увидеть вас.", "ru"), - # Turkish - ("Merhaba B. Yılmaz.", "Merhaba bay Yılmaz.", "tr"), - ("Dr. Ayşe burada.", "doktor Ayşe burada.", "tr"), - # Hungarian - ("Dr. Szabó itt van.", "doktor Szabó itt van.", "hu"), - ] - - for a, b, lang in test_cases: - out = expand_abbreviations_multilingual(a, lang=lang) - assert out == b, f"'{out}' vs '{b}'" - - -def test_symbols_multilingual(): - test_cases = [ - ("I have 14% battery", "I have 14 percent battery", "en"), - ("Te veo @ la fiesta", "Te veo arroba la fiesta", "es"), - ("J'ai 14° de fièvre", "J'ai 14 degrés de fièvre", "fr"), - ("Die Rechnung beträgt £ 20", "Die Rechnung beträgt pfund 20", "de"), - ("O meu email é ana&joao@gmail.com", "O meu email é ana e joao arroba gmail.com", "pt"), - ("linguaggio di programmazione C#", "linguaggio di programmazione C cancelletto", "it"), - ("Moja temperatura to 36.6°", "Moja temperatura to 36.6 stopnie", "pl"), - ("Mám 14% baterie", "Mám 14 procento baterie", "cs"), - ("Těším se na tebe @ party", "Těším se na tebe na party", "cs"), - ("У меня 14% заряда", "У меня 14 процентов заряда", "ru"), - ("Я буду @ дома", "Я буду собака дома", "ru"), - ("Ik heb 14% batterij", "Ik heb 14 procent batterij", "nl"), - ("Ik zie je @ het feest", "Ik zie je bij het feest", "nl"), - ("لدي 14% في البطارية", "لدي 14 في المئة في البطارية", "ar"), - ("我的电量为 14%", "我的电量为 14 百分之", "zh"), - ("Pilim %14 dolu.", "Pilim yüzde 14 dolu.", "tr"), - ("Az akkumulátorom töltöttsége 14%", "Az akkumulátorom töltöttsége 14 százalék", "hu"), - ("배터리 잔량이 14%입니다.", "배터리 잔량이 14 퍼센트입니다.", "ko"), - ] - - for a, b, lang in test_cases: - out = expand_symbols_multilingual(a, lang=lang) - assert out == b, f"'{out}' vs '{b}'" - - -if __name__ == "__main__": - test_expand_numbers_multilingual() - test_abbreviations_multilingual() - test_symbols_multilingual() diff --git a/system/requirements/requirements_googlecolab.txt b/system/requirements/requirements_googlecolab.txt deleted file mode 100644 index 2c47b8e..0000000 --- a/system/requirements/requirements_googlecolab.txt +++ /dev/null @@ -1,171 +0,0 @@ -absl-py==2.1.0 -aiofiles==23.2.1 -aiohttp==3.9.3 -aiosignal==1.3.1 -altair==5.2.0 -annotated-types==0.6.0 -anyascii==0.3.2 -anyio==4.3.0 -attrs==23.2.0 -audioread==3.0.1 -av==11.0.0 -Babel==2.14.0 -bangla==0.0.2 -blinker==1.7.0 -blis==0.7.11 -bnnumerizer==0.0.2 -bnunicodenormalizer==0.1.6 -catalogue==2.0.10 -certifi==2024.2.2 -cffi==1.16.0 -charset-normalizer==3.3.2 -click==8.1.7 -cloudpathlib==0.16.0 -colorama==0.4.6 -coloredlogs==15.0.1 -confection==0.1.4 -contourpy==1.2.0 -coqpit==0.0.17 -ctranslate2==4.1.0 -cutlet==0.4.0 -cycler==0.12.1 -cymem==2.0.8 -Cython==3.0.9 -dateparser==1.1.8 -decorator==5.1.1 -docopt==0.6.2 -einops==0.7.0 -encodec==0.1.1 -fastapi==0.110.0 -faster-whisper==1.0.1 -ffmpy==0.3.2 -filelock==3.13.3 -Flask==3.0.2 -flatbuffers==24.3.25 -fonttools==4.50.0 -frozenlist==1.4.1 -fsspec==2024.3.1 -fugashi==1.3.1 -fuzzywuzzy>=0.18.0 -g2pkk==0.1.2 -gradio==3.50.2 -gradio_client==0.6.1 -grpcio==1.62.1 -gruut==2.2.3 -gruut-ipa==0.13.0 -gruut-lang-de==2.0.0 -gruut-lang-en==2.0.0 -gruut-lang-es==2.0.0 -gruut-lang-fr==2.0.2 -h11==0.14.0 -hangul-romanize==0.1.0 -httpcore==1.0.4 -httpx==0.27.0 -huggingface-hub==0.22.1 -humanfriendly==10.0 -idna==3.6 -importlib_metadata==7.1.0 -importlib_resources==6.4.0 -inflect==7.0.0 -itsdangerous==2.1.2 -jaconv==0.3.4 -jamo==0.4.1 -jieba==0.42.1 -Jinja2==3.1.3 -joblib==1.3.2 -jsonlines==1.2.0 -jsonschema==4.21.1 -jsonschema-specifications==2023.12.1 -kiwisolver==1.4.5 -langcodes==3.3.0 -lazy_loader==0.3 -librosa==0.10.1 -llvmlite==0.42.0 -Markdown==3.6 -MarkupSafe==2.1.5 -matplotlib==3.8.3 -mojimoji==0.0.13 -mpmath==1.3.0 -msgpack==1.0.8 -multidict==6.0.5 -murmurhash==1.0.10 -networkx==2.8.8 -nltk==3.8.1 -num2words==0.5.13 -numba==0.59.1 -numpy>=1.17.0,<1.29.0 -nvidia-cublas-cu11>=11.11.3.6 -nvidia-cudnn-cu11>=9.0.0.312 -onnxruntime==1.17.1 -orjson==3.9.15 -packaging==24.0 -pandas==1.5.3 -pillow==10.2.0 -platformdirs==4.2.0 -pooch==1.8.1 -preshed==3.0.9 -protobuf==5.26.0 -psutil==5.9.8 -pycparser==2.21 -pydantic==2.6.4 -pydantic_core==2.16.3 -pydub==0.25.1 -pynndescent==0.5.11 -pyparsing==3.1.2 -pypinyin==0.51.0 -pyreadline3==3.4.1 -pysbd==0.3.4 -python-crfsuite==0.9.10 -python-dateutil==2.9.0.post0 -python-Levenshtein>=0.25.0 -python-multipart==0.0.9 -pytz==2024.1 -PyYAML==6.0.1 -referencing==0.34.0 -regex==2023.12.25 -requests==2.31.0 -rpds-py==0.18.0 -safetensors==0.4.2 -scikit-learn==1.4.1.post1 -scipy==1.12.0 -semantic-version==2.10.0 -six==1.16.0 -smart-open==6.4.0 -sniffio==1.3.1 -sounddevice==0.4.6 -soundfile==0.12.1 -soxr==0.3.7 -spacy==3.7.4 -spacy-legacy==3.0.12 -spacy-loggers==1.0.5 -srsly==2.4.8 -starlette==0.36.3 -SudachiDict-core==20240109 -SudachiPy==0.6.8 -sympy==1.12 -tensorboard==2.16.2 -tensorboard-data-server==0.7.2 -thinc==8.2.3 -threadpoolctl==3.4.0 -tokenizers==0.15.2 -toolz==0.12.1 -torch>=2.2.0 -torchaudio>=2.2.0 -tqdm==4.66.2 -trainer==0.0.36 -transformers==4.39.1 -typer==0.9.4 -typing_extensions==4.10.0 -tzdata==2024.1 -tzlocal==5.2 -umap-learn==0.5.5 -Unidecode==1.3.8 -unidic-lite==1.0.8 -urllib3==2.2.1 -uvicorn==0.29.0 -wasabi==1.1.2 -weasel==0.3.4 -websockets==11.0.3 -Werkzeug==3.0.1 -yarl==1.9.4 -zipp==3.18.1 diff --git a/system/requirements/requirements_standalone.txt b/system/requirements/requirements_standalone.txt deleted file mode 100644 index 59cec9c..0000000 --- a/system/requirements/requirements_standalone.txt +++ /dev/null @@ -1,172 +0,0 @@ -absl-py==2.1.0 -aiofiles==23.2.1 -aiohttp==3.9.3 -aiosignal==1.3.1 -altair==5.2.0 -annotated-types==0.6.0 -anyascii==0.3.2 -anyio==4.3.0 -attrs==23.2.0 -audioread==3.0.1 -av==11.0.0 -Babel==2.14.0 -bangla==0.0.2 -blinker==1.7.0 -blis==0.7.11 -bnnumerizer==0.0.2 -bnunicodenormalizer==0.1.6 -catalogue==2.0.10 -certifi==2024.2.2 -cffi==1.16.0 -charset-normalizer==3.3.2 -click==8.1.7 -cloudpathlib==0.16.0 -colorama==0.4.6 -coloredlogs==15.0.1 -confection==0.1.4 -contourpy==1.2.0 -coqpit==0.0.17 -ctranslate2==4.1.0 -cutlet==0.4.0 -cycler==0.12.1 -cymem==2.0.8 -Cython==3.0.9 -dateparser==1.1.8 -decorator==5.1.1 -docopt==0.6.2 -einops==0.7.0 -encodec==0.1.1 -fastapi==0.110.0 -faster-whisper==1.0.1 -ffmpy==0.3.2 -filelock==3.13.3 -Flask==3.0.2 -flatbuffers==24.3.25 -fonttools==4.50.0 -frozenlist==1.4.1 -fsspec==2024.3.1 -fugashi==1.3.1 -fuzzywuzzy>=0.18.0 -g2pkk==0.1.2 -gradio==3.50.2 -gradio_client==0.6.1 -grpcio==1.62.1 -gruut==2.2.3 -gruut-ipa==0.13.0 -gruut-lang-de==2.0.0 -gruut-lang-en==2.0.0 -gruut-lang-es==2.0.0 -gruut-lang-fr==2.0.2 -h11==0.14.0 -hangul-romanize==0.1.0 -httpcore==1.0.4 -httpx==0.27.0 -huggingface-hub==0.22.1 -humanfriendly==10.0 -idna==3.6 -importlib_metadata==7.1.0 -importlib_resources==6.4.0 -inflect==7.0.0 -itsdangerous==2.1.2 -jaconv==0.3.4 -jamo==0.4.1 -jieba==0.42.1 -Jinja2==3.1.3 -joblib==1.3.2 -jsonlines==1.2.0 -jsonschema==4.21.1 -jsonschema-specifications==2023.12.1 -kiwisolver==1.4.5 -langcodes==3.3.0 -lazy_loader==0.3 -librosa==0.10.1 -llvmlite==0.42.0 -Markdown==3.6 -MarkupSafe==2.1.5 -matplotlib==3.8.3 -mojimoji==0.0.13 -mpmath==1.3.0 -msgpack==1.0.8 -multidict==6.0.5 -murmurhash==1.0.10 -networkx==2.8.8 -nltk==3.8.1 -num2words==0.5.13 -numba==0.59.1 -numpy==1.26.4 -nvidia-cublas-cu11>=11.11.3.6 -nvidia-cudnn-cu11>=9.0.0.312 -onnxruntime==1.17.1 -orjson==3.9.15 -packaging==24.0 -pandas==1.5.3 -pillow==10.2.0 -platformdirs==4.2.0 -pooch==1.8.1 -preshed==3.0.9 -protobuf==5.26.0 -psutil==5.9.8 -pycparser==2.21 -pydantic==2.6.4 -pydantic_core==2.16.3 -pydub==0.25.1 -pynndescent==0.5.11 -pyparsing==3.1.2 -pypinyin==0.51.0 -pyreadline3==3.4.1 -pysbd==0.3.4 -python-crfsuite==0.9.10 -python-dateutil==2.9.0.post0 -python-Levenshtein>=0.25.0 -python-multipart==0.0.9 -pytz==2024.1 -PyYAML==6.0.1 -referencing==0.34.0 -regex==2023.12.25 -requests==2.31.0 -rpds-py==0.18.0 -safetensors==0.4.2 -scikit-learn==1.4.1.post1 -scipy==1.12.0 -semantic-version==2.10.0 -six==1.16.0 -smart-open==6.4.0 -sniffio==1.3.1 -sounddevice==0.4.6 -soundfile==0.12.1 -soxr==0.3.7 -spacy==3.7.4 -spacy-legacy==3.0.12 -spacy-loggers==1.0.5 -srsly==2.4.8 -starlette==0.36.3 -SudachiDict-core==20240109 -SudachiPy==0.6.8 -sympy==1.12 -tensorboard==2.16.2 -tensorboard-data-server==0.7.2 -thinc==8.2.3 -threadpoolctl==3.4.0 -tokenizers==0.15.2 -toolz==0.12.1 -torch>=2.2.0 -torchaudio>=2.2.0 -tqdm==4.66.2 -trainer==0.0.36 -transformers==4.39.1 -TTS==0.22.0 -typer==0.9.4 -typing_extensions==4.10.0 -tzdata==2024.1 -tzlocal==5.2 -umap-learn==0.5.5 -Unidecode==1.3.8 -unidic-lite==1.0.8 -urllib3==2.2.1 -uvicorn==0.29.0 -wasabi==1.1.2 -weasel==0.3.4 -websockets==11.0.3 -Werkzeug==3.0.1 -yarl==1.9.4 -zipp==3.18.1 diff --git a/system/requirements/requirements_textgen.txt b/system/requirements/requirements_textgen.txt deleted file mode 100644 index a0ba28e..0000000 --- a/system/requirements/requirements_textgen.txt +++ /dev/null @@ -1,24 +0,0 @@ -numpy>=1.22.0 -soundfile>=0.12.1 -uvicorn>=0.24.0.post1 -transformers>=4.37.1 -TTS>=0.22.0 -torch>=2.1.0+cu118 -torchaudio>=2.1.0+cu118 -fastapi>=0.104.1 -Jinja2>=3.1.2 -requests>=2.31.0 -tqdm>=4.66.1 -importlib-metadata>=4.8.1 -packaging>=23.2 -pydantic>=1.10.13 -sounddevice>=0.4.6 -python-multipart>=0.0.6 -cutlet>=0.3.0 -unidic-lite>=1.0.8 -faster-whisper>=0.10.0 -pandas>=1.5.0 -nvidia-cublas-cu11>=11.11.3.6 -nvidia-cudnn-cu11>=9.0.0.312 -fuzzywuzzy>=0.18.0 -python-Levenshtein>=0.25.0 diff --git a/system/st_files/alltalk.js b/system/st_files/alltalk.js deleted file mode 100644 index f70daf5..0000000 --- a/system/st_files/alltalk.js +++ /dev/null @@ -1,833 +0,0 @@ -import { doExtrasFetch, getApiUrl, modules } from '../../extensions.js'; -import { saveTtsProviderSettings } from './index.js'; - -export { AllTalkTtsProvider }; - -class AllTalkTtsProvider { - //########// - // Config // - //########// - - settings = {}; - constructor() { - // Initialize with default settings if they are not already set - this.settings = { - provider_endpoint: this.settings.provider_endpoint || 'http://localhost:7851', - language: this.settings.language || 'en', - voiceMap: this.settings.voiceMap || {}, - at_generation_method: this.settings.at_generation_method || 'standard_generation', - narrator_enabled: this.settings.narrator_enabled || 'false', - at_narrator_text_not_inside: this.settings.at_narrator_text_not_inside || 'narrator', - narrator_voice_gen: this.settings.narrator_voice_gen || 'female_01.wav', - finetuned_model: this.settings.finetuned_model || 'false' - }; - // Separate property for dynamically updated settings from the server - this.dynamicSettings = { - modelsAvailable: [], - currentModel: '', - deepspeed_available: false, - deepSpeedEnabled: false, - lowVramEnabled: false, - }; - } - ready = false; - voices = []; - separator = '. '; - audioElement = document.createElement('audio'); - - languageLabels = { - 'Arabic': 'ar', - 'Brazilian Portuguese': 'pt', - 'Chinese': 'zh-cn', - 'Czech': 'cs', - 'Dutch': 'nl', - 'English': 'en', - 'French': 'fr', - 'German': 'de', - 'Italian': 'it', - 'Polish': 'pl', - 'Russian': 'ru', - 'Spanish': 'es', - 'Turkish': 'tr', - 'Japanese': 'ja', - 'Korean': 'ko', - 'Hungarian': 'hu', - 'Hindi': 'hi', - }; - - get settingsHtml() { - let html = `
AllTalk Settings
`; - - html += `
- -
- - -
-
`; - - html += `
- -
- - -
- -
- - -
- -
`; - - html += `
-
- - -
-
- - -
-
`; - - - html += `
-
- - -
- -
- - -
-
`; - - - html += `
-
- - -
-
- - -
-
- Status: Ready -
-
- -
-
`; - - - html += `
-
- AllTalk Config & Docs. -
- -
- AllTalk Website. -
-
`; - - html += `
-
-Text-generation-webui users - Uncheck Enable TTS in Text-generation-webui. -
-
`; - - return html; - } - - - //#################// - // Startup ST & AT // - //#################// - - async loadSettings(settings) { - updateStatus('Offline'); - - if (Object.keys(settings).length === 0) { - console.info('Using default AllTalk TTS Provider settings'); - } else { - // Populate settings with provided values, ignoring server-provided settings - for (const key in settings) { - if (key in this.settings) { - this.settings[key] = settings[key]; - } else { - console.debug(`Ignoring non-user-configurable setting: ${key}`); - } - } - } - - // Update UI elements to reflect the loaded settings - $('#at_server').val(this.settings.provider_endpoint); - $('#language_options').val(this.settings.language); - //$('#voicemap').val(this.settings.voiceMap); - $('#at_generation_method').val(this.settings.at_generation_method); - $('#at_narrator_enabled').val(this.settings.narrator_enabled); - $('#at_narrator_text_not_inside').val(this.settings.at_narrator_text_not_inside); - $('#narrator_voice').val(this.settings.narrator_voice_gen); - - console.debug('AllTalkTTS: Settings loaded'); - try { - // Check if TTS provider is ready - await this.checkReady(); - await this.updateSettingsFromServer(); // Fetch dynamic settings from the TTS server - await this.fetchTtsVoiceObjects(); // Fetch voices only if service is ready - this.updateNarratorVoicesDropdown(); - this.updateLanguageDropdown(); - this.setupEventListeners(); - this.applySettingsToHTML(); - updateStatus('Ready'); - } catch (error) { - console.error("Error loading settings:", error); - updateStatus('Offline'); - } - } - - applySettingsToHTML() { - // Apply loaded settings or use defaults - const narratorVoiceSelect = document.getElementById('narrator_voice'); - const atNarratorSelect = document.getElementById('at_narrator_enabled'); - const textNotInsideSelect = document.getElementById('at_narrator_text_not_inside'); - const generationMethodSelect = document.getElementById('at_generation_method'); - this.settings.narrator_voice = this.settings.narrator_voice_gen; - // Apply settings to Narrator Voice dropdown - if (narratorVoiceSelect && this.settings.narrator_voice) { - narratorVoiceSelect.value = this.settings.narrator_voice.replace('.wav', ''); - } - // Apply settings to AT Narrator Enabled dropdown - if (atNarratorSelect) { - // Sync the state with the checkbox in index.js - const ttsPassAsterisksCheckbox = document.getElementById('tts_pass_asterisks'); // Access the checkbox from index.js - const ttsNarrateQuotedCheckbox = document.getElementById('tts_narrate_quoted'); // Access the checkbox from index.js - const ttsNarrateDialoguesCheckbox = document.getElementById('tts_narrate_dialogues'); // Access the checkbox from index.js - // Sync the state with the checkbox in index.js - if (this.settings.narrator_enabled) { - ttsPassAsterisksCheckbox.checked = false; - $('#tts_pass_asterisks').click(); // Simulate a click event - $('#tts_pass_asterisks').trigger('change'); - } - if (!this.settings.narrator_enabled) { - ttsPassAsterisksCheckbox.checked = true; - $('#tts_pass_asterisks').click(); // Simulate a click event - $('#tts_pass_asterisks').trigger('change'); - } - // Uncheck and set tts_narrate_quoted to false if narrator is enabled - if (this.settings.narrator_enabledd) { - ttsNarrateQuotedCheckbox.checked = true; - ttsNarrateDialoguesCheckbox.checked = true; - // Trigger click events instead of change events - $('#tts_narrate_quoted').click(); - $('#tts_narrate_quoted').trigger('change'); - $('#tts_narrate_dialogues').click(); - $('#tts_narrate_dialogues').trigger('change'); - } - atNarratorSelect.value = this.settings.narrator_enabled.toString(); - this.settings.narrator_enabled = this.settings.narrator_enabled.toString(); - } - // Apply settings to the Language dropdown - const languageSelect = document.getElementById('language_options'); - if (languageSelect && this.settings.language) { - languageSelect.value = this.settings.language; - } - // Apply settings to Text Not Inside dropdown - if (textNotInsideSelect && this.settings.text_not_inside) { - textNotInsideSelect.value = this.settings.text_not_inside; - this.settings.at_narrator_text_not_inside = this.settings.text_not_inside; - } - // Apply settings to Generation Method dropdown - if (generationMethodSelect && this.settings.at_generation_method) { - generationMethodSelect.value = this.settings.at_generation_method; - } - // Additional logic to disable/enable dropdowns based on the selected generation method - const isStreamingEnabled = this.settings.at_generation_method === 'streaming_enabled'; - if (isStreamingEnabled) { - // Disable certain dropdowns when streaming is enabled - if (atNarratorSelect) atNarratorSelect.disabled = true; - if (textNotInsideSelect) textNotInsideSelect.disabled = true; - if (narratorVoiceSelect) narratorVoiceSelect.disabled = true; - } else { - // Enable dropdowns for standard generation - if (atNarratorSelect) atNarratorSelect.disabled = false; - if (textNotInsideSelect) textNotInsideSelect.disabled = !this.settings.narrator_enabled; - if (narratorVoiceSelect) narratorVoiceSelect.disabled = !this.settings.narrator_enabled; - } - const modelSelect = document.getElementById('switch_model'); - if (this.settings.finetuned_model === 'true') { - const ftOption = document.createElement('option'); - ftOption.value = 'XTTSv2 FT'; - ftOption.textContent = 'XTTSv2 FT'; - modelSelect.appendChild(ftOption); - } - } - - //##############################// - // Check AT Server is Available // - //##############################// - - async checkReady() { - try { - const response = await fetch(`${this.settings.provider_endpoint}/api/ready`); - // Check if the HTTP request was successful - if (!response.ok) { - throw new Error(`HTTP Error Response: ${response.status} ${response.statusText}`); - } - const statusText = await response.text(); - // Check if the response is 'Ready' - if (statusText === 'Ready') { - this.ready = true; // Set the ready flag to true - console.log('TTS service is ready.'); - } else { - this.ready = false; - console.log('TTS service is not ready.'); - } - } catch (error) { - console.error('Error checking TTS service readiness:', error); - this.ready = false; // Ensure ready flag is set to false in case of error - } - } - - //######################// - // Get Available Voices // - //######################// - - async fetchTtsVoiceObjects() { - const response = await fetch(`${this.settings.provider_endpoint}/api/voices`); - if (!response.ok) { - const errorText = await response.text(); - throw new Error(`HTTP ${response.status}: ${errorText}`); - } - const data = await response.json(); - const voices = data.voices.map(filename => { - const voiceName = filename.replace('.wav', ''); - return { - name: voiceName, - voice_id: voiceName, - preview_url: null, // Preview URL will be dynamically generated - lang: 'en' // Default language - }; - }); - this.voices = voices; // Assign to the class property - return voices; // Also return this list - } - - //##########################################// - // Get Current AT Server Config & Update ST // - //##########################################// - - async updateSettingsFromServer() { - try { - // Fetch current settings - const response = await fetch(`${this.settings.provider_endpoint}/api/currentsettings`); - if (!response.ok) { - throw new Error(`Failed to fetch current settings: ${response.statusText}`); - } - const currentSettings = await response.json(); - // Update internal settings - this.settings.modelsAvailable = currentSettings.models_available; - this.settings.currentModel = currentSettings.current_model_loaded; - this.settings.deepspeed_available = currentSettings.deepspeed_available; - this.settings.deepSpeedEnabled = currentSettings.deepspeed_status; - this.settings.lowVramEnabled = currentSettings.low_vram_status; - this.settings.finetuned_model = currentSettings.finetuned_model - // Update HTML elements - this.updateModelDropdown(); - this.updateCheckboxes(); - } catch (error) { - console.error(`Error updating settings from server: ${error}`); - } - } - - //###################################################// - // Get Current AT Server Config & Update ST (Models) // - //###################################################// - - updateModelDropdown() { - const modelSelect = document.getElementById('switch_model'); - if (modelSelect) { - modelSelect.innerHTML = ''; // Clear existing options - this.settings.modelsAvailable.forEach(model => { - const option = document.createElement('option'); - option.value = model.model_name; - option.textContent = model.model_name; // Use model_name instead of name - option.selected = model.model_name === this.settings.currentModel; - modelSelect.appendChild(option); - }); - } - } - - //#######################################################// - // Get Current AT Server Config & Update ST (DS and LVR) // - //#######################################################// - - updateCheckboxes() { - const deepspeedCheckbox = document.getElementById('deepspeed'); - const lowVramCheckbox = document.getElementById('low_vram'); - if (lowVramCheckbox) lowVramCheckbox.checked = this.settings.lowVramEnabled; - if (deepspeedCheckbox) { - deepspeedCheckbox.checked = this.settings.deepSpeedEnabled; - deepspeedCheckbox.disabled = !this.settings.deepspeed_available; // Disable checkbox if deepspeed is not available - } - } - - //###############################################################// - // Get Current AT Server Config & Update ST (AT Narrator Voices) // - //###############################################################// - - updateNarratorVoicesDropdown() { - const narratorVoiceSelect = document.getElementById('narrator_voice'); - if (narratorVoiceSelect && this.voices) { - // Clear existing options - narratorVoiceSelect.innerHTML = ''; - // Add new options - for (let voice of this.voices) { - const option = document.createElement('option'); - option.value = voice.voice_id; - option.textContent = voice.name; - narratorVoiceSelect.appendChild(option); - } - } - } - - //######################################################// - // Get Current AT Server Config & Update ST (Languages) // - //######################################################// - - updateLanguageDropdown() { - const languageSelect = document.getElementById('language_options'); - if (languageSelect) { - // Ensure default language is set - this.settings.language = this.settings.language; - - languageSelect.innerHTML = ''; - for (let language in this.languageLabels) { - const option = document.createElement('option'); - option.value = this.languageLabels[language]; - option.textContent = language; - if (this.languageLabels[language] === this.settings.language) { - option.selected = true; - } - languageSelect.appendChild(option); - } - } - } - - //########################################// - // Start AT TTS extenstion page listeners // - //########################################// - - setupEventListeners() { - - let debounceTimeout; - const debounceDelay = 500; // Milliseconds - - // Define the event handler function - const onModelSelectChange = async (event) => { - console.log("Model select change event triggered"); // Debugging statement - const selectedModel = event.target.value; - console.log(`Selected model: ${selectedModel}`); // Debugging statement - // Set status to Processing - updateStatus('Processing'); - try { - const response = await fetch(`${this.settings.provider_endpoint}/api/reload?tts_method=${encodeURIComponent(selectedModel)}`, { - method: 'POST' - }); - if (!response.ok) { - throw new Error(`HTTP Error: ${response.status}`); - } - const data = await response.json(); - console.log("POST response data:", data); // Debugging statement - // Set status to Ready if successful - updateStatus('Ready'); - } catch (error) { - console.error("POST request error:", error); // Debugging statement - // Set status to Error in case of failure - updateStatus('Error'); - } - - // Handle response or error - }; - - const debouncedModelSelectChange = (event) => { - clearTimeout(debounceTimeout); - debounceTimeout = setTimeout(() => { - onModelSelectChange(event); - }, debounceDelay); - }; - - // Switch Model Listener - const modelSelect = document.getElementById('switch_model'); - if (modelSelect) { - // Remove the event listener if it was previously added - modelSelect.removeEventListener('change', debouncedModelSelectChange); - // Add the debounced event listener - modelSelect.addEventListener('change', debouncedModelSelectChange); - } - - // DeepSpeed Listener - const deepspeedCheckbox = document.getElementById('deepspeed'); - if (deepspeedCheckbox) { - deepspeedCheckbox.addEventListener('change', async (event) => { - const deepSpeedValue = event.target.checked ? 'True' : 'False'; - // Set status to Processing - updateStatus('Processing'); - try { - const response = await fetch(`${this.settings.provider_endpoint}/api/deepspeed?new_deepspeed_value=${deepSpeedValue}`, { - method: 'POST' - }); - if (!response.ok) { - throw new Error(`HTTP Error: ${response.status}`); - } - const data = await response.json(); - console.log("POST response data:", data); // Debugging statement - // Set status to Ready if successful - updateStatus('Ready'); - } catch (error) { - console.error("POST request error:", error); // Debugging statement - // Set status to Error in case of failure - updateStatus('Error'); - } - }); - } - - // Low VRAM Listener - const lowVramCheckbox = document.getElementById('low_vram'); - if (lowVramCheckbox) { - lowVramCheckbox.addEventListener('change', async (event) => { - const lowVramValue = event.target.checked ? 'True' : 'False'; - // Set status to Processing - updateStatus('Processing'); - try { - const response = await fetch(`${this.settings.provider_endpoint}/api/lowvramsetting?new_low_vram_value=${lowVramValue}`, { - method: 'POST' - }); - if (!response.ok) { - throw new Error(`HTTP Error: ${response.status}`); - } - const data = await response.json(); - console.log("POST response data:", data); // Debugging statement - // Set status to Ready if successful - updateStatus('Ready'); - } catch (error) { - console.error("POST request error:", error); // Debugging statement - // Set status to Error in case of failure - updateStatus('Error'); - } - }); - } - - // Narrator Voice Dropdown Listener - const narratorVoiceSelect = document.getElementById('narrator_voice'); - if (narratorVoiceSelect) { - narratorVoiceSelect.addEventListener('change', (event) => { - this.settings.narrator_voice_gen = `${event.target.value}.wav`; - this.onSettingsChange(); // Save the settings after change - }); - } - - const textNotInsideSelect = document.getElementById('at_narrator_text_not_inside'); - if (textNotInsideSelect) { - textNotInsideSelect.addEventListener('change', (event) => { - this.settings.text_not_inside = event.target.value; - this.onSettingsChange(); // Save the settings after change - }); - } - - // AT Narrator Dropdown Listener - const atNarratorSelect = document.getElementById('at_narrator_enabled'); - const ttsPassAsterisksCheckbox = document.getElementById('tts_pass_asterisks'); // Access the checkbox from index.js - const ttsNarrateQuotedCheckbox = document.getElementById('tts_narrate_quoted'); // Access the checkbox from index.js - const ttsNarrateDialoguesCheckbox = document.getElementById('tts_narrate_dialogues'); // Access the checkbox from index.js - - if (atNarratorSelect && textNotInsideSelect && narratorVoiceSelect) { - atNarratorSelect.addEventListener('change', (event) => { - const isNarratorEnabled = event.target.value === 'true'; - this.settings.narrator_enabled = isNarratorEnabled; // Update the setting here - textNotInsideSelect.disabled = !isNarratorEnabled; - narratorVoiceSelect.disabled = !isNarratorEnabled; - - // Sync the state with the checkbox in index.js - if (isNarratorEnabled) { - ttsPassAsterisksCheckbox.checked = false; - $('#tts_pass_asterisks').click(); // Simulate a click event - $('#tts_pass_asterisks').trigger('change'); - } - if (!isNarratorEnabled) { - ttsPassAsterisksCheckbox.checked = true; - $('#tts_pass_asterisks').click(); // Simulate a click event - $('#tts_pass_asterisks').trigger('change'); - } - // Uncheck and set tts_narrate_quoted to false if narrator is enabled - if (isNarratorEnabled) { - ttsNarrateQuotedCheckbox.checked = true; - ttsNarrateDialoguesCheckbox.checked = true; - // Trigger click events instead of change events - $('#tts_narrate_quoted').click(); - $('#tts_narrate_quoted').trigger('change'); - $('#tts_narrate_dialogues').click(); - $('#tts_narrate_dialogues').trigger('change'); - } - this.onSettingsChange(); // Save the settings after change - }); - } - - - // Event Listener for AT Generation Method Dropdown - const atGenerationMethodSelect = document.getElementById('at_generation_method'); - const atNarratorEnabledSelect = document.getElementById('at_narrator_enabled'); - if (atGenerationMethodSelect) { - atGenerationMethodSelect.addEventListener('change', (event) => { - const selectedMethod = event.target.value; - - if (selectedMethod === 'streaming_enabled') { - // Disable and unselect AT Narrator - atNarratorEnabledSelect.disabled = true; - atNarratorEnabledSelect.value = 'false'; - textNotInsideSelect.disabled = true; - narratorVoiceSelect.disabled = true; - } else if (selectedMethod === 'standard_generation') { - // Enable AT Narrator - atNarratorEnabledSelect.disabled = false; - } - this.settings.at_generation_method = selectedMethod; // Update the setting here - this.onSettingsChange(); // Save the settings after change - }); - } - - // Listener for Language Dropdown - const languageSelect = document.getElementById('language_options'); - if (languageSelect) { - languageSelect.addEventListener('change', (event) => { - this.settings.language = event.target.value; - this.onSettingsChange(); // Save the settings after change - }); - } - - // Listener for AllTalk Endpoint Input - const atServerInput = document.getElementById('at_server'); - if (atServerInput) { - atServerInput.addEventListener('input', (event) => { - this.settings.provider_endpoint = event.target.value; - this.onSettingsChange(); // Save the settings after change - }); - } - - } - - //#############################// - // Store ST interface settings // - //#############################// - - onSettingsChange() { - // Update settings based on the UI elements - //this.settings.provider_endpoint = $('#at_server').val(); - this.settings.language = $('#language_options').val(); - //this.settings.voiceMap = $('#voicemap').val(); - this.settings.at_generation_method = $('#at_generation_method').val(); - this.settings.narrator_enabled = $('#at_narrator_enabled').val(); - this.settings.at_narrator_text_not_inside = $('#at_narrator_text_not_inside').val(); - this.settings.narrator_voice_gen = $('#narrator_voice').val(); - // Save the updated settings - saveTtsProviderSettings(); - } - - //#########################// - // ST Handle Reload button // - //#########################// - - async onRefreshClick() { - await this.checkReady(); // Check if the TTS provider is ready - await this.loadSettings(this.settings); // Reload the settings - // Additional actions as needed - } - - //##################// - // Preview AT Voice // - //##################// - - async previewTtsVoice(voiceName) { - try { - // Prepare data for POST request - const postData = new URLSearchParams(); - postData.append("voice", `${voiceName}.wav`); - // Making the POST request - const response = await fetch(`${this.settings.provider_endpoint}/api/previewvoice/`, { - method: "POST", - headers: { - 'Content-Type': 'application/x-www-form-urlencoded' - }, - body: postData, - }); - if (!response.ok) { - const errorText = await response.text(); - console.error(`[previewTtsVoice] Error Response Text:`, errorText); - throw new Error(`HTTP ${response.status}: ${errorText}`); - } - // Assuming the server returns a URL to the .wav file - const data = await response.json(); - if (data.output_file_url) { - // Use an audio element to play the .wav file - const audioElement = new Audio(data.output_file_url); - audioElement.play().catch(e => console.error("Error playing audio:", e)); - } else { - console.warn("[previewTtsVoice] No output file URL received in the response"); - throw new Error("No output file URL received in the response"); - } - - } catch (error) { - console.error("[previewTtsVoice] Exception caught during preview generation:", error); - throw error; - } - } - - //#####################// - // Populate ST voices // - //#####################// - - async getVoice(voiceName, generatePreview = false) { - // Ensure this.voices is populated - if (this.voices.length === 0) { - // Fetch voice objects logic - } - // Find the object where the name matches voiceName - const match = this.voices.find(voice => voice.name === voiceName); - if (!match) { - // Error handling - } - // Generate preview URL only if requested - if (!match.preview_url && generatePreview) { - // Generate preview logic - } - return match; // Return the found voice object - } - - //##########################################// - // Generate TTS Streaming or call Standard // - //##########################################// - - async generateTts(inputText, voiceId) { - try { - if (this.settings.at_generation_method === 'streaming_enabled') { - // Construct the streaming URL - const streamingUrl = `${this.settings.provider_endpoint}/api/tts-generate-streaming?text=${encodeURIComponent(inputText)}&voice=${encodeURIComponent(voiceId)}.wav&language=${encodeURIComponent(this.settings.language)}&output_file=stream_output.wav`; - console.log("Streaming URL:", streamingUrl); - - // Return the streaming URL directly - return streamingUrl; - } else { - // For standard method - const outputUrl = await this.fetchTtsGeneration(inputText, voiceId); - const audioResponse = await fetch(outputUrl); - if (!audioResponse.ok) { - throw new Error(`HTTP ${audioResponse.status}: Failed to fetch audio data`); - } - return audioResponse; // Return the fetch response directly - } - } catch (error) { - console.error("Error in generateTts:", error); - throw error; - } - } - - - //####################// - // Generate Standard // - //####################// - - async fetchTtsGeneration(inputText, voiceId) { - // Prepare the request payload - const requestBody = new URLSearchParams({ - 'text_input': inputText, - 'text_filtering': "standard", - 'character_voice_gen': voiceId + ".wav", - 'narrator_enabled': this.settings.narrator_enabled, - 'narrator_voice_gen': this.settings.narrator_voice_gen + ".wav", - 'text_not_inside': this.settings.at_narrator_text_not_inside, - 'language': this.settings.language, - 'output_file_name': "st_output", - 'output_file_timestamp': "true", - 'autoplay': "false", - 'autoplay_volume': "0.8" - }).toString(); - - try { - const response = await doExtrasFetch( - `${this.settings.provider_endpoint}/api/tts-generate`, - { - method: 'POST', - headers: { - 'Content-Type': 'application/x-www-form-urlencoded', - 'Cache-Control': 'no-cache', - }, - body: requestBody - } - ); - - if (!response.ok) { - const errorText = await response.text(); - console.error(`[fetchTtsGeneration] Error Response Text:`, errorText); - // toastr.error(response.statusText, 'TTS Generation Failed'); - throw new Error(`HTTP ${response.status}: ${errorText}`); - } - const data = await response.json(); - const outputUrl = data.output_file_url; - return outputUrl; // Return only the output_file_url - } catch (error) { - console.error("[fetchTtsGeneration] Exception caught:", error); - throw error; // Rethrow the error for further handling - } - } -} - -//#########################// -// Update Status Messages // -//#########################// - -function updateStatus(message) { - const statusElement = document.getElementById('status_info'); - if (statusElement) { - statusElement.textContent = message; - switch (message) { - case 'Offline': - statusElement.style.color = 'red'; - break; - case 'Ready': - statusElement.style.color = 'lightgreen'; - break; - case 'Processing': - statusElement.style.color = 'blue'; - break; - case 'Error': - statusElement.style.color = 'red'; - break; - } - } -} \ No newline at end of file diff --git a/system/st_files/index.js b/system/st_files/index.js deleted file mode 100644 index dd84a22..0000000 --- a/system/st_files/index.js +++ /dev/null @@ -1,1046 +0,0 @@ -import { callPopup, cancelTtsPlay, eventSource, event_types, name2, saveSettingsDebounced } from '../../../script.js'; -import { ModuleWorkerWrapper, doExtrasFetch, extension_settings, getApiUrl, getContext, modules } from '../../extensions.js'; -import { delay, escapeRegex, getBase64Async, getStringHash, onlyUnique } from '../../utils.js'; -import { EdgeTtsProvider } from './edge.js'; -import { ElevenLabsTtsProvider } from './elevenlabs.js'; -import { SileroTtsProvider } from './silerotts.js'; -import { CoquiTtsProvider } from './coqui.js'; -import { SystemTtsProvider } from './system.js'; -import { NovelTtsProvider } from './novel.js'; -import { power_user } from '../../power-user.js'; -import { registerSlashCommand } from '../../slash-commands.js'; -import { OpenAITtsProvider } from './openai.js'; -import { XTTSTtsProvider } from './xtts.js'; -import { AllTalkTtsProvider } from './alltalk.js'; -export { talkingAnimation }; - -const UPDATE_INTERVAL = 1000; - -let voiceMapEntries = []; -let voiceMap = {}; // {charName:voiceid, charName2:voiceid2} -let storedvalue = false; -let lastChatId = null; -let lastMessageHash = null; - -const DEFAULT_VOICE_MARKER = '[Default Voice]'; -const DISABLED_VOICE_MARKER = 'disabled'; - -export function getPreviewString(lang) { - const previewStrings = { - 'en-US': 'The quick brown fox jumps over the lazy dog', - 'en-GB': 'Sphinx of black quartz, judge my vow', - 'fr-FR': 'Portez ce vieux whisky au juge blond qui fume', - 'de-DE': 'Victor jagt zwölf Boxkämpfer quer über den großen Sylter Deich', - 'it-IT': 'Pranzo d\'acqua fa volti sghembi', - 'es-ES': 'Quiere la boca exhausta vid, kiwi, piña y fugaz jamón', - 'es-MX': 'Fabio me exige, sin tapujos, que añada cerveza al whisky', - 'ru-RU': 'В чащах юга жил бы цитрус? Да, но фальшивый экземпляр!', - 'pt-BR': 'Vejo xá gritando que fez show sem playback.', - 'pt-PR': 'Todo pajé vulgar faz boquinha sexy com kiwi.', - 'uk-UA': 'Фабрикуймо гідність, лящім їжею, ґав хапаймо, з\'єднавці чаш!', - 'pl-PL': 'Pchnąć w tę łódź jeża lub ośm skrzyń fig', - 'cs-CZ': 'Příliš žluťoučký kůň úpěl ďábelské ódy', - 'sk-SK': 'Vyhŕňme si rukávy a vyprážajme čínske ryžové cestoviny', - 'hu-HU': 'Árvíztűrő tükörfúrógép', - 'tr-TR': 'Pijamalı hasta yağız şoföre çabucak güvendi', - 'nl-NL': 'De waard heeft een kalfje en een pinkje opgegeten', - 'sv-SE': 'Yxskaftbud, ge vårbygd, zinkqvarn', - 'da-DK': 'Quizdeltagerne spiste jordbær med fløde, mens cirkusklovnen Walther spillede på xylofon', - 'ja-JP': 'いろはにほへと　ちりぬるを　わかよたれそ　つねならむ　うゐのおくやま　けふこえて　あさきゆめみし　ゑひもせす', - 'ko-KR': '가나다라마바사아자차카타파하', - 'zh-CN': '我能吞下玻璃而不伤身体', - 'ro-RO': 'Muzicologă în bej vând whisky și tequila, preț fix', - 'bg-BG': 'Щъркелите се разпръснаха по цялото небе', - 'el-GR': 'Ταχίστη αλώπηξ βαφής ψημένη γη, δρασκελίζει υπέρ νωθρού κυνός', - 'fi-FI': 'Voi veljet, miksi juuri teille myin nämä vehkeet?', - 'he-IL': 'הקצינים צעקו: "כל הכבוד לצבא הצבאות!"', - 'id-ID': 'Jangkrik itu memang enak, apalagi kalau digoreng', - 'ms-MY': 'Muzik penyanyi wanita itu menggambarkan kehidupan yang penuh dengan duka nestapa', - 'th-TH': 'เป็นไงบ้างครับ ผมชอบกินข้าวผัดกระเพราหมูกรอบ', - 'vi-VN': 'Cô bé quàng khăn đỏ đang ngồi trên bãi cỏ xanh', - 'ar-SA': 'أَبْجَدِيَّة عَرَبِيَّة', - 'hi-IN': 'श्वेता ने श्वेता के श्वेते हाथों में श्वेता का श्वेता चावल पकड़ा', - }; - const fallbackPreview = 'Neque porro quisquam est qui dolorem ipsum quia dolor sit amet'; - - return previewStrings[lang] ?? fallbackPreview; -} - -let ttsProviders = { - ElevenLabs: ElevenLabsTtsProvider, - Silero: SileroTtsProvider, - XTTSv2: XTTSTtsProvider, - System: SystemTtsProvider, - Coqui: CoquiTtsProvider, - Edge: EdgeTtsProvider, - Novel: NovelTtsProvider, - OpenAI: OpenAITtsProvider, - AllTalk: AllTalkTtsProvider, -}; -let ttsProvider; -let ttsProviderName; - -let ttsLastMessage = null; - -async function onNarrateOneMessage() { - audioElement.src = '/sounds/silence.mp3'; - const context = getContext(); - const id = $(this).closest('.mes').attr('mesid'); - const message = context.chat[id]; - - if (!message) { - return; - } - - resetTtsPlayback(); - ttsJobQueue.push(message); - moduleWorker(); -} - -async function onNarrateText(args, text) { - if (!text) { - return; - } - - audioElement.src = '/sounds/silence.mp3'; - - // To load all characters in the voice map, set unrestricted to true - await initVoiceMap(true); - - const baseName = args?.voice || name2; - const name = (baseName === 'SillyTavern System' ? DEFAULT_VOICE_MARKER : baseName) || DEFAULT_VOICE_MARKER; - - const voiceMapEntry = voiceMap[name] === DEFAULT_VOICE_MARKER - ? voiceMap[DEFAULT_VOICE_MARKER] - : voiceMap[name]; - - if (!voiceMapEntry || voiceMapEntry === DISABLED_VOICE_MARKER) { - toastr.info(`Specified voice for ${name} was not found. Check the TTS extension settings.`); - return; - } - - resetTtsPlayback(); - ttsJobQueue.push({ mes: text, name: name }); - await moduleWorker(); - - // Return back to the chat voices - await initVoiceMap(false); -} - -async function moduleWorker() { - // Primarily determining when to add new chat to the TTS queue - const enabled = $('#tts_enabled').is(':checked'); - $('body').toggleClass('tts', enabled); - if (!enabled) { - return; - } - - const context = getContext(); - const chat = context.chat; - - processTtsQueue(); - processAudioJobQueue(); - updateUiAudioPlayState(); - - // Auto generation is disabled - if (extension_settings.tts.auto_generation == false) { - return; - } - - // no characters or group selected - if (!context.groupId && context.characterId === undefined) { - return; - } - - // Chat changed - if ( - context.chatId !== lastChatId - ) { - currentMessageNumber = context.chat.length ? context.chat.length : 0; - saveLastValues(); - - // Force to speak on the first message in the new chat - if (context.chat.length === 1) { - lastMessageHash = -1; - } - - return; - } - - // take the count of messages - let lastMessageNumber = context.chat.length ? context.chat.length : 0; - - // There's no new messages - let diff = lastMessageNumber - currentMessageNumber; - let hashNew = getStringHash((chat.length && chat[chat.length - 1].mes) ?? ''); - - // if messages got deleted, diff will be < 0 - if (diff < 0) { - // necessary actions will be taken by the onChatDeleted() handler - return; - } - - // if no new messages, or same message, or same message hash, do nothing - if (diff == 0 && hashNew === lastMessageHash) { - return; - } - - // If streaming, wait for streaming to finish before processing new messages - if (context.streamingProcessor && !context.streamingProcessor.isFinished) { - return; - } - - // clone message object, as things go haywire if message object is altered below (it's passed by reference) - const message = structuredClone(chat[chat.length - 1]); - - // if last message within current message, message got extended. only send diff to TTS. - if (ttsLastMessage !== null && message.mes.indexOf(ttsLastMessage) !== -1) { - let tmp = message.mes; - message.mes = message.mes.replace(ttsLastMessage, ''); - ttsLastMessage = tmp; - } else { - ttsLastMessage = message.mes; - } - - // We're currently swiping. Don't generate voice - if (!message || message.mes === '...' || message.mes === '') { - return; - } - - // Don't generate if message doesn't have a display text - if (extension_settings.tts.narrate_translated_only && !(message?.extra?.display_text)) { - return; - } - - // Don't generate if message is a user message and user message narration is disabled - if (message.is_user && !extension_settings.tts.narrate_user) { - return; - } - - // New messages, add new chat to history - lastMessageHash = hashNew; - currentMessageNumber = lastMessageNumber; - - console.debug( - `Adding message from ${message.name} for TTS processing: "${message.mes}"`, - ); - ttsJobQueue.push(message); -} - -function talkingAnimation(switchValue) { - if (!modules.includes('talkinghead')) { - console.debug('Talking Animation module not loaded'); - return; - } - - const apiUrl = getApiUrl(); - const animationType = switchValue ? 'start' : 'stop'; - - if (switchValue !== storedvalue) { - try { - console.log(animationType + ' Talking Animation'); - doExtrasFetch(`${apiUrl}/api/talkinghead/${animationType}_talking`); - storedvalue = switchValue; // Update the storedvalue to the current switchValue - } catch (error) { - // Handle the error here or simply ignore it to prevent logging - } - } - updateUiAudioPlayState(); -} - -function resetTtsPlayback() { - // Stop system TTS utterance - cancelTtsPlay(); - - // Clear currently processing jobs - currentTtsJob = null; - currentAudioJob = null; - - // Reset audio element - audioElement.currentTime = 0; - audioElement.src = ''; - - // Clear any queue items - ttsJobQueue.splice(0, ttsJobQueue.length); - audioJobQueue.splice(0, audioJobQueue.length); - - // Set audio ready to process again - audioQueueProcessorReady = true; -} - -function isTtsProcessing() { - let processing = false; - - // Check job queues - if (ttsJobQueue.length > 0 || audioJobQueue.length > 0) { - processing = true; - } - // Check current jobs - if (currentTtsJob != null || currentAudioJob != null) { - processing = true; - } - return processing; -} - -function debugTtsPlayback() { - console.log(JSON.stringify( - { - 'ttsProviderName': ttsProviderName, - 'voiceMap': voiceMap, - 'currentMessageNumber': currentMessageNumber, - 'audioPaused': audioPaused, - 'audioJobQueue': audioJobQueue, - 'currentAudioJob': currentAudioJob, - 'audioQueueProcessorReady': audioQueueProcessorReady, - 'ttsJobQueue': ttsJobQueue, - 'currentTtsJob': currentTtsJob, - 'ttsConfig': extension_settings.tts, - }, - )); -} -window.debugTtsPlayback = debugTtsPlayback; - -//##################// -// Audio Control // -//##################// - -let audioElement = new Audio(); -audioElement.id = 'tts_audio'; -audioElement.autoplay = true; - -let audioJobQueue = []; -let currentAudioJob; -let audioPaused = false; -let audioQueueProcessorReady = true; - -async function playAudioData(audioBlob) { - // Since current audio job can be cancelled, don't playback if it is null - if (currentAudioJob == null) { - console.log('Cancelled TTS playback because currentAudioJob was null'); - } - if (audioBlob instanceof Blob) { - const srcUrl = await getBase64Async(audioBlob); - audioElement.src = srcUrl; - } else if (typeof audioBlob === 'string') { - audioElement.src = audioBlob; - } else { - throw `TTS received invalid audio data type ${typeof audioBlob}`; - } - audioElement.addEventListener('ended', completeCurrentAudioJob); - audioElement.addEventListener('canplay', () => { - console.debug('Starting TTS playback'); - audioElement.play(); - }); -} - -window['tts_preview'] = function (id) { - const audio = document.getElementById(id); - - if (audio && !$(audio).data('disabled')) { - audio.play(); - } - else { - ttsProvider.previewTtsVoice(id); - } -}; - -async function onTtsVoicesClick() { - let popupText = ''; - - try { - const voiceIds = await ttsProvider.fetchTtsVoiceObjects(); - - for (const voice of voiceIds) { - popupText += ` -
- ${voice.lang || ''} - ${voice.name} - -
`; - if (voice.preview_url) { - popupText += ``; - } - } - } catch { - popupText = 'Could not load voices list. Check your API key.'; - } - - callPopup(popupText, 'text'); -} - -function updateUiAudioPlayState() { - if (extension_settings.tts.enabled == true) { - $('#ttsExtensionMenuItem').show(); - let img; - // Give user feedback that TTS is active by setting the stop icon if processing or playing - if (!audioElement.paused || isTtsProcessing()) { - img = 'fa-solid fa-stop-circle extensionsMenuExtensionButton'; - } else { - img = 'fa-solid fa-circle-play extensionsMenuExtensionButton'; - } - $('#tts_media_control').attr('class', img); - } else { - $('#ttsExtensionMenuItem').hide(); - } -} - -function onAudioControlClicked() { - audioElement.src = '/sounds/silence.mp3'; - let context = getContext(); - // Not pausing, doing a full stop to anything TTS is doing. Better UX as pause is not as useful - if (!audioElement.paused || isTtsProcessing()) { - resetTtsPlayback(); - talkingAnimation(false); - } else { - // Default play behavior if not processing or playing is to play the last message. - ttsJobQueue.push(context.chat[context.chat.length - 1]); - } - updateUiAudioPlayState(); -} - -function addAudioControl() { - - $('#extensionsMenu').prepend(` -
-
- TTS Playback -
`); - $('#ttsExtensionMenuItem').attr('title', 'TTS play/pause').on('click', onAudioControlClicked); - updateUiAudioPlayState(); -} - -function completeCurrentAudioJob() { - audioQueueProcessorReady = true; - currentAudioJob = null; - talkingAnimation(false); //stop lip animation - // updateUiPlayState(); -} - -/** - * Accepts an HTTP response containing audio/mpeg data, and puts the data as a Blob() on the queue for playback - * @param {Response} response - */ -async function addAudioJob(response) { - if (typeof response === 'string') { - audioJobQueue.push(response); - } else { - const audioData = await response.blob(); - if (!audioData.type.startsWith('audio/')) { - throw `TTS received HTTP response with invalid data format. Expecting audio/*, got ${audioData.type}`; - } - audioJobQueue.push(audioData); - } - console.debug('Pushed audio job to queue.'); -} - -async function processAudioJobQueue() { - // Nothing to do, audio not completed, or audio paused - stop processing. - if (audioJobQueue.length == 0 || !audioQueueProcessorReady || audioPaused) { - return; - } - try { - audioQueueProcessorReady = false; - currentAudioJob = audioJobQueue.shift(); - playAudioData(currentAudioJob); - talkingAnimation(true); - } catch (error) { - console.error(error); - audioQueueProcessorReady = true; - } -} - -//################// -// TTS Control // -//################// - -let ttsJobQueue = []; -let currentTtsJob; // Null if nothing is currently being processed -let currentMessageNumber = 0; - -function completeTtsJob() { - console.info(`Current TTS job for ${currentTtsJob?.name} completed.`); - currentTtsJob = null; -} - -function saveLastValues() { - const context = getContext(); - lastChatId = context.chatId; - lastMessageHash = getStringHash( - (context.chat.length && context.chat[context.chat.length - 1].mes) ?? '', - ); -} - -async function tts(text, voiceId, char) { - async function processResponse(response) { - // RVC injection - if (extension_settings.rvc.enabled && typeof window['rvcVoiceConversion'] === 'function') - response = await window['rvcVoiceConversion'](response, char, text); - - await addAudioJob(response); - } - - let response = await ttsProvider.generateTts(text, voiceId); - - // If async generator, process every chunk as it comes in - if (typeof response[Symbol.asyncIterator] === 'function') { - for await (const chunk of response) { - await processResponse(chunk); - } - } else { - await processResponse(response); - } - - completeTtsJob(); -} - -async function processTtsQueue() { - // Called each moduleWorker iteration to pull chat messages from queue - if (currentTtsJob || ttsJobQueue.length <= 0 || audioPaused) { - return; - } - - console.debug('New message found, running TTS'); - currentTtsJob = ttsJobQueue.shift(); - let text = extension_settings.tts.narrate_translated_only ? (currentTtsJob?.extra?.display_text || currentTtsJob.mes) : currentTtsJob.mes; - - if (extension_settings.tts.skip_codeblocks) { - text = text.replace(/^\s{4}.*$/gm, '').trim(); - text = text.replace(/```.*?```/gs, '').trim(); - } - - if (!extension_settings.tts.pass_asterisks) { - text = extension_settings.tts.narrate_dialogues_only - ? text.replace(/\*[^*]*?(\*|$)/g, '').trim() // remove asterisks content - : text.replaceAll('*', '').trim(); // remove just the asterisks - } - - if (extension_settings.tts.narrate_quoted_only) { - const special_quotes = /[“”]/g; // Extend this regex to include other special quotes - text = text.replace(special_quotes, '"'); - const matches = text.match(/".*?"/g); // Matches text inside double quotes, non-greedily - const partJoiner = (ttsProvider?.separator || ' ... '); - text = matches ? matches.join(partJoiner) : text; - } - - if (typeof ttsProvider?.processText === 'function') { - text = await ttsProvider.processText(text); - } - - // Collapse newlines and spaces into single space - text = text.replace(/\s+/g, ' ').trim(); - - console.log(`TTS: ${text}`); - const char = currentTtsJob.name; - - // Remove character name from start of the line if power user setting is disabled - if (char && !power_user.allow_name2_display) { - const escapedChar = escapeRegex(char); - text = text.replace(new RegExp(`^${escapedChar}:`, 'gm'), ''); - } - - try { - if (!text) { - console.warn('Got empty text in TTS queue job.'); - completeTtsJob(); - return; - } - - const voiceMapEntry = voiceMap[char] === DEFAULT_VOICE_MARKER ? voiceMap[DEFAULT_VOICE_MARKER] : voiceMap[char]; - - if (!voiceMapEntry || voiceMapEntry === DISABLED_VOICE_MARKER) { - throw `${char} not in voicemap. Configure character in extension settings voice map`; - } - const voice = await ttsProvider.getVoice(voiceMapEntry); - const voiceId = voice.voice_id; - if (voiceId == null) { - toastr.error(`Specified voice for ${char} was not found. Check the TTS extension settings.`); - throw `Unable to attain voiceId for ${char}`; - } - tts(text, voiceId, char); - } catch (error) { - console.error(error); - currentTtsJob = null; - } -} - -// Secret function for now -async function playFullConversation() { - const context = getContext(); - const chat = context.chat; - ttsJobQueue = chat; -} -window.playFullConversation = playFullConversation; - -//#############################// -// Extension UI and Settings // -//#############################// - -function loadSettings() { - if (Object.keys(extension_settings.tts).length === 0) { - Object.assign(extension_settings.tts, defaultSettings); - } - for (const key in defaultSettings) { - if (!(key in extension_settings.tts)) { - extension_settings.tts[key] = defaultSettings[key]; - } - } - $('#tts_provider').val(extension_settings.tts.currentProvider); - $('#tts_enabled').prop( - 'checked', - extension_settings.tts.enabled, - ); - $('#tts_narrate_dialogues').prop('checked', extension_settings.tts.narrate_dialogues_only); - $('#tts_narrate_quoted').prop('checked', extension_settings.tts.narrate_quoted_only); - $('#tts_auto_generation').prop('checked', extension_settings.tts.auto_generation); - $('#tts_narrate_translated_only').prop('checked', extension_settings.tts.narrate_translated_only); - $('#tts_narrate_user').prop('checked', extension_settings.tts.narrate_user); - $('#tts_pass_asterisks').prop('checked', extension_settings.tts.pass_asterisks); - $('body').toggleClass('tts', extension_settings.tts.enabled); -} - -const defaultSettings = { - voiceMap: '', - ttsEnabled: false, - currentProvider: 'ElevenLabs', - auto_generation: true, - narrate_user: false, -}; - -function setTtsStatus(status, success) { - $('#tts_status').text(status); - if (success) { - $('#tts_status').removeAttr('style'); - } else { - $('#tts_status').css('color', 'red'); - } -} - -function onRefreshClick() { - Promise.all([ - ttsProvider.onRefreshClick(), - // updateVoiceMap() - ]).then(() => { - extension_settings.tts[ttsProviderName] = ttsProvider.settings; - saveSettingsDebounced(); - setTtsStatus('Successfully applied settings', true); - console.info(`Saved settings ${ttsProviderName} ${JSON.stringify(ttsProvider.settings)}`); - initVoiceMap(); - updateVoiceMap(); - }).catch(error => { - console.error(error); - setTtsStatus(error, false); - }); -} - -function onEnableClick() { - extension_settings.tts.enabled = $('#tts_enabled').is( - ':checked', - ); - updateUiAudioPlayState(); - saveSettingsDebounced(); -} - - -function onAutoGenerationClick() { - extension_settings.tts.auto_generation = !!$('#tts_auto_generation').prop('checked'); - saveSettingsDebounced(); -} - - -function onNarrateDialoguesClick() { - extension_settings.tts.narrate_dialogues_only = !!$('#tts_narrate_dialogues').prop('checked'); - saveSettingsDebounced(); -} - -function onNarrateUserClick() { - extension_settings.tts.narrate_user = !!$('#tts_narrate_user').prop('checked'); - saveSettingsDebounced(); -} - -function onNarrateQuotedClick() { - extension_settings.tts.narrate_quoted_only = !!$('#tts_narrate_quoted').prop('checked'); - saveSettingsDebounced(); -} - - -function onNarrateTranslatedOnlyClick() { - extension_settings.tts.narrate_translated_only = !!$('#tts_narrate_translated_only').prop('checked'); - saveSettingsDebounced(); -} - -function onSkipCodeblocksClick() { - extension_settings.tts.skip_codeblocks = !!$('#tts_skip_codeblocks').prop('checked'); - saveSettingsDebounced(); -} - -function onPassAsterisksClick() { - extension_settings.tts.pass_asterisks = !!$('#tts_pass_asterisks').prop('checked'); - saveSettingsDebounced(); - console.log("setting pass asterisks", extension_settings.tts.pass_asterisks) -} - -//##############// -// TTS Provider // -//##############// - -async function loadTtsProvider(provider) { - //Clear the current config and add new config - $('#tts_provider_settings').html(''); - - if (!provider) { - return; - } - - // Init provider references - extension_settings.tts.currentProvider = provider; - ttsProviderName = provider; - ttsProvider = new ttsProviders[provider]; - - // Init provider settings - $('#tts_provider_settings').append(ttsProvider.settingsHtml); - if (!(ttsProviderName in extension_settings.tts)) { - console.warn(`Provider ${ttsProviderName} not in Extension Settings, initiatilizing provider in settings`); - extension_settings.tts[ttsProviderName] = {}; - } - await ttsProvider.loadSettings(extension_settings.tts[ttsProviderName]); - await initVoiceMap(); -} - -function onTtsProviderChange() { - const ttsProviderSelection = $('#tts_provider').val(); - extension_settings.tts.currentProvider = ttsProviderSelection; - loadTtsProvider(ttsProviderSelection); -} - -// Ensure that TTS provider settings are saved to extension settings. -export function saveTtsProviderSettings() { - extension_settings.tts[ttsProviderName] = ttsProvider.settings; - updateVoiceMap(); - saveSettingsDebounced(); - console.info(`Saved settings ${ttsProviderName} ${JSON.stringify(ttsProvider.settings)}`); -} - - -//###################// -// voiceMap Handling // -//###################// - -async function onChatChanged() { - await resetTtsPlayback(); - const voiceMapInit = initVoiceMap(); - await Promise.race([voiceMapInit, delay(1000)]); - ttsLastMessage = null; -} - -async function onChatDeleted() { - const context = getContext(); - - // update internal references to new last message - lastChatId = context.chatId; - currentMessageNumber = context.chat.length ? context.chat.length : 0; - - // compare against lastMessageHash. If it's the same, we did not delete the last chat item, so no need to reset tts queue - let messageHash = getStringHash((context.chat.length && context.chat[context.chat.length - 1].mes) ?? ''); - if (messageHash === lastMessageHash) { - return; - } - lastMessageHash = messageHash; - ttsLastMessage = (context.chat.length && context.chat[context.chat.length - 1].mes) ?? ''; - - // stop any tts playback since message might not exist anymore - await resetTtsPlayback(); -} - -/** - * Get characters in current chat - * @param {boolean} unrestricted - If true, will include all characters in voiceMapEntries, even if they are not in the current chat. - * @returns {string[]} - Array of character names - */ -function getCharacters(unrestricted) { - const context = getContext(); - - if (unrestricted) { - const names = context.characters.map(char => char.name); - names.unshift(DEFAULT_VOICE_MARKER); - return names.filter(onlyUnique); - } - - let characters = []; - if (context.groupId === null) { - // Single char chat - characters.push(DEFAULT_VOICE_MARKER); - characters.push(context.name1); - characters.push(context.name2); - } else { - // Group chat - characters.push(DEFAULT_VOICE_MARKER); - characters.push(context.name1); - const group = context.groups.find(group => context.groupId == group.id); - for (let member of group.members) { - const character = context.characters.find(char => char.avatar == member); - if (character) { - characters.push(character.name); - } - } - } - return characters.filter(onlyUnique); -} - -function sanitizeId(input) { - // Remove any non-alphanumeric characters except underscore (_) and hyphen (-) - let sanitized = input.replace(/[^a-zA-Z0-9-_]/g, ''); - - // Ensure first character is always a letter - if (!/^[a-zA-Z]/.test(sanitized)) { - sanitized = 'element_' + sanitized; - } - - return sanitized; -} - -function parseVoiceMap(voiceMapString) { - let parsedVoiceMap = {}; - for (const [charName, voiceId] of voiceMapString - .split(',') - .map(s => s.split(':'))) { - if (charName && voiceId) { - parsedVoiceMap[charName.trim()] = voiceId.trim(); - } - } - return parsedVoiceMap; -} - - - -/** - * Apply voiceMap based on current voiceMapEntries - */ -function updateVoiceMap() { - const tempVoiceMap = {}; - for (const voice of voiceMapEntries) { - if (voice.voiceId === null) { - continue; - } - tempVoiceMap[voice.name] = voice.voiceId; - } - if (Object.keys(tempVoiceMap).length !== 0) { - voiceMap = tempVoiceMap; - console.log(`Voicemap updated to ${JSON.stringify(voiceMap)}`); - } - if (!extension_settings.tts[ttsProviderName].voiceMap) { - extension_settings.tts[ttsProviderName].voiceMap = {}; - } - Object.assign(extension_settings.tts[ttsProviderName].voiceMap, voiceMap); - saveSettingsDebounced(); -} - -class VoiceMapEntry { - name; - voiceId; - selectElement; - constructor(name, voiceId = DEFAULT_VOICE_MARKER) { - this.name = name; - this.voiceId = voiceId; - this.selectElement = null; - } - - addUI(voiceIds) { - let sanitizedName = sanitizeId(this.name); - let defaultOption = this.name === DEFAULT_VOICE_MARKER ? - `` : - ``; - let template = ` -
- ${this.name} - -
- `; - $('#tts_voicemap_block').append(template); - - // Populate voice ID select list - for (const voiceId of voiceIds) { - const option = document.createElement('option'); - option.innerText = voiceId.name; - option.value = voiceId.name; - $(`#tts_voicemap_char_${sanitizedName}_voice`).append(option); - } - - this.selectElement = $(`#tts_voicemap_char_${sanitizedName}_voice`); - this.selectElement.on('change', args => this.onSelectChange(args)); - this.selectElement.val(this.voiceId); - } - - onSelectChange(args) { - this.voiceId = this.selectElement.find(':selected').val(); - updateVoiceMap(); - } -} - -/** - * Init voiceMapEntries for character select list. - * @param {boolean} unrestricted - If true, will include all characters in voiceMapEntries, even if they are not in the current chat. - */ -export async function initVoiceMap(unrestricted = false) { - // Gate initialization if not enabled or TTS Provider not ready. Prevents error popups. - const enabled = $('#tts_enabled').is(':checked'); - if (!enabled) { - return; - } - - // Keep errors inside extension UI rather than toastr. Toastr errors for TTS are annoying. - try { - await ttsProvider.checkReady(); - } catch (error) { - const message = `TTS Provider not ready. ${error}`; - setTtsStatus(message, false); - return; - } - - setTtsStatus('TTS Provider Loaded', true); - - // Clear existing voiceMap state - $('#tts_voicemap_block').empty(); - voiceMapEntries = []; - - // Get characters in current chat - const characters = getCharacters(unrestricted); - - // Get saved voicemap from provider settings, handling new and old representations - let voiceMapFromSettings = {}; - if ('voiceMap' in extension_settings.tts[ttsProviderName]) { - // Handle previous representation - if (typeof extension_settings.tts[ttsProviderName].voiceMap === 'string') { - voiceMapFromSettings = parseVoiceMap(extension_settings.tts[ttsProviderName].voiceMap); - // Handle new representation - } else if (typeof extension_settings.tts[ttsProviderName].voiceMap === 'object') { - voiceMapFromSettings = extension_settings.tts[ttsProviderName].voiceMap; - } - } - - // Get voiceIds from provider - let voiceIdsFromProvider; - try { - voiceIdsFromProvider = await ttsProvider.fetchTtsVoiceObjects(); - } - catch { - toastr.error('TTS Provider failed to return voice ids.'); - } - - // Build UI using VoiceMapEntry objects - for (const character of characters) { - if (character === 'SillyTavern System') { - continue; - } - // Check provider settings for voiceIds - let voiceId; - if (character in voiceMapFromSettings) { - voiceId = voiceMapFromSettings[character]; - } else if (character === DEFAULT_VOICE_MARKER) { - voiceId = DISABLED_VOICE_MARKER; - } else { - voiceId = DEFAULT_VOICE_MARKER; - } - const voiceMapEntry = new VoiceMapEntry(character, voiceId); - voiceMapEntry.addUI(voiceIdsFromProvider); - voiceMapEntries.push(voiceMapEntry); - } - updateVoiceMap(); -} - -$(document).ready(function () { - function addExtensionControls() { - const settingsHtml = ` -
-
-
- TTS -
-
-
-
-
- Select TTS Provider
-
- - -
-
- - - - - - - - -
-
-
-
-
-
-
- -
-
-
-
- - `; - $('#extensions_settings').append(settingsHtml); - $('#tts_refresh').on('click', onRefreshClick); - $('#tts_enabled').on('click', onEnableClick); - $('#tts_narrate_dialogues').on('click', onNarrateDialoguesClick); - $('#tts_narrate_quoted').on('click', onNarrateQuotedClick); - $('#tts_narrate_translated_only').on('click', onNarrateTranslatedOnlyClick); - $('#tts_skip_codeblocks').on('click', onSkipCodeblocksClick); - $('#tts_pass_asterisks').on('click', onPassAsterisksClick); - $('#tts_auto_generation').on('click', onAutoGenerationClick); - $('#tts_narrate_user').on('click', onNarrateUserClick); - $('#tts_voices').on('click', onTtsVoicesClick); - for (const provider in ttsProviders) { - $('#tts_provider').append($('