diff --git a/README.md b/README.md index 026cc66..c98d3e0 100644 --- a/README.md +++ b/README.md @@ -16,6 +16,8 @@ The latest build (13 Dec 2023) has had the entire text filtering engine and narr Should you want the older version of the narrator engine+text filtering, I will leave this older copy [here](https://github.com/erew123/alltalk_tts/releases/tag/v1-old-narrator) + + #### The one thing I cant easily work around With a RP chat with your AI, **on your character card** `parameters menu` > `character tab` > `greeting` make sure that anything in there that is the **narrator is in asterisks** and anything **spoken is in double quotes**, then hit the `save` (disk) button. Greeting paragraphs/sentences are handled differently from how the AI sends text and so its difficut to account for them both. @@ -72,3 +74,99 @@ To start AllTalk every Text generation webUI loads, edit the Text generation web ## Screenshots |![image](https://github.com/erew123/alltalk_tts/assets/35898566/a4d983ab-f9e1-42dd-94ee-a85043f74ab2) | ![image](https://github.com/erew123/alltalk_tts/assets/35898566/3497d656-9729-4cb7-8d0d-6367078794ee) | |:---:|:---:| + +## DeepSpeed 11.2 (for Windows & Python 3.11) +DeepSpeed v11.2 will work on the current text-generation-webui Python 3.11 environment! + +Thanks to [@S95Sedan](https://github.com/S95Sedan) - They managed to get DeepSpeed 11.2 working on Windows via making some edits to the original Microsoft DeepSpeed v11.2 installation. The original post is [here](https://github.com/oobabooga/text-generation-webui/issues/4734#issuecomment-1843984142). + +#### Pre-Compiled Wheel (for Windows and Python 3.11) - Quick and easy! +[@S95Sedan](https://github.com/S95Sedan) has kindly provided a pre-compiled wheel file, which you can download and use [deepspeed-0.11.1+e9503fe-cp311-cp311-win_amd64.rar.zip](https://github.com/oobabooga/text-generation-webui/files/13593455/deepspeed-0.11.1%2Be9503fe-cp311-cp311-win_amd64.rar.zip). To use this, you will need to: + +1) Download the file and put it inside your **text-generation-webui** folder. + +3) Extract out the zip file [deepspeed-0.11.1+e9503fe-cp311-cp311-win_amd64.rar.zip](https://github.com/oobabooga/text-generation-webui/files/13593455/deepspeed-0.11.1%2Be9503fe-cp311-cp311-win_amd64.rar.zip), which will give you a RAR file *(this is because github wont allow rar files, only zip, so it had to be compressed twice)*. + +5) Extract out the rar file `deepspeed-0.11.1+e9503fe-cp311-cp311-win_amd64.rar`. + +7) That should now have extracted a file called `deepspeed-0.11.1+e9503fe-cp311-cp311-win_amd64.whl` **note the .whl extension on it**. + +9) Still in the **text-generation-webui folder**, you can now start the Python environment for text-generation-webui: + +`cmd_windows.bat` + +6) Move into the folder where the `whl` file was extracted to and then `pip install "deepspeed-0.11.1+e9503fe-cp311-cp311-win_amd64.whl"` + +8) This should install through cleanly and you should now have DeepSpeed 11.2 installed within the Python 3.11 environment of text-generation-webui. + +10) When you start up text-generation-webui, you should note that AllTalk's startup says **[AllTalk Startup] DeepSpeed Detected** + +12) Within AllTalk, you will now have a checkbox for **Activate DeepSpeed" though remember you can only change 1x setting every 15 or so seconds, so dont try to activate DeepSpeed **and* LowVRAM/Change your model simultantiously. Do one of those, wait 15-20 seconds until the change is confirmed in the console, then you can change the other. When you are happy it works, you can set the default start-up settings in the settings page. + +#### Manual Build (for Windows and Python 3.11) - A bit more complicated! +To perform a manual build of DeepSpeed 11.2, you would follow the instructions for creating DeepSpeed v8.3, but in its place, you would download DeepSpeed v11.2 **Source code (zip)** [here](https://github.com/microsoft/DeepSpeed/releases/tag/v0.11.2) **and** you would use the text-generation-webui's Python environment `cmd_windows.bat`. Extract the downloaded file and you would have to make some file edits to the files in that folder before you can compile DeepSpeed v11.2. As Follows: + +**DeepSpeed-0.11.2\build_win.bat** At the top of the file, add: + +`set DS_BUILD_EVOFORMER_ATTN=0` + +**DeepSpeed-0.11.2\csrc\quantization\pt_binding.cpp - lines 244-250** change to: + +``` + std::vector sz_vector(input_vals.sizes().begin(), input_vals.sizes().end()); + sz_vector[sz_vector.size() - 1] = sz_vector.back() / devices_per_node; // num of GPU per nodes + at::IntArrayRef sz(sz_vector); + auto output = torch::empty(sz, output_options); + + const int elems_per_in_tensor = at::numel(input_vals) / devices_per_node; + const int elems_per_in_group = elems_per_in_tensor / (in_groups / devices_per_node); + const int elems_per_out_group = elems_per_in_tensor / out_groups; +``` + +**DeepSpeed-0.11.2\csrc\transformer\inference\csrc\pt_binding.cpp - lines 541-542** change to: + +``` + {static_cast(hidden_dim * InferenceContext::Instance().GetMaxTokenLength()), + static_cast(k * InferenceContext::Instance().GetMaxTokenLength()), +``` + +**DeepSpeed-0.11.2\csrc\transformer\inference\csrc\pt_binding.cpp - lines 550-551** change to: + +``` + {static_cast(hidden_dim * InferenceContext::Instance().GetMaxTokenLength()), + static_cast(k * InferenceContext::Instance().GetMaxTokenLength()), +``` +**DeepSpeed-0.11.2\csrc\transformer\inference\csrc\pt_binding.cpp - line 1581** change to: + +``` + at::from_blob(intermediate_ptr, {input.size(0), input.size(1), static_cast(mlp_1_out_neurons)}, options); +``` + +**DeepSpeed-0.11.2\deepspeed\env_report.py - line 10** add: + +``` +import psutil +``` + +**DeepSpeed-0.11.2\deepspeed\env_report.py - line 83 - 100** change to: + +``` +def get_shm_size(): + try: + temp_dir = os.getenv('TEMP') or os.getenv('TMP') or os.path.join(os.path.expanduser('~'), 'tmp') + shm_stats = psutil.disk_usage(temp_dir) + shm_size = shm_stats.total + shm_hbytes = human_readable_size(shm_size) + warn = [] + if shm_size < 512 * 1024**2: + warn.append( + f" {YELLOW} [WARNING] Shared memory size might be too small, consider increasing it. {END}" + ) + # Add additional warnings specific to your use case if needed. + return shm_hbytes, warn + except Exception as e: + return "UNKNOWN", [f"Error getting shared memory size: {e}"] +``` + +You can now compile DeepSpeed and build your whl (wheel) file. +