Update README.md
DeepSpeed 11.2 instructions
This commit is contained in:
erew123 authored and GitHub committed
1 parent
d9cf355b79
commit
1c29a8391d
1 file changed
+98
@@ -16,6 +16,8 @@ The latest build (13 Dec 2023) has had the entire text filtering engine and narr
|
||||
|
||||
Should you want the older version of the narrator engine+text filtering, I will leave this older copy [here](https://github.com/erew123/alltalk_tts/releases/tag/v1-old-narrator)
|
||||
|
||||
|
||||
|
||||
#### The one thing I cant easily work around
|
||||
With a RP chat with your AI, **on your character card** `parameters menu` > `character tab` > `greeting` make sure that anything in there that is the **narrator is in asterisks** and anything **spoken is in double quotes**, then hit the `save` (disk) button. Greeting paragraphs/sentences are handled differently from how the AI sends text and so its difficut to account for them both.
|
||||
|
||||
@@ -72,3 +74,99 @@ To start AllTalk every Text generation webUI loads, edit the Text generation web
|
||||
## Screenshots
|
||||
| |  |
|
||||
|:---:|:---:|
|
||||
|
||||
## DeepSpeed 11.2 (for Windows & Python 3.11)
|
||||
DeepSpeed v11.2 will work on the current text-generation-webui Python 3.11 environment!
|
||||
|
||||
Thanks to [@S95Sedan](https://github.com/S95Sedan) - They managed to get DeepSpeed 11.2 working on Windows via making some edits to the original Microsoft DeepSpeed v11.2 installation. The original post is [here](https://github.com/oobabooga/text-generation-webui/issues/4734#issuecomment-1843984142).
|
||||
|
||||
#### Pre-Compiled Wheel (for Windows and Python 3.11) - Quick and easy!
|
||||
[@S95Sedan](https://github.com/S95Sedan) has kindly provided a pre-compiled wheel file, which you can download and use [deepspeed-0.11.1+e9503fe-cp311-cp311-win_amd64.rar.zip](https://github.com/oobabooga/text-generation-webui/files/13593455/deepspeed-0.11.1%2Be9503fe-cp311-cp311-win_amd64.rar.zip). To use this, you will need to:
|
||||
|
||||
1) Download the file and put it inside your **text-generation-webui** folder.
|
||||
|
||||
3) Extract out the zip file [deepspeed-0.11.1+e9503fe-cp311-cp311-win_amd64.rar.zip](https://github.com/oobabooga/text-generation-webui/files/13593455/deepspeed-0.11.1%2Be9503fe-cp311-cp311-win_amd64.rar.zip), which will give you a RAR file *(this is because github wont allow rar files, only zip, so it had to be compressed twice)*.
|
||||
|
||||
5) Extract out the rar file `deepspeed-0.11.1+e9503fe-cp311-cp311-win_amd64.rar`.
|
||||
|
||||
7) That should now have extracted a file called `deepspeed-0.11.1+e9503fe-cp311-cp311-win_amd64.whl` **note the .whl extension on it**.
|
||||
|
||||
9) Still in the **text-generation-webui folder**, you can now start the Python environment for text-generation-webui:
|
||||
|
||||
`cmd_windows.bat`
|
||||
|
||||
6) Move into the folder where the `whl` file was extracted to and then `pip install "deepspeed-0.11.1+e9503fe-cp311-cp311-win_amd64.whl"`
|
||||
|
||||
8) This should install through cleanly and you should now have DeepSpeed 11.2 installed within the Python 3.11 environment of text-generation-webui.
|
||||
|
||||
10) When you start up text-generation-webui, you should note that AllTalk's startup says **[AllTalk Startup] DeepSpeed Detected**
|
||||
|
||||
12) Within AllTalk, you will now have a checkbox for **Activate DeepSpeed" though remember you can only change 1x setting every 15 or so seconds, so dont try to activate DeepSpeed **and* LowVRAM/Change your model simultantiously. Do one of those, wait 15-20 seconds until the change is confirmed in the console, then you can change the other. When you are happy it works, you can set the default start-up settings in the settings page.
|
||||
|
||||
#### Manual Build (for Windows and Python 3.11) - A bit more complicated!
|
||||
To perform a manual build of DeepSpeed 11.2, you would follow the instructions for creating DeepSpeed v8.3, but in its place, you would download DeepSpeed v11.2 **Source code (zip)** [here](https://github.com/microsoft/DeepSpeed/releases/tag/v0.11.2) **and** you would use the text-generation-webui's Python environment `cmd_windows.bat`. Extract the downloaded file and you would have to make some file edits to the files in that folder before you can compile DeepSpeed v11.2. As Follows:
|
||||
|
||||
**DeepSpeed-0.11.2\build_win.bat** At the top of the file, add:
|
||||
|
||||
`set DS_BUILD_EVOFORMER_ATTN=0`
|
||||
|
||||
**DeepSpeed-0.11.2\csrc\quantization\pt_binding.cpp - lines 244-250** change to:
|
||||
|
||||
```
|
||||
std::vector<int64_t> sz_vector(input_vals.sizes().begin(), input_vals.sizes().end());
|
||||
sz_vector[sz_vector.size() - 1] = sz_vector.back() / devices_per_node; // num of GPU per nodes
|
||||
at::IntArrayRef sz(sz_vector);
|
||||
auto output = torch::empty(sz, output_options);
|
||||
|
||||
const int elems_per_in_tensor = at::numel(input_vals) / devices_per_node;
|
||||
const int elems_per_in_group = elems_per_in_tensor / (in_groups / devices_per_node);
|
||||
const int elems_per_out_group = elems_per_in_tensor / out_groups;
|
||||
```
|
||||
|
||||
**DeepSpeed-0.11.2\csrc\transformer\inference\csrc\pt_binding.cpp - lines 541-542** change to:
|
||||
|
||||
```
|
||||
{static_cast<unsigned>(hidden_dim * InferenceContext::Instance().GetMaxTokenLength()),
|
||||
static_cast<unsigned>(k * InferenceContext::Instance().GetMaxTokenLength()),
|
||||
```
|
||||
|
||||
**DeepSpeed-0.11.2\csrc\transformer\inference\csrc\pt_binding.cpp - lines 550-551** change to:
|
||||
|
||||
```
|
||||
{static_cast<unsigned>(hidden_dim * InferenceContext::Instance().GetMaxTokenLength()),
|
||||
static_cast<unsigned>(k * InferenceContext::Instance().GetMaxTokenLength()),
|
||||
```
|
||||
**DeepSpeed-0.11.2\csrc\transformer\inference\csrc\pt_binding.cpp - line 1581** change to:
|
||||
|
||||
```
|
||||
at::from_blob(intermediate_ptr, {input.size(0), input.size(1), static_cast<int64_t>(mlp_1_out_neurons)}, options);
|
||||
```
|
||||
|
||||
**DeepSpeed-0.11.2\deepspeed\env_report.py - line 10** add:
|
||||
|
||||
```
|
||||
import psutil
|
||||
```
|
||||
|
||||
**DeepSpeed-0.11.2\deepspeed\env_report.py - line 83 - 100** change to:
|
||||
|
||||
```
|
||||
def get_shm_size():
|
||||
try:
|
||||
temp_dir = os.getenv('TEMP') or os.getenv('TMP') or os.path.join(os.path.expanduser('~'), 'tmp')
|
||||
shm_stats = psutil.disk_usage(temp_dir)
|
||||
shm_size = shm_stats.total
|
||||
shm_hbytes = human_readable_size(shm_size)
|
||||
warn = []
|
||||
if shm_size < 512 * 1024**2:
|
||||
warn.append(
|
||||
f" {YELLOW} [WARNING] Shared memory size might be too small, consider increasing it. {END}"
|
||||
)
|
||||
# Add additional warnings specific to your use case if needed.
|
||||
return shm_hbytes, warn
|
||||
except Exception as e:
|
||||
return "UNKNOWN", [f"Error getting shared memory size: {e}"]
|
||||
```
|
||||
|
||||
You can now compile DeepSpeed and build your whl (wheel) file.
|
||||
|
||||
Reference in new issue
Block a user