bigbird version

[liliuhaha]

I have tried to train a bigbird version on transformer， the training speed is extremely slow than the original transformer. I'd like to know your training ways for bigbird version. hope your reply! thanks

[minjaf]

I have tried to train a bigbird version on transformer， the training speed is extremely slow than the original transformer. I'd like to know your training ways for bigbird version. hope your reply! thanks

Hi! Please follow #22 . TL;DR: pretraining code, including space attention, is here

[yurakuratov]

We have two versions of the GENA-LM model based on the BigBird architecture:

• gena-lm-bigbird-base-t2t uses the HuggingFace implementation and is not very efficient.
• gena-lm-bigbird-base-sparse-t2t uses sparse operations from DeepSpeed, so to use it you should install it and set up your environment (refer to https://huggingface.co/AIRI-Institute/gena-lm-bigbird-base-sparse-t2t#deepspeed and discussion here: Problem with DeepSpeed for BigBird Sparse #12).

[liliuhaha]

thank your quick reply very much！ yes， i first try to use bigbird of huggingface transformer， and find the training speed extremely slow which even takes four times more time than normal version. First, i'd like to know if there is a similar phenomenon for the training efficiency of huggingface bigbird. In addition, i am trying to follow your code for deepspeed sparse attention, meeting some failure. I want to know how is the efficiency of deepspeed sparse attention compared to huggingface version and original transformer. Theoretically, the purpose of sparse attention is to reduce the training time, however the actual use of sparse attention is not consistent to the theory. is there better implementation for sparse attention? hope your reply, thanks.

[liliuhaha]

Triton 1.0.0 ,deepspeed 0.6.0, torch 1.13.1+cu117. This is my environment， it failed.

[yurakuratov]

In my experience, DeepSpeed's block-sparse attention usually gives about 2-4x faster training compared to the Hugging Face BigBird implementation. I don't have exact numbers right now, but overall, it's been noticeably more efficient in all use cases.

Could you provide more details on what do you mean by "it failed"? What is the output of ds_report?

[liliuhaha]

This is my ds_report:

DeepSpeed C++/CUDA extension op report

NOTE: Ops not installed will be just-in-time (JIT) compiled at
runtime if needed. Op compatibility means that your system
meet the required dependencies to JIT install the op.

JIT compiled ops requires ninja
ninja .................. [OKAY]

op name ................ installed .. compatible

cpu_adam ............... [NO] ....... [OKAY]
cpu_adagrad ............ [NO] ....... [OKAY]
fused_adam ............. [NO] ....... [OKAY]
fused_lamb ............. [NO] ....... [OKAY]
sparse_attn ............ [YES] ...... [OKAY]
transformer ............ [NO] ....... [OKAY]
stochastic_transformer . [NO] ....... [OKAY]
[WARNING] async_io requires the dev libaio .so object and headers but these were not found.
[WARNING] async_io: please install the libaio-dev package with apt
[WARNING] If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
async_io ............... [NO] ....... [NO]
utils .................. [NO] ....... [OKAY]
quantizer .............. [NO] ....... [OKAY]
transformer_inference .. [NO] ....... [OKAY]

DeepSpeed general environment info:
torch install path ............... ['/data1/lx/.conda/envs/dnabert/lib/python3.8/site-packages/torch']
torch version .................... 1.13.1+cu117
torch cuda version ............... 11.7
torch hip version ................ None
nvcc version ..................... 11.7
deepspeed install path ........... ['/data1/lx/.conda/envs/dnabert/lib/python3.8/site-packages/deepspeed']
deepspeed info ................... 0.6.0, unknown, unknown
deepspeed wheel compiled w. ...... torch 1.13, cuda 11.7, hip 0.0