# MSVQA

**Repository Path**: horizon-mind/msvqa

## Basic Information

- **Project Name**: MSVQA
- **Description**: MS Model for Video Question Answering
- **Primary Language**: Python
- **License**: Not specified
- **Default Branch**: master
- **Homepage**: None
- **GVP Project**: No

## Statistics

- **Stars**: 0
- **Forks**: 0
- **Created**: 2023-08-05
- **Last Updated**: 2024-04-16

## Categories & Tags

**Categories**: Uncategorized

**Tags**: None

## README

# MSVQA：MS Model for Video Question Answering

![msvqa](./img/msvqa.png)

## What to prepare

1. prepare MSVD-QA dataset and place it to `./data/msvd-qa`
1. download resnext101 pretrained model from [here](https://drive.google.com/drive/folders/1zvl89AgFAApbH0At-gMuZSeQB_LpNP-M), and put it in `./data/resnext-101-kinetics.pth` (path can be changed in `base_options.py`)
1. download `glove.840B.300d.txt` to `./data/glove/glove.840B.300d`

## How to run

### preprocess

1. process `glove.840B.300d.txt` into a pickle file

   ```sh
    python txt2pickle.py
   ```

2. preprocess questions

   ```sh
   python preprocess/preprocess_questions.py --dataset msvd-qa --glove_pt data/glove/glove.840.300d.pkl --mode train
   
   python preprocess/preprocess_questions.py --dataset msvd-qa --mode val
   
   python preprocess/preprocess_questions.py --dataset msvd-qa --mode test
   ```

3. preprocess videos

   ```shell
   python preprocess/preprocess_videos.py --annotation_file data/msvd-qa/train_qa.json --model resnet101
   
   python preprocess/preprocess_videos.py --annotation_file data/msvd-qa/train_qa.json --model resnext101
   
   python preprocess/preprocess_videos.py --annotation_file data/msvd-qa/val_qa.json --model resnet101
   
   python preprocess/preprocess_videos.py --annotation_file data/msvd-qa/val_qa.json --model resnext101
   
   python preprocess/preprocess_videos.py --annotation_file data/msvd-qa/test_qa.json --model resnet101
   
   python preprocess/preprocess_videos.py --annotation_file data/msvd-qa/test_qa.json --model resnext101
   ```

> `--output_dir /root/autodl-fs`

it should look like this:

```
# pwd
./data/msvd-qa
# ls -la
total 11756
drwxr-xr-x 3 root root     151 Aug 28 20:34 .
drwxr-xr-x 4 root root      82 Aug 28 20:28 ..
-rw-r--r-- 1 root root 8901078 Aug 28 20:29 msvd-qa_train_questions.pt
-rw-r--r-- 1 root root  165926 Aug 28 20:29 msvd-qa_vocab.json
-rw-r--r-- 1 root root 2796877 Aug 28 20:32 train_qa.json
drwxr-xr-x 2 root root   77824 Aug 28 20:34 video
-rw-r--r-- 1 root root   50486 Aug 28 20:33 youtube_mapping.txt
```

```
# pwd
./data
# ls -la
total 378324
drwxr-xr-x  4 root root        82 Aug 28 20:28 .
drwxr-xr-x 12 root root       298 Aug 28 20:27 ..
drwxr-xr-x  2 root root        35 Aug 28 20:25 glove
drwxr-xr-x  3 root root       151 Aug 28 20:34 msvd-qa
-rwxr-xr-x  1 root root 387403754 Aug 28 20:27 resnext-101-kinetics.pth
```

### get help

```bash
python train.py --help
```

### train

```bash
python train.py --msvd_data_path ./data/msvd_qa_data
```

## References

![](./img/DualVGRUnit.png)

![](./img/vigan.png)

![](./img/CycleGAN.png)

![](./img/AE.png)

![](./img/AIMC.png)