Codes for ECBSR

Edge-oriented Convolution Block for Real-time Super Resolution on Mobile Devices
Xindong Zhang, Hui Zeng, Lei Zhang
ACM Multimedia 2021

Codes

An older version implemented based on EDSR is place on /legacy folder. For more details, please refer to /legacy/README.md. The following is the lighten version implemented by us.

Dependencies & Installation

Please refer to the following simple steps for installation.

git clone https://github.com/xindongzhang/ECBSR.git
cd ECBSR
pip install -r requirements.txt

Training and benchmarking data can be downloaded from DIV2K and benchmark, respectively. Thanks for excellent work by EDSR.

Training & Testing

You could also try less/larger batch-size, if there are limited/enough hardware resources in your GPU-server. ECBSR is trained and tested with colors=1, e.g Y channel out of Ycbcr.

cd ECBSR

## ecbsr-m4c8-x2-prelu(you can revise the parameters of the yaml-config file accordding to your environments)
python train.py --config ./configs/ecbsr_x2_m4c8_prelu.yml

## ecbsr-m4c8-x4-prelu
python train.py --config ./configs/ecbsr_x4_m4c8_prelu.yml

## ecbsr-m4c16-x2-prelu
python train.py --config ./configs/ecbsr_x2_m4c16_prelu.yml

## ecbsr-m4c16-x4-prelu
python train.py --config ./configs/ecbsr_x4_m4c16_prelu.yml

Hardware deployment

Frontend conversion

We provide convertor for model conversion to different frontend, e.g. onnx/pb/tflite. We currently developed and tested the model with only one-channel(Y out of Ycbcr). Since the internal data-layout are quite different between tf(NHWC) and pytorch(NCHW), espetially for the pixelshuffle operation. Care must be taken to handle the data-layout, if you want to extend the pytorch-based training framework to RGB input data and deploy it on tensorflow. Follow are the demo scripts for model conversion to specific frontend:

## convert the trained pytorch model to onnx with plain-topology.
python convert.py --config xxx.yml --target_frontend onnx --output_folder XXX --inp_n 1 --inp_c 1 --inp_h 270 --inp_w 480

## convert the trained pytorch model to pb-1.x with plain-topology.
python convert.py --config xxx.yml --target_frontend pb-1.x --output_folder XXX --inp_n 1 --inp_c 1 --inp_h 270 --inp_w 480

## convert the trained pytorch model to pb-ckpt with plain-topology
python convert.py --config xxx.yml --target_frontend pb-ckpt --output_folder XXX --inp_n 1 --inp_c 1 --inp_h 270 --inp_w 480

AI-Benchmark

You can download the newest version of evaluation tool from AI-Benchmark. Then you can install the app via ADB tools,

adb install -r [name-of-ai-benchmar].apk

MNN (Come soon!)

For universal CPU & GPU of mobile hardware implementation.

RKNN (Come soon!)

For NPU inplementation of Rockchip hardware, e.g. RK3399Pro/RK1808.

MiniNet (Come soon!)

A super light-weight CNN inference framework implemented by us, with only conv-3x3, element-wise op, ReLU(PReLU) activations, and pixel-shuffle for common super resolution task. For more details, please refer to /ECBSR/deploy/mininet

Quantization tools (Come soon!)

For fixed-arithmetic quantization of image super resolution.

Citation

@article{zhang2021edge,
  title={Edge-oriented Convolution Block for Real-time Super Resolution on Mobile Devices},
  author={Zhang, Xindong and Zeng, Hui and Zhang, Lei},
  booktitle={Proceedings of the 29th ACM International Conference on Multimedia (ACM MM)},
  year={2021}
}

Acknowledgement

Thanks EDSR for the pioneering work and excellent codebase! The implementation integrated with EDSR is placed on /legacy