I write a model using Pytorch, and save its state_dict() to .pth. Now I want to use tensorlayerx to write it, so other people (using tensorflow etc.) can use this model. My model definition is same in Pytorch and Tensorlayerx, but I can't load pretrained model of .pth in tensorlayerx. Below is my code. (simple model is used here for clarity, the actual model is more complex than this)

"""
a_torch.py
"""
import torch
from torch import nn

class A(nn.Module):
    def __init__(self):
        super(A, self).__init__()
        self.conv = nn.Conv2d(3, 16, kernel_size=1)
        self.bn = nn.BatchNorm2d(16)
        self.relu = nn.ReLU(inplace=True)
    
    def forward(self, x):
        return self.act(self.bn(self.conv(x)))

if __name__ == '__main__':
    a = A()
    torch.save(a.state_dict(), 'a.pth')

"""
a_tlx.py
"""
import tensorlayerx as tlx
import torch
from tensorlayerx import nn

class A(nn.Module):
    def __init__(self):
        super(A, self).__init__()
        self.conv = nn.Conv2d(16, kernel_size=1, data_format='channels_first')
        self.bn = nn.BatchNorm2d(num_features=16, data_format='channels_first')
        self.relu = nn.activation.ReLU()
    
    def forward(self, x):
        return self.act(self.bn(self.conv(x)))

def pth2npz(pth_path):
    temp = torch.load(pth_path)   # type(temp) = OrderedDict
    tlx.files.save_npz_dict(temp.items(), pth_path.split('.')[0] + '.npz')

if __name__ == '__main__':
    a = A()
    pth2npz('a.pth')
    tlx.files.load_and_assign_npz_dict('a.npz', a)

First run a_torch.py, then run a_tlx.py. The error is below.

Using PyTorch backend.
Traceback (most recent call last):
  File "test/test_03.py", line 25, in <module>
    tlx.files.load_and_assign_npz_dict('test/a.npz', a)
  File "/home/mchen/anaconda3/envs/kpconv/lib/python3.8/site-packages/tensorlayerx/files/utils.py", line 2208, in load_and_assign_npz_dict
    raise RuntimeError(
RuntimeError: Weights named 'conv.weight' not found in network. Hint: set argument skip=Ture if you want to skip redundant or mismatch weights

Then I debug and look at the tlx.files.load_and_assign_npz_dict() source code. I find tensorlayerx parameter name is different from PyTorch. This results in key mismatch when loading pre-trained model. In the following two figures, the first is the parameter name of PyTorch and the second is the parameter name of TensorLayerx. Now the solution I can think of is to write a key map table, but it is hard for large model. So can you give me a simple solution ? (same model definition in pytorch and tensorlayerx, load pretrained model in .pth) :grin:

tlx：

[-0.16246916, 1.40204561, 0.85213524, ..., 0.85800600, 1.10605156, 1.11549926], [-0.04873780, 0.28885114, 0.15792340, ..., 0.12375022, 0.22599602, 0.53073120], [-0.09840852, 0.40172467, 0.15602632, ..., 0.09853011, 0.29177830, 0.52241892]

paddle:

[-0.16246916, 1.40204573, 0.85213524, ..., 0.85800600, 1.10605145, 1.11549926], [-0.04873780, 0.28885114, 0.15792342, ..., 0.12375022, 0.22599602, 0.53073120], [-0.09840852, 0.40172467, 0.15602632, ..., 0.09853011, 0.29177833, 0.52241892]

Checklist

• [ ] I've tested that my changes are compatible with the latest version of Tensorflow.
• [ ] I've read the Contribution Guidelines
• [ ] I've updated the documentation if necessary.

Motivation and Context

Description

Checklist

• [ ] I've tested that my changes are compatible with the latest version of Tensorflow.
• [ ] I've read the Contribution Guidelines
• [ ] I've updated the documentation if necessary.

Motivation and Context

Description

Checklist

• [ ] I've tested that my changes are compatible with the latest version of Tensorflow.
• [ ] I've read the Contribution Guidelines
• [ ] I've updated the documentation if necessary.

Motivation and Context

Description

Checklist

• [ ] I've tested that my changes are compatible with the latest version of Tensorflow.
• [ ] I've read the Contribution Guidelines
• [ ] I've updated the documentation if necessary.

Motivation and Context

Description

Checklist

• [ ] I've tested that my changes are compatible with the latest version of Tensorflow.
• [ ] I've read the Contribution Guidelines
• [ ] I've updated the documentation if necessary.

Motivation and Context

Description

Checklist

• [ ] I've tested that my changes are compatible with the latest version of Tensorflow.
• [ ] I've read the Contribution Guidelines
• [ ] I've updated the documentation if necessary.

Motivation and Context

Description

Description

oneflow backend:

backends/ops/oneflow_nn.py
backends/ops/oneflow_backend.py
nn/core/core_oneflow.py

tutorials: 6 MarkDown files in /home/user/pyprojects/TensorLayerX/docs/tutorials

other: Training progressbar using rich bugs fixed

Checklist

• [ ] I've tested that my changes are compatible with the latest version of Tensorflow.
• [ ] I've read the Contribution Guidelines
• [ ] I've updated the documentation if necessary.

Motivation and Context

Description

Checklist

• [ ] I've tested that my changes are compatible with the latest version of Tensorflow.
• [ ] I've read the Contribution Guidelines
• [ ] I've updated the documentation if necessary.

Motivation and Context

Description

Issue Description

When I test my pspnet model, I find if not use "with torch.no_grad()" or "gradient()", the gpu memory will be full after testing several photos. I guess set_eval() function seems to have failed. Or I used the wrong method to test? This is my code, thank you!

In addition, I found that the batch size will affect the final test results. If net. eval() is not performed in the pytorch, it will cause similar problems. It seems that this is caused by the BatchNorm layer.

    os.environ['TL_BACKEND'] = 'torch'
    tlx.set_device(device='GPU', id=3)
    # ...
    net = models[backend]()
    net.load_weights('test.npz', format='npz_dict', skip=True)
    test_dataset = MyDataset(root_dir="test/")
    test_loader = DataLoader(test_dataset, batch_size=4, shuffle=True)

    train_weights = net.trainable_weights
    scheduler = tlx.optimizers.lr.StepDecay(learning_rate=0, step_size=30, gamma=0.5, last_epoch=-1)
    optimizer = tlx.optimizers.Adam(lr=scheduler)

    hist = np.zeros((num_classes, num_classes))
    net.set_eval()
    # with torch.no_grad():
    for x, y, y_cls in test_loader:
        _out, _out_cls = net(x)
        seg_loss = tlx.losses.softmax_cross_entropy_with_logits(_out, y)
        cls_loss = tlx.losses.sigmoid_cross_entropy(_out_cls, y_cls)
        _loss = seg_loss + 1 * cls_loss
        # grads = optimizer.gradient(_loss, train_weights)
        # optimizer.apply_gradients(zip(grads, train_weights))
        '''
            compute miou matrix
        '''
        out = tlx.convert_to_numpy(_out)
        y = tlx.convert_to_numpy(y)
        out = np.argmax(out, axis=1)
        for i in range(0, out.shape[0]):
            pred = out[i]
            gt = y[i]
            hist += fast_hist(gt.flatten(), pred.flatten(), num_classes)
            
    # compute miou then print
    mIoUs = per_class_iu(hist)
    for ind_class in range(num_classes):
        print('===>' + name_classes[ind_class] + ':\t' + str(round(mIoUs[ind_class] * 100, 2)))
    print('===> mIoU: ' + str(round(np.nanmean(mIoUs) * 100, 2)))
    print("test loss: {}".format(train_loss))

New Issue Checklist

• [ ] I have read the Contribution Guidelines
• [ ] I searched for existing GitHub issues

Issue Description

[INSERT DESCRIPTION OF THE PROBLEM]

Reproducible Code

• Which OS are you using ?
• Please provide a reproducible code of your issue. Without any reproducible code, you will probably not receive any help.

[INSERT CODE HERE]

# ======================================================== #
###### tensorlayerx.ops.Pad源码######
# ======================================================== #

class Pad(object):

    def __init__(self, paddings, mode="REFLECT", constant_values=0):
        if mode not in ['CONSTANT', 'REFLECT', 'SYMMETRIC']:
            raise Exception("Unsupported mode: {}".format(mode))
        if mode == 'SYMMETRIC':
            raise NotImplementedError
        self.paddings = paddings
        self.mode = mode.lower()
        self.constant_values = constant_values

    def __call__(self, x):
        if len(x.shape) == 3:
            data_format = 'NLC'
            self.paddings = self.correct_paddings(len(x.shape), self.paddings, data_format)
        elif len(x.shape) == 4:
            data_format = 'NHWC'
            self.paddings = self.correct_paddings(len(x.shape), self.paddings, data_format)
        elif len(x.shape) == 5:
            data_format = 'NDHWC'
            self.paddings = self.correct_paddings(len(x.shape), self.paddings, data_format)
        else:
            raise NotImplementedError('Please check the input shape.')
        return pd.nn.functional.pad(x, self.paddings, self.mode, value=self.constant_values, data_format=data_format)

    def correct_paddings(self, in_shape, paddings, data_format):
        if in_shape == 3 and data_format == 'NLC':
            correct_output = [paddings[1][0], paddings[1][1]]
        elif in_shape == 4 and data_format == 'NHWC':
            correct_output = [paddings[2][0], paddings[2][1], paddings[1][0], paddings[1][1]]
        elif in_shape == 5 and data_format == 'NDHWC':
            correct_output = [
                paddings[3][0], paddings[3][1], paddings[2][0], paddings[2][1], paddings[1][0], paddings[1][1]
            ]
        else:
            raise NotImplementedError('Does not support channels first')
        return correct_output

paddle.nn.InstanceNorm2D(num_features, epsilon=1e-05, momentum=0.9, weight_attr=None, bias_attr=None, data_format="NCHW", name=None) 更多见接口文档https://www.paddlepaddle.org.cn/documentation/docs/zh/2.3/api/paddle/nn/InstanceNorm2D_cn.html#instancenorm2d

# ======================================================== #
###### THIS CODE IS AN EXAMPLE, REPLACE WITH YOUR OWN ######
# ======================================================== #
import tensorlayerx as tlx

New Issue Checklist

• [ ] I have read the Contribution Guidelines
• [ ] I searched for existing GitHub issues

Issue Description

[INSERT DESCRIPTION OF THE PROBLEM]

Reproducible Code

• Which OS are you using ?
• Please provide a reproducible code of your issue. Without any reproducible code, you will probably not receive any help.

[INSERT CODE HERE]

# ======================================================== #
###### THIS CODE IS AN EXAMPLE, REPLACE WITH YOUR OWN ######
# ======================================================== #
# paddle
import paddle
x = 13
print(isinstance(x, paddle.optimizers.Optimizer))

# tensorlayer
import os
os.environ['TL_BACKEND'] = 'paddle'
import tensorlayer as tlx
x = 13
print(isinstance(x, tlx.optimizers.paddle_optimizers.Optimizer))
# ======================================================== #
###### THIS CODE IS AN EXAMPLE, REPLACE WITH YOUR OWN ######
# ======================================================== #

New Issue Checklist

• [ ] I have read the Contribution Guidelines
• [ ] I searched for existing GitHub issues

Issue Description

[INSERT DESCRIPTION OF THE PROBLEM]

Reproducible Code

• Which OS are you using ?
• Please provide a reproducible code of your issue. Without any reproducible code, you will probably not receive any help.

[INSERT CODE HERE]

# ======================================================== #
###### THIS CODE IS AN EXAMPLE, REPLACE WITH YOUR OWN ######
# ======================================================== #
import os
import paddle
os.environ['TL_BACKEND'] = 'paddle'
import tensorlayerx as tlx

tlx_ni = tlx.nn.Input([4, 32, 50, 50], name='input')
tlx_out = tlx.nn.UpSampling2d(scale=(2, 2), data_format="channels_first")(tlx_ni)
print(f"tlx_out.shape={tlx_out.shape}")

pd_ni = paddle.rand([4, 32, 50, 50], dtype="float32")
pd_out = paddle.nn.Upsample(scale_factor=2, data_format="NCHW")(pd_ni)
print(f"pd_out.shape={pd_out.shape}")

# ======================================================== #
###### THIS CODE IS AN EXAMPLE, REPLACE WITH YOUR OWN ######
# ======================================================== #

输出结果 tlx_out.shape=[4, 32, 64, 100] pd_out.shape=[4, 32, 100, 100]