循環神經網絡的簡介實現

定義模型

我們使用Pytorch中的nn.RNN來構造循環神經網絡。在本節中，我們主要關注nn.RNN的以下幾個構造函數參數：

• input_size - The number of expected features in the input x
• hidden_size – The number of features in the hidden state h
• nonlinearity – The non-linearity to use. Can be either ‘tanh’ or ‘relu’. Default: ‘tanh’
• batch_first – If True, then the input and output tensors are provided as (batch_size, num_steps, input_size). Default: False
  這裏的batch_first決定了輸入的形狀，我們使用默認的參數False，對應的輸入形狀是 (num_steps, batch_size, input_size)。

forward函數的參數爲：

• input of shape (num_steps, batch_size, input_size): tensor containing the features of the input sequence.

• h_0 of shape (num_layers * num_directions, batch_size, hidden_size): tensor containing the initial hidden state for each element in the batch. Defaults to zero if not provided. If the RNN is bidirectional, num_directions should be 2, else it should be 1.

• forward函數的返回值是：

• output of shape (num_steps, batch_size, num_directions * hidden_size): tensor containing the output features (h_t) from the last layer of the RNN, for each t.

• h_n of shape (num_layers * num_directions, batch_size, hidden_size): tensor containing the hidden state for t = num_steps.
  現在我們構造一個nn.RNN實例，並用一個簡單的例子來看一下輸出的形狀。

rnn_layer = nn.RNN(input_size=vocab_size, hidden_size=num_hiddens)
num_steps, batch_size = 35, 2
X = torch.rand(num_steps, batch_size, vocab_size)
state = None
Y, state_new = rnn_layer(X, state)
print(Y.shape, state_new.shape)

torch.Size([35, 2, 256]) torch.Size([1, 2, 256])

我們定義一個完整的基於循環神經網絡的語言模型.

class RNNModel(nn.Module):
    def __init__(self, rnn_layer, vocab_size):
        super(RNNModel, self).__init__()
        self.rnn = rnn_layer
        self.hidden_size = rnn_layer.hidden_size * (2 if rnn_layer.bidirectional else 1) 
        self.vocab_size = vocab_size
        self.dense = nn.Linear(self.hidden_size, vocab_size)

    def forward(self, inputs, state):
        # inputs.shape: (batch_size, num_steps)
        X = to_onehot(inputs, vocab_size)
        X = torch.stack(X)  # X.shape: (num_steps, batch_size, vocab_size)
        hiddens, state = self.rnn(X, state)
        hiddens = hiddens.view(-1, hiddens.shape[-1])  # hiddens.shape: (num_steps * batch_size, hidden_size)
        output = self.dense(hiddens)
        return output, state

類似的，我們需要實現一個預測函數，與前面的區別在於前向計算和初始化隱藏狀態。

def predict_rnn_pytorch(prefix, num_chars, model, vocab_size, device, idx_to_char,
                      char_to_idx):
    state = None
    output = [char_to_idx[prefix[0]]]  # output記錄prefix加上預測的num_chars個字符
    for t in range(num_chars + len(prefix) - 1):
        X = torch.tensor([output[-1]], device=device).view(1, 1)
        (Y, state) = model(X, state)  # 前向計算不需要傳入模型參數
        if t < len(prefix) - 1:
            output.append(char_to_idx[prefix[t + 1]])
        else:
            output.append(Y.argmax(dim=1).item())
    return ''.join([idx_to_char[i] for i in output])

使用權重爲隨機值的模型來預測一次。

model = RNNModel(rnn_layer, vocab_size).to(device)
predict_rnn_pytorch('分開', 10, model, vocab_size, device, idx_to_char, char_to_idx)

def train_and_predict_rnn_pytorch(model, num_hiddens, vocab_size, device,
                                corpus_indices, idx_to_char, char_to_idx,
                                num_epochs, num_steps, lr, clipping_theta,
                                batch_size, pred_period, pred_len, prefixes):
    loss = nn.CrossEntropyLoss()
    optimizer = torch.optim.Adam(model.parameters(), lr=lr)
    model.to(device)
    for epoch in range(num_epochs):
        l_sum, n, start = 0.0, 0, time.time()
        data_iter = d2l.data_iter_consecutive(corpus_indices, batch_size, num_steps, device) # 相鄰採樣
        state = None
        for X, Y in data_iter:
            if state is not None:
                # 使用detach函數從計算圖分離隱藏狀態
                if isinstance (state, tuple): # LSTM, state:(h, c)  
                    state[0].detach_()
                    state[1].detach_()
                else: 
                    state.detach_()
            (output, state) = model(X, state) # output.shape: (num_steps * batch_size, vocab_size)
            y = torch.flatten(Y.T)
            l = loss(output, y.long())
            
            optimizer.zero_grad()
            l.backward()
            grad_clipping(model.parameters(), clipping_theta, device)
            optimizer.step()
            l_sum += l.item() * y.shape[0]
            n += y.shape[0]
        

        if (epoch + 1) % pred_period == 0:
            print('epoch %d, perplexity %f, time %.2f sec' % (
                epoch + 1, math.exp(l_sum / n), time.time() - start))
            for prefix in prefixes:
                print(' -', predict_rnn_pytorch(
                    prefix, pred_len, model, vocab_size, device, idx_to_char,
                    char_to_idx))

訓練模型.

num_epochs, batch_size, lr, clipping_theta = 250, 32, 1e-3, 1e-2
pred_period, pred_len, prefixes = 50, 50, ['分開', '不分開']
train_and_predict_rnn_pytorch(model, num_hiddens, vocab_size, device,
                            corpus_indices, idx_to_char, char_to_idx,
                            num_epochs, num_steps, lr, clipping_theta,
                            batch_size, pred_period, pred_len, prefixes)

epoch 50, perplexity 9.405654, time 0.52 sec
 - 分開始一起 三步四步望著天 看星星 一顆兩顆三顆四顆 連成線背著背默默許下心願  一枝楊柳 你的那我 在
 - 不分開 愛情你的手 一人的老斑鳩 腿短毛不多 快使用雙截棍 哼哼哈兮 快使用雙截棍 哼哼哈兮 快使用雙截棍
epoch 100, perplexity 1.255020, time 0.54 sec
 - 分開 我人了的屋我 一定令它心儀的母斑鳩 愛像一陣風 吹完美主  這樣 還人的太快就是學怕眼口讓我碰恨這
 - 不分開不想我多的腦袋有問題 隨便說說 其實我早已經猜透看透不想多說 只是我怕眼淚撐不住 不懂 你的黑色幽默
epoch 150, perplexity 1.064527, time 0.53 sec
 - 分開 我輕外的溪邊 默默在一心抽離 有話不知不覺 一場悲劇 我對不起 藤蔓植物的爬滿了伯爵的墳墓 古堡裏
 - 不分開不想不多的腦 有教堂有你笑 我有多煩惱  沒有你煩 有有樣 別怪走 快後悔沒說你 我不多難熬 我想就
epoch 200, perplexity 1.033074, time 0.53 sec
 - 分開 我輕外的溪邊 默默在一心向昏 的願  古無着我只能 一個黑遠 這想太久 這樣我 不要再是你打我媽媽
 - 不分開你只會我一起睡著 樣 娘子卻只想你和漢堡 我想要你的微笑每天都能看到  我知道這裏很美但家鄉的你更美
epoch 250, perplexity 1.047890, time 0.68 sec
 - 分開 我輕多的漫 卻已在你人演  想要再直你 我想要這樣牽着你的手不放開 愛可不可以簡簡單單沒有傷害 你
 - 不分開不想不多的假  已無能爲力再提起 決定中斷熟悉 然後在這裏 不限日期 然後將過去 慢慢溫習 讓我愛上