PyTorch softmax - torch.nn - torch.nn.functional

PyTorch softmax - torch.nn - torch.nn.functional

PyTorch documentation
https://pytorch.org/docs/stable/index.html

0. softmax - torch.nn - torch.nn.functional

• torch.nn.functional.softmax (Python function, in torch.nn.functional)
  定義的是一個函數。torch.nn.functional 中定義的是函數，由 def function() 定義，是一個固定的運算公式。

• torch.nn.Softmax (Python class, in torch.nn)
  定義的是一個類。torch.nn 中定義的是類，以 class xx 來定義的，可以提取變化的學習參數。

torch.nn.Softmax 是 Module 類，在實例化類後會初始化運算所需要的參數。這些參數會在 forward 和 backward 之後根據 loss 進行更新，通常存放在定義模型的 __init__() 中。在 Module 類裏的 __call__ 實現了 forward() 函數的調用。

train 和 test 階段運行方法一致時，儘量用 torch.nn，避免手動控制操作。

深度學習中權重需要不斷更新，需要採用類的方式，以確保能在參數發生變化時仍能使用之前定好的運算步驟。如果模型有可學習的參數，應該使用 torch.nn。但是簡單的計算不需要建類來做，所以使用 torch.nn.functional 定義函數即可。

torch.nn.name	torch.nn.functional.name
類	函數
結構中包含所需要初始化的參數。	在函數外定義並初始化相應參數，並作爲參數傳入。
在 __init__() 中實例化，並在 forward 中運算。	在 __init__() 中初始化相應參數，在 forward 中傳入參數。

1. softmax - torch.nn.functional.softmax (Python function, in torch.nn.functional)

https://pytorch.org/docs/stable/nn.functional.html

torch.nn.functional.softmax(input, dim=None, _stacklevel=3, dtype=None)

Applies a softmax function.
應用 softmax 函數。

Softmax is defined as:

$\text{Softmax}(x_{i}) = \frac{exp(x_i)}{\sum_j exp(x_j)}$

It is applied to all slices along dim, and will re-scale them so that the elements lie in the range [0, 1] and sum to 1.
它應用於 dim 上的所有切片, 並將對它們進行重新縮放, 使元素位於 [0, 1] 範圍內，和爲 1。

See Softmax for more details.
https://pytorch.org/docs/stable/nn.html#torch.nn.Softmax

1.1 Parameters

input (Tensor) – input

dim (int) – A dimension along which softmax will be computed.
dtype (torch.dtype, optional) – the desired data type of returned tensor. If specified, the input tensor is casted to dtype before the operation is performed. This is useful for preventing data type overflows. Default: None.
返回 tenosr 的期望數據類型。如果指定了參數，輸入張量在執行操作之前被轉換爲 dtype。這對於防止數據類型溢出非常有用。默認值：None。

This function doesn’t work directly with NLLLoss, which expects the Log to be computed between the Softmax and itself. Use log_softmax instead (it’s faster and has better numerical properties).
此函數不能直接與 NLLLoss 一起使用，NLLLoss 希望在 Softmax 及其自身之間計算 Log。使用 log_softmax 來代替 (它更快，並且具有更好的數值屬性)。

2. softmax - torch.nn.Softmax (Python class, in torch.nn)

https://pytorch.org/docs/stable/nn.html

CLASS torch.nn.Softmax(dim=None)

Applies the Softmax function to an n-dimensional input Tensor rescaling them so that the elements of the n-dimensional output Tensor lie in the range [0, 1] and sum to 1.
將 Softmax 函數應用於 n 維輸入 Tensor，對其進行重新縮放，以使 n 維輸出 Tensor 的元素位於 [0, 1] 範圍內，且總和爲 1。

Softmax is defined as:

$\text{Softmax}(x_{i}) = \frac{\exp(x_i)}{\sum_j \exp(x_j)}$

2.1 Shape

Input: (*) where * means, any number of additional dimensions
Output: (*), same shape as the input

2.2 Returns

a Tensor of the same dimension and shape as the input with values in the range [0, 1].

2.3 Parameters

dim (int) – A dimension along which Softmax will be computed (so every slice along dim will sum to 1).
用來計算 Softmax 的尺寸 (因此，沿 dim 的每個切片的總和爲 1)。

This module doesn’t work directly with NLLLoss, which expects the Log to be computed between the Softmax and itself. Use LogSoftmax instead (it’s faster and has better numerical properties).

3. Examples

3.1 example

(pt-1.4_py-3.6) yongqiang@yongqiang:~$ python
Python 3.6.10 |Anaconda, Inc.| (default, May  8 2020, 02:54:21)
[GCC 7.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> import torch.nn
>>>
>>> obj = torch.nn.Softmax(dim=1)
>>> input = torch.randn(2, 3)
>>> input
tensor([[ 0.0414, -0.7946,  1.5696],
        [ 0.2665, -0.2709, -2.0720]])
>>>
>>> output = obj(input)
>>> output
tensor([[0.1655, 0.0717, 0.7628],
        [0.5950, 0.3476, 0.0574]])
>>>
>>> exit()
(pt-1.4_py-3.6) yongqiang@yongqiang:~$

3.2 example

執行算子後，在 dim=0 維上的分佈，output_0[0][0][0]+ output_0[1][0][0] = 1。

(pt-1.4_py-3.6) yongqiang@yongqiang:~$ python
Python 3.6.10 |Anaconda, Inc.| (default, May  8 2020, 02:54:21)
[GCC 7.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> import torch.nn.functional as F
>>>
>>> input = torch.randn(2, 3, 4)
>>> input
tensor([[[-0.3976,  0.4142, -0.5061, -0.4063],
         [-0.2401, -1.5699,  1.4867, -1.9940],
         [ 0.5525, -1.4140, -1.2408, -0.6638]],

        [[-1.7829,  0.1077,  0.8127, -2.8241],
         [ 1.4750,  0.5804,  1.1887, -0.6570],
         [ 0.2279,  0.9583, -1.9489, -0.5876]]])
>>>
>>> output_0 = F.softmax(input, dim=0)
>>> output_0
tensor([[[0.7999, 0.5760, 0.2110, 0.9182],
         [0.1525, 0.1043, 0.5740, 0.2080],
         [0.5805, 0.0853, 0.6700, 0.4810]],

        [[0.2001, 0.4240, 0.7890, 0.0818],
         [0.8475, 0.8957, 0.4260, 0.7920],
         [0.4195, 0.9147, 0.3300, 0.5190]]])
>>>
>>> output_1 = F.softmax(input, dim=1)
>>> output_1
tensor([[[0.2102, 0.7703, 0.1134, 0.5057],
         [0.2461, 0.1059, 0.8322, 0.1034],
         [0.5437, 0.1238, 0.0544, 0.3909]],

        [[0.0290, 0.2022, 0.3969, 0.0524],
         [0.7543, 0.3244, 0.5780, 0.4574],
         [0.2167, 0.4734, 0.0251, 0.4903]]])
>>>
>>> output_2 = F.softmax(input, dim=2)
>>> output_2
tensor([[[0.1945, 0.4381, 0.1745, 0.1928],
         [0.1416, 0.0375, 0.7964, 0.0245],
         [0.6239, 0.0873, 0.1038, 0.1849]],

        [[0.0468, 0.3098, 0.6269, 0.0165],
         [0.4389, 0.1794, 0.3296, 0.0521],
         [0.2753, 0.5716, 0.0312, 0.1218]]])
>>>
>>> exit()
(pt-1.4_py-3.6) yongqiang@yongqiang:~$

3.3 example

dim=0 是張量的 0 軸，dim=1 是張量的 1 軸。

(pt-1.4_py-3.6) yongqiang@yongqiang:~$ python
Python 3.6.10 |Anaconda, Inc.| (default, May  8 2020, 02:54:21)
[GCC 7.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> import torch.nn.functional as F
>>>
>>> input = torch.randn(2, 3)
>>> input
tensor([[ 0.6812,  0.7187, -2.9340],
        [-0.0287,  0.5622, -0.7260]])
>>>
>>> output_0 = F.softmax(input, dim=0) # 每一列執行 softmax 運算
>>> output_0
tensor([[0.6704, 0.5390, 0.0990],
        [0.3296, 0.4610, 0.9010]])
>>>
>>> output_1 = F.softmax(input, dim=1) # 每一行執行 softmax 運算
>>> output_1
tensor([[0.4842, 0.5027, 0.0130],
        [0.3027, 0.5466, 0.1507]])
>>>
>>> exit()
(pt-1.4_py-3.6) yongqiang@yongqiang:~$

3.4 example

(pt-1.4_py-3.6) yongqiang@yongqiang:~$ python
Python 3.6.10 |Anaconda, Inc.| (default, May  8 2020, 02:54:21)
[GCC 7.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> import torch.nn.functional as F
>>>
>>> input = torch.tensor([[1, 2], [3, 4]], dtype=torch.float)
>>> input
tensor([[1., 2.],
        [3., 4.]])
>>>
>>> output_0 = F.softmax(input, dim=0)
>>> output_0
tensor([[0.1192, 0.1192],
        [0.8808, 0.8808]])
>>>
>>> output_1 = F.softmax(input, dim=1)
>>> output_1
tensor([[0.2689, 0.7311],
        [0.2689, 0.7311]])
>>>
>>> input = torch.tensor([[1, 2], [3, 6]], dtype=torch.float)
>>> input
tensor([[1., 2.],
        [3., 6.]])
>>>
>>> output_0 = F.softmax(input, dim=0)
>>> output_0
tensor([[0.1192, 0.0180],
        [0.8808, 0.9820]])
>>>
>>> output_1 = F.softmax(input, dim=1)
>>> output_1
tensor([[0.2689, 0.7311],
        [0.0474, 0.9526]])
>>>
>>> exit()
(pt-1.4_py-3.6) yongqiang@yongqiang:~$