Pytorch——Broadcasting and Element-wise Operations with PyTorch

Element-wise tensor operations for deep learning

An element-wise operation is an operation between two tensors that operates on corresponding elements within the respective tensors.元素操作是兩個張量之間的操作，它對各自張量內的對應元素進行操作。
An element-wise operation operates on corresponding elements between tensors
Two elements are said to be corresponding if the two elements occupy the same position within the tensor. The position is determined by the indexes used to locate each element.兩個元素如果在相同的位置上，則說這兩個元素對應。

> t1 = torch.tensor([
    [1,2],
    [3,4]
], dtype=torch.float32)

> t2 = torch.tensor([
    [9,8],
    [7,6]
], dtype=torch.float32)

# Example of the first axis
> print(t1[0])
tensor([1., 2.])


# Example of the second axis
> print(t1[0][0])
tensor(1.)

> t1[0][0]
tensor(1.)


> t2[0][0]
tensor(9.)

This allows us to see that the corresponding element for the 1 in t1 is the 9 in t2.
The correspondence is defined by the indexes.

Addition is an element-wise operation

> t1 + t2
tensor([[10., 10.],
        [10., 10.]])

Arithmetic operations are element-wise operations

> print(t1 + 2)
tensor([[3., 4.],
        [5., 6.]])

> print(t1 - 2)
tensor([[-1.,  0.],
        [ 1.,  2.]])

> print(t1 * 2)
tensor([[2., 4.],
        [6., 8.]])

> print(t1 / 2)
tensor([[0.5000, 1.0000],
        [1.5000, 2.0000]])

> print(t1.add(2))
tensor([[3., 4.],
        [5., 6.]])

> print(t1.sub(2))
tensor([[-1.,  0.],
        [ 1.,  2.]])

> print(t1.mul(2))
tensor([[2., 4.],
        [6., 8.]])

> print(t1.div(2))
tensor([[0.5000, 1.0000],
        [1.5000, 2.0000]])

Broadcasting tensors廣播

Broadcasting describes how tensors with different shapes are treated during element-wise operations.
**Broadcasting is the concept whose implementation allows us to add scalars to higher dimensional tensors.**廣播是一個概念，它的實現允許我們向高維張量添加標量。
We can see what the broadcasted scalar value looks like using the broadcast_to() Numpy function:

> np.broadcast_to(2, t1.shape)
array([[2, 2],
        [2, 2]])

This

> t1 + 2
tensor([[3., 4.],
        [5., 6.]])

is really this:

> t1 + torch.tensor(
    np.broadcast_to(2, t1.shape)
    ,dtype=torch.float32
)
tensor([[3., 4.],
        [5., 6.]])

Another example is here：

t1 = torch.tensor([
    [1,1],
    [1,1]
], dtype=torch.float32)

t2 = torch.tensor([2,4], dtype=torch.float32)

> t1.shape
torch.Size([2, 2])

> t2.shape
torch.Size([2])

The lower rank tensor t2 will be transformed via broadcasting to match the shape of the higher rank tensor t1, and the element-wise operation will be performed as usual.

> np.broadcast_to(t2.numpy(), t1.shape)
array([[2., 4.],
        [2., 4.]], dtype=float32)

> t1 + t2
tensor([[3., 5.],
        [3., 5.]])

Comparison Operations are Element-wise

For a given comparison operation between two tensors, a new tensor of the same shape is returned with each element containing either a torch.bool value of True or False.

> torch.tensor([1, 2, 3]) < torch.tensor([3, 1, 2])
tensor([True, False, False])

Element-wise Comparison Operation Examples

> t = torch.tensor([
    [0,5,0],
    [6,0,7],
    [0,8,0]
], dtype=torch.float32)

> t.eq(0)  # 是否等於零
tensor([[True, False, True],
        [False, True, False],
        [True, False, True]])


> t.ge(0)  # 大於等於某個值 great/equal
tensor([[True, True, True],
        [True, True, True],
        [True, True, True]])


> t.gt(0)  # 大於某個值，greater than
tensor([[False, True, False],
        [True, False, True],
        [False, True, False]])


> t.lt(0)  # 小於某個值，less than
tensor([[False, False, False],
        [False, False, False],
        [False, False, False]])

> t.le(7)  # 小於等於某個值， less/equal
tensor([[True, True, True],
        [True, True, True],
        [True, False, True]])

Thinking about these operations from a broadcasting perspective, we can see that the last one, t.le(7), is really this:

> t <= torch.tensor(
    np.broadcast_to(7, t.shape)
    ,dtype=torch.float32
)

tensor([[True, True, True],
        [True, True, True],
        [True, False, True]])

# And equivalently this
> t <= torch.tensor([
    [7,7,7],
    [7,7,7],
    [7,7,7]
], dtype=torch.float32)

tensor([[True, True, True],
        [True, True, True],
        [True, False, True]])

Element-wise Operations using Functions

> t.abs() 
tensor([[0., 5., 0.],
        [6., 0., 7.],
        [0., 8., 0.]])


> t.sqrt()
tensor([[0.0000, 2.2361, 0.0000],
        [2.4495, 0.0000, 2.6458],
        [0.0000, 2.8284, 0.0000]])

> t.neg()
tensor([[-0., -5., -0.],
        [-6., -0., -7.],
        [-0., -8., -0.]])

> t.neg().abs()
tensor([[0., 5., 0.],
        [6., 0., 7.],
        [0., 8., 0.]])

Element-wise / Component-wise / Point-wise are meaning the same thing.