1. DL Shape Error

* 因很多DL都是对矩阵进行乘法和运算，而矩阵可对组合(combine)的形状shape和
大小size有严格的规则strict rule，故DL常见错误之一是形状不匹配shape mismatch；

tc = torch.tensor([[1,2],[3,4],[5,6]],dtype=torch.float32)
td = torch.tensor([[7,10],[8,11],[9,12]],dtype=torch.float32)
torch.matmul(tc,td)
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
Cell In[143], line 3
      1 tc = torch.tensor([[1,2],[3,4],[5,6]],dtype=torch.float32)
      2 td = torch.tensor([[7,10],[8,11],[9,12]],dtype=torch.float32)
----> 3 torch.matmul(tc,td)

RuntimeError: mat1 and mat2 shapes cannot be multiplied (3x2 and 3x2)

  • 可通过使tc和td的内部维度匹配(inner dimension match)
    使他们之间的矩阵乘法运算matrix multiplication；

  • 其中一种方法：使用转置(切换给定张量的维度)，
    transpose：switch given tensor dimension；

  • 两种方式转换：torch.transpose(input,dim0,dim1)和tensor.T；

1.1. Tensor.T

print(tc)
print(td)
print(tc)
print(td.T);
tensor([[1., 2.],
        [3., 4.],
        [5., 6.]])
tensor([[ 7., 10.],
        [ 8., 11.],
        [ 9., 12.]])
tensor([[1., 2.],
        [3., 4.],
        [5., 6.]])
tensor([[ 7.,  8.,  9.],
        [10., 11., 12.]])
print(f"Original Shape：tc = {tc.shape}，td = {td.shape}\n")

print(f"New Shape: tc = {tc.shape} (same as above)，td.T = {td.T.shape}\n")
print(f"Multiplying: {tc.shape} * {td.T.shape} <- inner dimension match\n")

print("Output:\n")
output = torch.matmul(tc,td.T)
print(output)
print(f"\nOutput Shape: {output.shape}")
Original Shape：tc = torch.Size([3, 2])，td = torch.Size([3, 2])

New Shape: tc = torch.Size([3, 2]) (same as above)，td.T = torch.Size([2, 3])

Multiplying: torch.Size([3, 2]) * torch.Size([2, 3]) <- inner dimension match

Output:

tensor([[ 27.,  30.,  33.],
        [ 61.,  68.,  75.],
        [ 95., 106., 117.]])

Output Shape: torch.Size([3, 3])

1.2. Transpose

tt = torch.randn(2,4)
tt,torch.transpose(tt,0,1)
(tensor([[-1.2739, -0.5950,  1.1307,  0.5441],
         [-0.4785, -0.3763,  0.2029, -0.1439]]),
 tensor([[-1.2739, -0.4785],
         [-0.5950, -0.3763],
         [ 1.1307,  0.2029],
         [ 0.5441, -0.1439]]))

2. Aggregation

  • Aggregation：min，max，mean，sum等；某些方法，如torch.mean()，
    要求torch.float32(最常见)或其它特定的数据类型，否则将操作失败；

ta = torch.arange(0, 100, 10)

print(f"Tensor: {ta}")
print(f"Minimum: {ta.min()}")
print(f"Maximum: {ta.max()}")

# this will throw exception
# print(f"Mean: {ta.mean()}")

# won't work without float datatype
print(f"Mean: {ta.type(torch.float32).mean()}")
print(f"Sum: {ta.sum()}")

print("Tensor:{}\nMinimum:{},Maximum:{},Mean:{},Sum:{}"
    .format(ta,ta.min(),ta.max(),ta.type(torch.float32).mean(),ta.sum()))

ta,torch.min(ta),torch.max(ta),torch.mean(ta.type(torch.float32)),torch.sum(ta)
Tensor: tensor([ 0, 10, 20, 30, 40, 50, 60, 70, 80, 90])
Minimum: 0
Maximum: 90
Mean: 45.0
Sum: 450
Tensor:tensor([ 0, 10, 20, 30, 40, 50, 60, 70, 80, 90])
Minimum:0,Maximum:90,Mean:45.0,Sum:450

(tensor([ 0, 10, 20, 30, 40, 50, 60, 70, 80, 90]),
 tensor(0),
 tensor(90),
 tensor(45.),
 tensor(450))

3. Positional

  • 张量的索引：只想要最高或最低只所在的位置/索引，而非实际值本身的情况下很有用；

  • 最大值：torch.argmax()，最小值：torch.argmin()；

tensor = torch.arange(10,100,10)
print("Tensor:{}\nMinimum Value Index:{},Maximum Value Index:{}"
    .format(tensor,tensor.argmin(),tensor.argmax()))
Tensor:tensor([10, 20, 30, 40, 50, 60, 70, 80, 90])
Minimum Value Index:0,Maximum Value Index:8

4. Change Datatype

  • DL常见问题之一：在不同数据类型中使用张量，如A张量在torch.float64，
    B张量在torch.float32，则可能会遇到错误；

  • 解决方案：更改张量的数据类型：torch.Tensor.type(dtype=None)，
    默认的数据类型：torch.float32；

  • 数字越低(如32，16，8)，存储的值就越不精确precise；

  • 较低的存储量，通常导致更快的计算和更小的整体模型，mobile-based神经网络
    通常用8位整数操作，与float32对应的网络相比，整数更小，运行更快，但精度更低；

  • https://pytorch.org/docs/stable/tensors.html

 
tensor_float32 = torch.arange(10., 100., 10.)
tensor_float16 = tensor_float32.type(torch.float16)
tensor_int8 = tensor_float32.type(torch.int8)

tensor_float32.dtype,tensor_float16,tensor_int8
(torch.float32,
 tensor([10., 20., 30., 40., 50., 60., 70., 80., 90.], dtype=torch.float16),
 tensor([10, 20, 30, 40, 50, 60, 70, 80, 90], dtype=torch.int8))

5. Tensor Action

  • DL模型(神经网络)是关于以某种方式操作manipulating张量的，
    因matrix multiplication规则，若有形状不匹配，将抛出错误，
    方法帮助确保张量中的right element与其他张量中的right element混合mixing；

6. Reshaping

  • torch.reshape(input,shape)或torch.Tensor.reshape()；

  • 重塑输入的形状(若兼容)，reshape input to shape (if compatible)；

tensor = torch.arange(1., 8.)

# add extra dimension with torch.reshape()
tensor_reshape = tensor.reshape(1, 7)

tensor,tensor.shape,tensor_reshape,tensor_reshape.shape
(tensor([1., 2., 3., 4., 5., 6., 7.]),
 torch.Size([7]),
 tensor([[1., 2., 3., 4., 5., 6., 7.]]),
 torch.Size([1, 7]))

7. Viewing

  • Tensor.view(shape)：以不同的形状返回原始张量的视图，但与原始张量共享相同的数据；

  • return original tensor view in different shape
    but share the same data as original tensor；

  • 用torch.view()更改张量的视图实际上只会创建相同张量的新视图，故改变视图也会改变原始张量；

tensor = torch.arange(1., 8.)

# change view with torch.view()
tensor_view = tensor.view(1, 7)

tensor,tensor.shape,tensor_view,tensor_view.shape
(tensor([1., 2., 3., 4., 5., 6., 7.]),
 torch.Size([7]),
 tensor([[1., 2., 3., 4., 5., 6., 7.]]),
 torch.Size([1, 7]))
# changing view tensor_view will change original tensor
tensor_view[:,0] = 5

tensor_view,tensor
(tensor([[5., 2., 3., 4., 5., 6., 7.]]),
tensor([5., 2., 3., 4., 5., 6., 7.]))

8. Stacking

  • torch.stack(tensors,dim=0)：
    沿新维度(dim)连接张量序列，所有张量的大小必须相同；

  • concatenate tensor sequence along new dimension(dim)，
    all tensor must be same size；

  • 若想把新的张量叠加在上面五次，可使用torch.stack()

tensor = torch.arange(1., 8.)

# stack tensor on top of each other
# try changing dim to dim=1 and see what happen
tensor_stack = torch.stack([tensor,tensor,tensor,tensor], dim=0)
tensor_stack
tensor([[1., 2., 3., 4., 5., 6., 7.],
        [1., 2., 3., 4., 5., 6., 7.],
        [1., 2., 3., 4., 5., 6., 7.],
        [1., 2., 3., 4., 5., 6., 7.]])

9. Squeezing

  • torch.squeeze(input)：压缩输入以移除所有值为1的维度；

  • squeeze input to remove all dimenions with value 1；

  • 用torch.squeeze()从张量中移除所有单个维度，这会将张量压缩到只有大于1的维度；

tensor = torch.arange(1., 8.)

# add extra dimension with torch.reshape()
tensor_reshape = tensor.reshape(1, 7)

print(f"Previous Tensor: {tensor_reshape}")
print(f"Previous Shape: {tensor_reshape.shape}")

# remove extra dimension from tensor_reshape
tensor_squeeze = tensor_reshape.squeeze()
print(f"\nNew Tensor: {tensor_squeeze}")
print(f"New Shape: {tensor_squeeze.shape}")
tensor = torch.arange(1., 8.)

# add extra dimension with torch.reshape()
tensor_reshape = tensor.reshape(1, 7)

print(f"Previous Tensor: {tensor_reshape}")
print(f"Previous Shape: {tensor_reshape.shape}")

# remove extra dimension from tensor_reshape
tensor_squeeze = tensor_reshape.squeeze()
print(f"\nNew Tensor: {tensor_squeeze}")
print(f"New Shape: {tensor_squeeze.shape}")

10. Unsqueezing

  • torch.unsqueeze(input,dim)：返回在dim处添加维度值为1的输入；

  • returns input with dimension value of 1 added at dim；

  • torch.unsqueeze()与torch.squeeze()相反，会在特定索引处添加维度值1；

tensor = torch.arange(1., 8.)

# add extra dimension with torch.reshape()
tensor_reshape = tensor.reshape(1, 7)

print(f"Reshape Tensor: {tensor_reshape}")
print(f"Reshape Shape: {tensor_reshape.shape}")

# remove extra dimension from tensor_reshape
tensor_squeeze = tensor_reshape.squeeze()
print(f"\nSqueeze Tensor: {tensor_squeeze}")
print(f"Squeeze Shape: {tensor_squeeze.shape}")

# add extra dimension with unsqueeze
tensor_unsqueeze = tensor_squeeze.unsqueeze(dim=0)
print(f"\nUnsqueeze Tensor: {tensor_unsqueeze}")
print(f"Unsqueeze Shape: {tensor_unsqueeze.shape}")
Reshape Tensor: tensor([[1., 2., 3., 4., 5., 6., 7.]])
Reshape Shape: torch.Size([1, 7])

Squeeze Tensor: tensor([1., 2., 3., 4., 5., 6., 7.])
Squeeze Shape: torch.Size([7])

Unsqueeze Tensor: tensor([[1., 2., 3., 4., 5., 6., 7.]])
Unsqueeze Shape: torch.Size([1, 7])

11. Permuting

  • torch.permute(input,dims)：返回原始输入的视图，其维度已排列(重新排列)为维度；

  • return original input view with its dimension permute(rearrange) to dim；

  • 可使用torch.permute(input,dims)重新排列轴值的顺序，而输入将变成具有新维度的视图；

  • rearrange axes value order with torch.permute(input,dims)，
    where input get turn into view with new dims；

  • 注：因permuting返回视图(与原视图共享相同数据)，故permuting张量中的值将
    与原张量相同，若更改视图中的值，它将更改原视图中的值，Permuting类似于Viewing；

# create specific shape tensor
tensor_original = torch.rand(size=(224, 224, 3))

# permute original tensor to rearrange axis order
# shift axis 0 -> 1,1 -> 2,2 -> 0
tensor_permute = tensor_original.permute(2,0,1)

print(f"Tensor Original Shape: {tensor_original.shape}")
print(f"Tensor Permute Shape: {tensor_permute.shape}")
# create specific shape tensor
tensor_original = torch.rand(size=(224, 224, 3))

# permute original tensor to rearrange axis order
# shift axis 0 -> 1,1 -> 2,2 -> 0
tensor_permute = tensor_original.permute(2,0,1)

print(f"Tensor Original Shape: {tensor_original.shape}")
print(f"Tensor Permute Shape: {tensor_permute.shape}")