07-01 Live Session

GAP (Global Average Pooling)

import torch.nn.functional as F
x = F.adaptive_avg_pool2d(x, (1, 1))

x = torch.randn(16, 14, 14)
out = F.adaptive_max_pool2d(x.unsqueeze(0), output_size=1)

# Calculate result manually to compare results
out_manual = torch.stack([out[:, i:i+4].mean() for i in range(0, 16, 4)])

out = out.view(out.size(0), out.size(1)//4, -1)
out = out.mean(2)

print(torch.allclose(out_manual, out))

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv = nn.Sequential(
            #3 224 128
            nn.Conv2d(3, 64, 3, padding=1),nn.LeakyReLU(0.2),
            nn.Conv2d(64, 64, 3, padding=1),nn.LeakyReLU(0.2),
            nn.MaxPool2d(2, 2),
            #64 112 64
            nn.Conv2d(64, 128, 3, padding=1),nn.LeakyReLU(0.2),
            nn.Conv2d(128, 128, 3, padding=1),nn.LeakyReLU(0.2),
            nn.MaxPool2d(2, 2),
            #128 56 32
            nn.Conv2d(128, 256, 3, padding=1),nn.LeakyReLU(0.2),
            nn.Conv2d(256, 256, 3, padding=1),nn.LeakyReLU(0.2),
            nn.Conv2d(256, 256, 3, padding=1),nn.LeakyReLU(0.2),
            nn.MaxPool2d(2, 2),
            #256 28 16
            nn.Conv2d(256, 512, 3, padding=1),nn.LeakyReLU(0.2),
            nn.Conv2d(512, 512, 3, padding=1),nn.LeakyReLU(0.2),
            nn.Conv2d(512, 512, 3, padding=1),nn.LeakyReLU(0.2),
            nn.MaxPool2d(2, 2),
            #512 14 8
            nn.Conv2d(512, 512, 3, padding=1),nn.LeakyReLU(0.2),
            nn.Conv2d(512, 512, 3, padding=1),nn.LeakyReLU(0.2),
            nn.Conv2d(512, 512, 3, padding=1),nn.LeakyReLU(0.2),
            nn.MaxPool2d(2, 2)
        )
        #512 7 4

        self.avg_pool = nn.AvgPool2d(7)
        #512 1 1
        self.classifier = nn.Linear(512, 10)
        """
        self.fc1 = nn.Linear(512*2*2,4096)
        self.fc2 = nn.Linear(4096,4096)
        self.fc3 = nn.Linear(4096,10)
        """

    def forward(self, x):

        #print(x.size())
        features = self.conv(x)
        #print(features.size())
        x = self.avg_pool(features)
        #print(avg_pool.size())
        x = x.view(features.size(0), -1)
        #print(flatten.size())
        x = self.classifier(x)
        #x = self.softmax(x)
        return x, features

CNN History

Alexnet

VGG

Inception

Resnet

resnet이후에는 깊이에 대한 경쟁은 줄어듬

Efficientnet

https://hoya012.github.io/blog/EfficientNet-review/

GAN

VAE

RNN

LSTM

RNN의 한계를 극복하고자 해서 나온 것.

Transfer Learning

Domain Adaptation

Generalization

mlp mixer

Q & A

1. 데이터를 보고 전이학습을 쓰면 좋겠다 / 쓰면 안되겠다 하는 판단의 기준이 있을까요?
   • inductive: 도메인이 똑같고 태스크만 달라질때, 쉽게 해결 가능
   • 도메인이 달라지는 경우 어렵다.
2. fc층에서 avg pooling을 써서 1대 1 대칭이 이루어진다고 하셧는데, fc층에는 linear층을 쓰지 않는다는 것일까요
   • 마지막 레이어층만 fc쓰고 그전에 gap로 추출한 것을 1:1로 대응.
3. 예전에는 연산량을 줄이려고 애썼다면 요즘은 하드웨어의 능력치를 믿고 연산량을 줄이기보다는 성능위주로 가고 있다고 생각하면 되나요? 아니면 지금도 연산량은 최대한 줄이려는 노력을 하나요?
   • deeper, lighter

http://cs231n.stanford.edu/slides/2021/lecture_4.pdf
http://cs231n.stanford.edu/slides/2021/lecture_5.pdf
http://cs231n.stanford.edu/slides/2021/lecture_6.pdf

Reference

gap: https://discuss.pytorch.org/t/tensor-global-max-pooling-and-average/38988
cam with gap: https://ctkim.tistory.com/117
CNN Development History: https://hoya012.github.io/blog/deeplearning-classification-guidebook-1/, https://hoya012.github.io/blog/deeplearning-classification-guidebook-2/, https://hoya012.github.io/blog/deeplearning-classification-guidebook-3/, https://hoya012.github.io/blog/deeplearning-classification-guidebook-4/ :http://cs231n.stanford.edu/slides/2021/lecture_5.pdf