## 1️⃣[Depthwise Separable Convolutions for Neural Machine Translation]

g-Sub-separable是指将channel分为几个group，每个group进行常规的convolution操作；g-Super-separable，也即本文中提出的convolution，同样是将channel分为几个group，然后对每个group进行depthwise-separable的卷积。

## 2️⃣[Squeeze-and-Excitation Networks]

### 符号

$\mathbf{V}$ 是可学习的卷积核参数： $\mathbf{V}=\left[\mathbf{v}_{1}, \mathbf{v}_{2}, \ldots, \mathbf{v}_{C}\right]$

### Squeeze: Global Information Embedding

${\mathbf{W}_{1} \in \mathbb{R}^{\frac{C}{r} \times C}}$ ${\mathbf{W}_{2} \in \mathbb{R}^{C \times \frac{C}{r}}}$
$r$是reduction ratio。

To make use of the information aggregated in the squeeze operation, we follow it with a second operation which aims to fully capture channel-wise dependencies. To fulfill this objective, the function must meet two criteria: first, it must be ﬂexible (in particular, it must be capable of learning a nonlinear interaction between channels) and second, it must learn a non-mutually-exclusive relationship since we would like to ensure that multiple channels are allowed to be emphasised (rather than enforcing a one-hot activation). To meet these criteria, we opt to employ a simple gating mechanism with a sigmoid activation.

## 3️⃣[Non-local Neural Networks]

### Non-local Network

#### $f$的具体形式

①Gaussian
$f\left(\mathbf{x}_{i}, \mathbf{x}_{j}\right)=e^{\mathbf{x}_{i}^{T} \mathbf{x}_{j}}$

②Embedded Gaussian
$f\left(\mathbf{x}_{i}, \mathbf{x}_{j}\right)=e^{\theta\left(\mathbf{x}_{i}\right)^{T} \phi\left(\mathbf{x}_{j}\right)}$

③Dot product
$f\left(\mathbf{x}_{i}, \mathbf{x}_{j}\right)=\theta\left(\mathbf{x}_{i}\right)^{T} \phi\left(\mathbf{x}_{j}\right)$

④Concatenation
$f\left(\mathbf{x}_{i}, \mathbf{x}_{j}\right)=\operatorname{ReLU}\left(\mathbf{w}_{f}^{T}\left[\theta\left(\mathbf{x}_{i}\right), \phi\left(\mathbf{x}_{j}\right)\right]\right)$
$\mathcal{C}(\mathbf{x})=N$

$\mathbf{z}_{i}=W_{z} \mathbf{y}_{i}+\mathbf{x}_{i}$
$y$则是non-local network的输出。

### Non-local block的策略/tricks

①设置$W_g$,$W_θ$,$W_ϕ$的channel的数目为x的channel数目的一半，这样就形成了一个bottleneck，能够减少一半的计算量。Wz再重新放大到x的channel数目，保证输入输出维度一致。

②在$\frac{1}{\mathcal{C}(\hat{\mathbf{x}})} \sum_{\forall j} f\left(\mathbf{x}_{i}, \hat{\mathbf{x}}_{j}\right) g\left(\hat{\mathbf{x}}_{j}\right)$使用下采样，如max-pooling，减少计算量。

## 4️⃣[Bilinear CNN Models for Fine-grained Visual Recognition]

Motivation(?不确定是不是这样)：对于细粒度物体的分类，先对局部定位，再提取特征。两个特征提取器一个是提取location，另一个提取特征。

outer product captures pairwise correlations between the feature channels

here are two main pathways, or “streams”. The ventral stream (or, “what pathway”) is involved with object identiﬁcation and recognition. The dorsal stream (or, “where pathway”) is involved with processing the object’s spatial location relative to the viewer.

$\text { bilinear }\left(l, \mathcal{I}, f_{A}, f_{B}\right)=f_{A}(l, \mathcal{I})^{T} f_{B}(l, \mathcal{I})$

pooling有好几种，可以直接加起来，或者使用max-pooling。这里使用直接加起来的方式，可以理解为，这些特征是无序(orderless)的叠加。

$\begin{array}{l}{\mathbf{y} \leftarrow \operatorname{sign}(\mathbf{x}) \sqrt{|\mathbf{x}|}} \\ {\mathbf{z} \leftarrow \mathbf{y} /|\mathbf{y}|_{2}}\end{array}$

①But do the networks specialize into roles of localization (“where”) and appearance modeling (“what”) when initialized asymmetrically and ﬁne-tuned?

Both these networks tend to activate strongly on highly speciﬁc semantic parts

②bilinear的好处还可以扩展成trilinear，添加更多的信息。