每周碎片知识17

1️⃣[Pytorch]

在有RNN的代码中,如果出现

Cuda Error : RuntimeError: CUDNN_STATUS_EXECUTION_FAILED

那么可能的出错原因是没有将init state放入cuda中。

Reference: https://discuss.pytorch.org/t/cuda-error-runtimeerror-cudnn-status-execution-failed/17625


2️⃣[Pytorch]

clone() → Tensor
Returns a copy of the self tensor. The copy has the same size and data type as self.
Unlike copy_(), this function is recorded in the computation graph. Gradients propagating to the cloned tensor will propagate to the original tensor.

如果需要另一个相同的tensor做其他计算,则使用clone()而不是copy_()

1
2
3
forward_vec=sent_vec
# backward_vec=sent_vec wrong
backward_vec=sent_vec.clone()

当然也不能直接赋值,因为赋的只是指针,改变backward_vec也会改变原来的值。


3️⃣[Python]

Python中==is的区别:
is表示是否是同一个object;而==表示是否是同一个值。

1
2
3
4
5
6
str='GRU'
str == 'GRU' # True
str is 'GRU' # True
str=str.upper()
str == 'GRU' # False
str is 'GRU' # True

4️⃣[RNN]

在RNN的初始化中,使用正交初始化会比其他方法好一些(待对比实验测验)。
Reference: https://smerity.com/articles/2016/orthogonal_init.html


5️⃣[Pytorch]

在提供预训练embedding作为初始化时,正确做法:

1
2
3
4
if pretrained_matrix is not None:
pretrained_matrix=torch.from_numpy(pretrained_matrix).type(torch.FloatTensor)
self.embedding.weight= nn.Parameter(pretrained_matrix,
requires_grad=True)

必须要有.type(torch.FloatTensor),否则会出错:CuDNN error: CUDNN_STATUS_EXECUTION_FAILED


6️⃣[Pytorch]

Pytorch中,将初始hidden state作为可学习参数实践:
https://discuss.pytorch.org/t/solved-train-initial-hidden-state-of-rnns/2589/9
https://discuss.pytorch.org/t/learn-initial-hidden-state-h0-for-rnn/10013/7