Beware of Mixed use of Property and __getattr__

One of the Python Debugging Pitfalls

Background

Recently I’ve been trying to apply Weight Dropped LSTM/RNN to a time series prediction project. The author of the paper released the source code, but it’s written for PyTorch 0.1.12_2. If you try to run it with PyTorch 0.2.0, you’d get the following error:

AttributeError: 'LSTM' object has no attribute 'all_weights'

Because the time series project has used some features in PyTorch 0.2.0, and I’d really rather not spend time making it backward-compatible, some investigation has been conducted to find a way to make the code work with 0.2.0.

The exception is raised from the class WeightDrop when tying to load the model parameters into CUDA. After some probing, I found the problem originates from this part of the code:

def _setup(self):                               
for name_w in self.weights:
print(
'Applying weight drop of {} to'
'{}'.format(self.dropout, name_w)
)
w = getattr(self.module, name_w)
del self.module._parameters[name_w]
self.module.register_parameter(
name_w + '_raw', Parameter(w.data))

all_weights will be missing from the object right after the del operation. I am puzzled for a while why deleting an element from an OrderedDict would mask a class property. It just doesn’t make sense.

Conflict between @property and __getattr__

Fortunately I was able to find the answer on Google. In this Github issue (Usage of both @property and __getattr__ can lead to obscure and hard to debug errors) , skrivanos wrote:

if a property getter raises an AttributeError, python (__getattribute__) falls back to using __getattr__. The issue is that the traceback is lost, which makes it extremely hard to debug, since the actual exception raised isn't shown anywhere.

If we overwrite the LSTM class to print the exception raised:

class LSTM(nn.LSTM):    
@property
def all_weights(self):
try:
print([[getattr(self, weight) for weight in weights] for weights in self._all_weights])
except Exception as e:
print(e)
return [[getattr(self, weight) for weight in weights] for weights in self._all_weights]

We’ll find the actual exception was:

'LSTM' object has no attribute 'weight_hh_l0'

Proposed Solution

Now the solution is clear. We just need to update self._all_weights in _setup. Namely, add this line inside the loop (sorry for the weird formatting):

self.module._all_weights[0][ \   
self.module._all_weights[0].index(name_w)] = \
(name_w + '_raw')

This effectively renames the weight matrix that has been applied dropout.

Please be advised this might not be the best way to do Pytorch 0.1.12 to 0.2.0 migration, but only a hacky way to make it work. It also serves as an example to demonstrate how mixing @property and __getattr__ can lead to confusing error messages that inexperience developers like me can get derailed and waste lots of time. I guess the lesson is to be doubtful of the error messages, and continue to explore other possibilities when the literal interpretation of the messages is leading you nowhere.

2017/09/22 Update

As this issue points out, the previously proposed solution is broken on GPU. It is due to the way PyTorch 0.2.0 flatten RNN weights. The proper solution is to stop PyTorch from flattening by killing the flatten_parameters function in _setup:

if issubclass(type(self.module), torch.nn.RNNBase):                                           
self.module.flatten_parameters = \
self.widget_demagnetizer_y2k_edition

where self.widget_demagnetizer_y2k_edition is a dummy method that returns nothing. (Note you no longer need to self.module._all_weights[0] as proposed previously.