Week 10: Some interesting results in our approach

Previously on my GSoC story:

I got the results for each task trained separately and build some scripts to pre-process and post-process the outputs.

What did I do this week?

In my last weekly meeting, We talked about implementing Beam search to reproduce the results of this paper. We could not talk a lot about the results of transfer learning and multitask learning because the results were not ready on time. However, I talked to them about and multi-input approach. This takes the idea of multilingual neural machine translation, in which we only have one encoder and one decoder and we train all tasks (as language pairs in multilingual neural machine translation) simultaneously leveraging some similarities in the tasks. To do this, I add a task token for each source. For example, if we want to generate the discourse ordering for the set of triples “<TRIPLE> Aarhus_Airport cityServed Aarhus </TRIPLE> <TRIPLE> Aarhus leaderName Jacob_Bundsgaard </TRIPLE>” we add a token called <ordering> at the beginning of the linearized form. Besides, for these experiments, we apply byte-pair encoding for the target for the lexicalization and the end2end tasks and tied embeddings.

The results of the training separately, with the multi-input, transfer learning, and transfer learning of the encoder are shown in Table 1.

In general, training each task separately seems to be the better alternative. However, the multi-input approach shows improvements in the lexicalization (a high improvement) and the end2end (this latter could be not significant)tasks. In particular, the lexicalization task seems to be the more benefited as the multi-input and the transfer learning approaches improve its performance.

It is worth noting that the Text structuring’s performance is damaged in all approaches. A possible explanation to this result is that could be an interference between the Discourse ordering and the Text structuring task as the input for the first one is an unordered version and the latter one an ordered version. It could be verify in the transfer learning approaches as the only pre-trained model used to train the text structuring is the discourse ordering task.

Table 1. Results for all tasks according to each approach.

Finally, the results of applying a beam search decoding with a beam size of 5 are presented in Table 2. As we may see, the result is improved from 29.03 (Table 1) to 30.65.

Table 2. Results for the End2End task with a beam size of 5.

What’s next?

I am waiting for the results of the Multitask learning and completing some documentation in my GitHub repo.