Actually, the small memory we asked you for is one on top of the three you can trade! That one Augustine would not remember himself until he and Togusa manage to swap back, and vice versa; each character would just wake up with the new memory, as well as the other three you had.

I didn't quite explain it well enough in the initial comment, I think, my apologies!