I think that idea of eventually having all the memories after death or every reincarnation is plausible, I've considered something like that before and think it could be true. I think Aidan108's thoughts are quite close to yours on that point.

The only issue I would have is that it implies there can be two consciousness experiencing at once (correct me if I'm wrong). Imagine from your perspective, why is your consciousness more real than mine? Even if you trust that I am real, you can feel that you're real whereas I'm just saying it (albeit truthfully)