I think you answered it. Everything is seen, experienced, and interpreted in the mind. In ones mind is the only place one will ever be, ever experience. If our minds match...reasonably well, we assume there must be something out there that we are both "seeing". But we can never know this for sure.

Both Neo and Morpheus experience Agent Smith as a real physical presence, but only because their minds are downloading similar information from their Matrix plugs. Of course that is just the physical representation in the mind. Then interpretation begins. Morpheus sees Agent Smith as a mortal danger, and so to him he was, to Neo...not so much.

One of the things that doesn't ever seem to get enough attention...imo...or, when it does it gets misinterpreted...imo..., is when in the Bible when Jesus healed someone they would give him credit, yet he would often respond it was their faith that had healed them. Jesus was not the only one to walk on water in the Bible, Peter did as well, until he started to doubt that he could, then be began to sink.