Is it weirder remembering something that didn't happen?
Or not-quite-remembering something that did happen, and mistakenly concluding that it didn't?

Admittedly, I probably do both several times a day, and there's no good reason for me to believe I do the latter more often. But my self-image as being in some non-accidental way engaged with and primarily constrained by an actual world that is singularly isomorphic to my perceptions tends to push me to that belief anyway.