I understand your hesitation to wear a shirt that has 卍 on it. It's really too bad; it really is a beautiful symbol. It's a shame it's been corrupted by the Nazis. I'm Native American (Navajo), and it's actually a symbol that we used to use a lot in our art!
Anyway, if you have other Japanese test with it, I don't think it would really be a problem, but I don't live in Germany, so I'm not familiar with just how serious it could be taken there.
As for 万字, don't worry about any warping of the meaning. The first symbol is pronounced "man" and it usually means "10,000." The second symbol is pronounced "ji" and, alone, can mean "word." So 万字 is literally just a phoenetic way to write "manji." It doesn't really have any meaning on its own.