Boundary effects and hadron masses in lattice QCD