The overhead of enabling the extra 128Mb used to be great in 2.4, especially before the later additions to the vm by -aa (around 2.4.23), so it was very true that performance would suffer. It had nothing to do with what the devices could access; it was the extra structures put in place to use high mem. 2.6, however, has taken away a lot of that overhead so it isn't a big performance hit.