Patrick points out an interesting snippet on Paul Graham's site:

I found, when I was studying mathematics, that 2 things were true: (1) the teacher was not too good and (2) the book was not too good. So I would always buy a half-dozen books on the topic and try to get the full picture by reading the same sections in each book. The combination helped me understand much more than the sum of the content. Also, I was never opposed to reading something as much as 10 times until I squeezed everything out of it.

I've also had this problem. I've also done a lot of math reading on my own time, and the biggest single thing I can recommend for reading math books is this: Force yourself to read every single line carefully. Some more than once. I always want to read math like I read other books, that is, very quickly. Math is more dense. Don't worry, you're not wasting time, you're not worse at math than you are at other things. It's just more information in less space. Take your time.