This is the book that put Bruce Schneier on the map. It is a great reference on the algorithms of cryptography and how to implement them. If you want to know how they work (i.e. Computer Science people) then this is a must. This is Bruce's follow on book, in which he tackles all the other issues around computer security, which honestly usually make a bigger difference. An indispensable guide for sysadmins and such. This is the third book of the security trilogy - it gets into practical issues in implementation and such: buffer overflows, message collisions, state, etc. Anyone who is serious needs all three.
This is more of a popular book on fun problems that even kids with secret decoders can enjoy. It is classic Gardner, and thus very worthwhile. Now this is an easy to read book that covers the basic ideas in a light but easy to read way. If you have no background this is a nice way to get an idea of what is going on. This is a chewier book than Martin Gardner's, but covers the common old ciphers and how to break them. I only recommend it for the die-hard fans.
A history book and more. I got this book for its coverage of the Enigma machine, but discovered it had lots more: languages, Babage vs. Vignere ciphers, Mary Queen of Scots' cipher, and more. Well worth it. Cryptography, the science of secret writing, by Laurence Dwight Smith This is more of a historical book, but it still has lots of easy problems with a note on how they were used. It covers transposition, substitution, and cryptanalysis. I didn't include a picture because they didn't have one. This is a great story of how Sarah Flannery became the young scientist of the year in Ireland and Europe for her work in encryption. It is worth reading on a number of levels: how to approach a problem, how to get your kids excited in math, and so on. It is a feel good book that didn't make me sick.
If you want to do work in algorithm and code design, you must know Number Theory (the branch of math used). This is an easy intro that still hits interesting topics. It is affordable too. A more advanced coverage of Number Theory, covering much of the same ground but with more on congruence and such which is important for Crypto. So you want a serious coverage? Not for the feint of heart, but you will really cover all the basics here. This is a great book (in coverage) by a legend - G.H. Hardy. If you aren't sure start with one of the other books.

Already read those? I generally suggest reading popular math or physics books.

Want to find your own good books? Search for yourself.