Grand bookcases can be humbled by a bad arrangement, while humble ones become grand with thoughtful accessorizing. There is no one rule for dressing bookcases--each needs a custom treatment.
That said, designer Charlie Birdsong has a few guidelines for making sure your shelves have decorative, as well as literary, appeal.
Tip: Use a tapestry of textures. Incorporate ceramics and leather or wood and silver. Be sure that the small items aren't so small that you can't see them, and keep the presentation of a collection from being so scattershot that it makes no sense.
Painting or papering the backs of bookcases deepens the space. I like to paint them dark olive drab. When papering, a good rule of thumb is to use small repetitive prints and paisleys.
Tip: Painted shelves should be a deeper shade to suggest shadow and depth.
Lighting is important. I like downlighting the backs of shelves to remove shadows and uplighting the fronts to illuminate the spines of the books and objects placed alongside them.
Sometimes there is a spot on a shelf for a small lamp. Sconces on the outside of open bookshelves are lovely with opaque, half-round shades. That way, light is directed just at the shelves and does not interfere with lighting in the rest of the room.
Realize you have fewer books than you think. Clients often tell me they need vast bookshelves because they have so many books. In truth, that is rarely the case. The boxes of books they have stored away will fill only a fraction of the space.
You also don't want to be stingy with books--there is something innately sad about thin clusters of books spread around vast shelves. In this case, too many is just right.
Place books loosely in some places and snugly in others; arrange them vertically and horizontally. If there are ample coffee table or art books, stack them horizontally with the spines telling a story--all on individual artists, gardening, or travel destinations, for example.
If a dust jacket on one of your large books is a favorite, try leaning it against the back of the shelf (like a painting). Or if you have a small painting that gets lost on a wall, place it between books on a shelf.