I sort them by genre, and alphbetically within genres. If a movie doesnt really fit into one of the gebres I have, I'll usually just put it with horror [IE: The Machinist, Eraserhead]
Here's a REALLY outdated pic to show how I store them. I have two of these now...all the single DVDs are on one, and all the boxsets and VHS tapes are in the other.
This is the tail end of horror, followed by "Mystery" (which is now mixed in with either horror or Drama), the whole comedy section, plus the first of Action.