Straightforward-ish, here's some step by step details to ensure you see it working:
Go to the Add Entry page
Type stuff
Add "topCategory" to the category list (type it then click "add")
Post
Go to the Add Entry page
Type stuff
Add "topCategory|nestedCategory" to the category list (type it then click "add")
Post
Go to the front page, and you should see "topCategory" displayed in the category list, with "nestedCategory" indented below it
The key for this to work, is that all categories must have at least one post associated with them, and "topCategory" and "topCategory|nestedCategory" are treated as distinct categories.