The URLs you are seeing are correct. The point of URL method 2 is to have "clean" URLs and ones that are more search engine friendly. When you renamed your htaccess file, did you put a period in front of it? It should be called .htaccess. Did you edit the line where you put your sub-folder if you're installed in one?
For your second question, try taking out the extra pipe (in red) and see if that fixes it
Code:
RewriteRule ^(all|system|)/([a-zA-Z0-9-]+)/?$ story.php?title=$2 [L]
RewriteRule ^(all|system|)/?$ ?category=$1 [L]