I cannot reproduce your problem. Could you post the words and characters that are giving you this problem? I've created a category, story title and story description using the following Georgian alphabet:
Code:
აბგდევზთიკლმნოპჟრსტუფქღყშჩცძწჭხჯჰ
The result is a working category URL, and none of the characters have been translated into HTML hexidecimal values. This is using the latest SVN version of Pligg, but I assume that RC5 would give you the same result. I attached a screenshot (please ignore the ugly template in progress) as proof that it's working for me according to the example characters I posted above.
You may want to look into whether any of the characters you are entering need to be added to /languages/translit.txt, but I don't think that would be related to the original post.