[SOLVED] Mangled accents in the 'link_title' field of the 'links' table

[liotier]

Hello, I am using Beta 9.7 (07.09.2007) since yesterday with default template. I am quite amazed how fast one can get a smooth Pligg site up and running - it seems like much work has already gone into this code. I still have to understand how to get "UrlMethod 2" to work, but I'm sure I'm the thousandth newby who will soon figure it out after digging the forums a bit...

What I currently can't figure out is why my accented characters are mangled when a link title is stored in the 'link_title' field of the 'links' table. For example, "é" and "ç" are stored wrong. I set up a first Pligg test site at #b^2 links / Published News to play with it. You can see examples of my problem at #b^2 links - Détournement de mobilier urbain pour un sondage (notice the mangled "é") and at #b^2 links - Chez moi ça marche (notice the mangled "ç").

In both cases the page is graphically fine but the URL is wrong, respectively http://infotain.ruwenzori.net/out.ph...our_un_sondage and http://infotain.ruwenzori.net/out.ph...%BF%BDa_marche in the above mentioned examples.

I started my database with utf8_unicode_ci and then tried with utf8_general_ci but that does not change anything.

Since the database record itself is mangled, I guessed after reading a few threads that the answer may have something to do with libs/utils.php. I found that "é" and "ç" were both missing from the list of "$output = str_replace" character replacements around line 260 of libs/utils.php. Can that be the cause of my problem ? I am the only French user who encountered that specific problem ? Has anybody figured what "$output = str_replace" lines should be added to support titles in the French language ?

[liotier]

I would have thought that with the history of issues with I8N that I found in the forums someone may have been able to hint me in the right direction...

[liotier]

Originally Posted by liotier

Since the database record itself is mangled, I guessed after reading a few threads that the answer may have something to do with libs/utils.php. I found that "é" and "ç" were both missing from the list of "$output = str_replace" character replacements around line 260 of libs/utils.php. Can that be the cause of my problem ?

Reflecting on that solution I thought that it would not work : the function I am talking about is named makeUrlFriendly() whereas my problem is sanitizing the title, not the URL.

But then more grepping led me to discover that this function is used in templates/yget/submit_step_3.tpl at line 58 to sanitize the title so that it can be used in the story's URL :

PHP Code:

$linkres->title_url = makeUrlFriendly($linkres->title); 


So I modified libs/utils.php so that makeUrlFriendly() supports my problematic accents :

PHP Code:

276a277,281
>    $output = str_replace("é", "e", $output);
>    $output = str_replace("è", "e", $output);
>    $output = str_replace("ç", "c", $output);
>    $output = str_replace("ù", "u", $output);
>    $output = str_replace("à", "a", $output); 


A few tests later I confirmed that I have solved my problem.

Now, can anyone add that in the dev tree ?

[liotier]

I forgot one... You may add the following to the list :

PHP Code:

$output = str_replace("ê", "e", $output); 


[beatniak]

Great job loitier!

Here are ALL the characters necessary for German / Turkish / Norwegian / Dutch / etc:

$output = str_replace("ì", "i", $output);
$output = str_replace("í", "i", $output);
$output = str_replace("î", "i", $output);
$output = str_replace("ï", "i", $output);
$output = str_replace("Ì", "I", $output);
$output = str_replace("Í", "I", $output);
$output = str_replace("Î", "I", $output);
$output = str_replace("Ï", "I", $output);
$output = str_replace("ò", "o", $output);
$output = str_replace("ó", "o", $output);
$output = str_replace("ô", "o", $output);
$output = str_replace("õ", "o", $output);
$output = str_replace("ö", "o", $output);
$output = str_replace("ø", "o", $output);
$output = str_replace("Ò", "O", $output);
$output = str_replace("Ó", "O", $output);
$output = str_replace("Ô", "O", $output);
$output = str_replace("Õ", "O", $output);
$output = str_replace("Ö", "O", $output);
$output = str_replace("Ø", "O", $output);
$output = str_replace("ù", "u", $output);
$output = str_replace("ú", "u", $output);
$output = str_replace("û", "u", $output);
$output = str_replace("ü", "u", $output);
$output = str_replace("Ù", "U", $output);
$output = str_replace("Ú", "U", $output);
$output = str_replace("Û", "U", $output);
$output = str_replace("Ü", "U", $output);
$output = str_replace("é", "e", $output);
$output = str_replace("è", "e", $output);
$output = str_replace("ê", "e", $output);
$output = str_replace("ë", "e", $output);
$output = str_replace("È", "E", $output);
$output = str_replace("É", "E", $output);
$output = str_replace("Ê", "E", $output);
$output = str_replace("Ë", "E", $output);
$output = str_replace("à", "a", $output);
$output = str_replace("á", "a", $output);
$output = str_replace("â", "a", $output);
$output = str_replace("ã", "a", $output);
$output = str_replace("ä", "a", $output);
$output = str_replace("å", "a", $output);
$output = str_replace("À", "A", $output);
$output = str_replace("Á", "A", $output);
$output = str_replace("Â", "A", $output);
$output = str_replace("Ã", "A", $output);
$output = str_replace("Ä", "A", $output);
$output = str_replace("Å", "A", $output);
$output = str_replace("ñ", "n", $output);
$output = str_replace("Ñ", "N", $output);
$output = str_replace("æ", "ae", $output);
$output = str_replace("Æ", "AE", $output);
$output = str_replace("ß", "ss", $output);
$output = str_replace("ç", "e", $output);
$output = str_replace("Ç", "C", $output);
$output = str_replace("ý", "y", $output);
$output = str_replace("ÿ", "y", $output);
$output = str_replace("Ý", "Y", $output);

[Elephant789]

Okay, but how would one go about it with Chinese characters?

[liotier]

I spotted an error in Beatniak's list :

$output = str_replace("ç", "e", $output);

is wrong and should be

$output = str_replace("ç", "c", $output);

[dollars5]

lol - I never saw it that deep - I even copied and used it on one of my sites - now fixing it.

[WebAbitibi]

From what I saw, before registering a title into your db, you're replacing your accents with non-accents letters ... that's not what you really want to do ... no?

actually, what you do with str_replace is to search for é and replace it with e, so in a title, you get

ORGINAL : Un avion est écrasé

AFTER STR_REPLACE : Un avion est ecrase

what I'm searching, is to register the title exactly as it is, no changes, with all the accents.

[liotier]

Originally Posted by WebAbitibi

From what I saw, before registering a title into your db, you're replacing your accents with non-accents letters ... that's not what you really want to do ... no?

[..]

what I'm searching, is to register the title exactly as it is, no changes, with all the accents.

I certainly agree with you, but just sanitizing the strings is a cheap way to get around the problem whereas properly i8nizing Pligg is a considerable endeavour that may not be a priority at the moment.