093: Couldn't import changeset with umlauts

(092)
Philipp Beckmann (reporter)
2010-02-15 10:07

This problem also exists under Linux. Import works, but all umlauts in the text are replaced by their UTF-8 codes. As a workaround, i encapsulated shell_exec and prepended a 'setlocale' & 'putenv' with the locale "de_DE.UTF-8".

(114)
oxomichael (reporter)
2010-03-01 09:03
edited on: 2010-03-01 09:06

I have the same problem with character encoding. I'm a french user.

For example, "à" is convert as "?\195?\160"

I work on Linux in utf8
My database is in utf8_general_ci

With mantis 1.1.x i simply add export LANG="fr_FR.UTF-8" at the begining of my post-commit script.

What is the best method to solve this issue ? utf8_encode or setlocale ?

(115)
Philipp Beckmann (reporter)
2010-03-01 09:23

utf8_encode fixed it on windows. Under Linux, it seems that shell_exec calls svn with a wrong locale (LOCALE="C").

(175)
obones (reporter)
2010-04-09 15:10

As indicated in 130, you need to add "LANG=de_DE.UTF-8" in front of the call to svn binary. And you also need the utf8_encode calls for the database to accept the values.

(176)
Philipp Beckmann (reporter)
2010-04-10 00:28

I already patched SourceSVN.php to call a new function svn_exec instead of shell_exec, which looks like this:

    private function svn_exec( $p_cmdline ) {
        $locale = 'de_DE.UTF-8';
        setlocale(LC_ALL, $locale);
        putenv('LC_ALL='.$locale);

        return( shell_exec( $p_cmdline ) );
    }

This works for me.

(200)
Philipp Beckmann (reporter)
2010-04-19 13:12
edited on: 2010-04-19 13:13

I've rewritten my hacks to combine both ways: setting the locale to en_US.UTF-8 (language isn't important) and encode latin to utf8, if necessary.

Please see attached patchfile. Tested with XAMPP 1.7.3 under Win7x64 and current Gentoo Linux with german UI.

(206)
Philipp Beckmann (reporter)
2010-04-26 08:04

Omitting 'setlocale' was not a good idea. Following code also works on Suse Linux:

$locale = 'en_US.UTF-8';
setlocale(LC_ALL, $locale);
putenv('LC_ALL='.$locale);

(207)
obones (reporter)
2010-04-27 05:59

setlocale is not safe under multithreaded web servers and should be avoided.
Prefixing the call to SVN with my proposed value is much safer in this regard and works just as well. This also avoids writing another function that calls the svn executable binary

(208)
Philipp Beckmann (reporter)
2010-04-27 07:15

Is there a way to prefix the call for windows?

(209)
obones (reporter)
2010-04-27 07:22

Hum, good point, I don't know of any. But for the windows part, I worked around this by reencoding the CP850 output into UTF-8.
Not sure which is the best method though...

(210)
Philipp Beckmann (reporter)
2010-04-30 12:37

Any known problems with putenv()? I successfully retested under Win7, Gentoo and Suse with:
    putenv('LANG=en_US.UTF-8');
    putenv('LC_ALL=en_US.UTF-8');
    $t_svnout = shell_exec( $svn );

    if( !mb_check_encoding( $t_svnout, 'UTF-8' ) ){
        $t_svnout = utf8_encode($t_svnout);
    }
setlocale is not neccessary, seems Gentoo and Suse listen to different envvars. Putenv works under Windows, it changes the UI of svn.exe to english on my german system.

Issue History
Date Modified	Username	Field	Change
2010-02-03 11:48	Philipp Beckmann	New Issue
2010-02-03 11:48	Philipp Beckmann	File Added: svn_output_utf8.patch
2010-02-15 10:07	Philipp Beckmann	Note Added: 092
2010-03-01 09:03	oxomichael	Note Added: 114
2010-03-01 09:06	oxomichael	Note Edited: 114	View Revisions
2010-03-01 09:23	Philipp Beckmann	Note Added: 115
2010-04-09 15:10	obones	Note Added: 175
2010-04-10 00:28	Philipp Beckmann	Note Added: 176
2010-04-19 13:04	Philipp Beckmann	File Added: SourceSVN.patch
2010-04-19 13:12	Philipp Beckmann	Note Added: 200
2010-04-19 13:13	Philipp Beckmann	Note Edited: 200	View Revisions
2010-04-26 08:04	Philipp Beckmann	Note Added: 206
2010-04-27 05:59	obones	Note Added: 207
2010-04-27 07:15	Philipp Beckmann	Note Added: 208
2010-04-27 07:22	obones	Note Added: 209
2010-04-30 12:37	Philipp Beckmann	Note Added: 210
2011-05-16 10:34	John Reese	Relationship added	has duplicate 205

Notes
(092) Philipp Beckmann (reporter) 2010-02-15 10:07	This problem also exists under Linux. Import works, but all umlauts in the text are replaced by their UTF-8 codes. As a workaround, i encapsulated shell_exec and prepended a 'setlocale' & 'putenv' with the locale "de_DE.UTF-8".

(114) oxomichael (reporter) 2010-03-01 09:03 edited on: 2010-03-01 09:06	I have the same problem with character encoding. I'm a french user. For example, "à" is convert as "?\195?\160" I work on Linux in utf8 My database is in utf8_general_ci With mantis 1.1.x i simply add export LANG="fr_FR.UTF-8" at the begining of my post-commit script. What is the best method to solve this issue ? utf8_encode or setlocale ?

(115) Philipp Beckmann (reporter) 2010-03-01 09:23	utf8_encode fixed it on windows. Under Linux, it seems that shell_exec calls svn with a wrong locale (LOCALE="C").

(175) obones (reporter) 2010-04-09 15:10	As indicated in 130, you need to add "LANG=de_DE.UTF-8" in front of the call to svn binary. And you also need the utf8_encode calls for the database to accept the values.

(176) Philipp Beckmann (reporter) 2010-04-10 00:28	I already patched SourceSVN.php to call a new function svn_exec instead of shell_exec, which looks like this: private function svn_exec( $p_cmdline ) { $locale = 'de_DE.UTF-8'; setlocale(LC_ALL, $locale); putenv('LC_ALL='.$locale); return( shell_exec( $p_cmdline ) ); } This works for me.

(200) Philipp Beckmann (reporter) 2010-04-19 13:12 edited on: 2010-04-19 13:13	I've rewritten my hacks to combine both ways: setting the locale to en_US.UTF-8 (language isn't important) and encode latin to utf8, if necessary. Please see attached patchfile. Tested with XAMPP 1.7.3 under Win7x64 and current Gentoo Linux with german UI.

(206) Philipp Beckmann (reporter) 2010-04-26 08:04	Omitting 'setlocale' was not a good idea. Following code also works on Suse Linux: $locale = 'en_US.UTF-8'; setlocale(LC_ALL, $locale); putenv('LC_ALL='.$locale);

(207) obones (reporter) 2010-04-27 05:59	setlocale is not safe under multithreaded web servers and should be avoided. Prefixing the call to SVN with my proposed value is much safer in this regard and works just as well. This also avoids writing another function that calls the svn executable binary

(208) Philipp Beckmann (reporter) 2010-04-27 07:15	Is there a way to prefix the call for windows?

(209) obones (reporter) 2010-04-27 07:22	Hum, good point, I don't know of any. But for the windows part, I worked around this by reencoding the CP850 output into UTF-8. Not sure which is the best method though...

(210) Philipp Beckmann (reporter) 2010-04-30 12:37	Any known problems with putenv()? I successfully retested under Win7, Gentoo and Suse with: putenv('LANG=en_US.UTF-8'); putenv('LC_ALL=en_US.UTF-8'); $t_svnout = shell_exec( $svn ); if( !mb_check_encoding( $t_svnout, 'UTF-8' ) ){ $t_svnout = utf8_encode($t_svnout); } setlocale is not neccessary, seems Gentoo and Suse listen to different envvars. Putenv works under Windows, it changes the UI of svn.exe to english on my german system.