Notice: this is a static mirror for historical purposes.

View Issue Details Jump to Notes ] Issue History ] Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
093Source IntegrationWebSVNpublic2010-02-03 11:482011-05-16 10:36
ReporterPhilipp Beckmann 
Assigned To 
PrioritynormalSeveritymajorReproducibilityalways
StatusnewResolutionopen 
Product Version 
Target VersionFixed in Version 
Summary093: Couldn't import changeset with umlauts
DescriptionImporting a changeset with umlauts in the log fails with following error:

 #1366: Incorrect string value: '\xFCgt' for column 'message' at row 1 für die Abfrage: INSERT INTO mantis_plugin_Source_changeset_table ( repo_id, revision, parent, branch, user_id,
timestamp, author, message, info, ported, author_email, committer, committer_email, committer_id
) VALUES ( ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ? ).
Additional InformationAdded a patch, which solved the issue in my environment by converting the output of svn.exe from ISO to UTF-8.
TagsNo tags attached.
Attached Files

- Relationships
has duplicate 205resolvedJohn Reese Couldn't import changeset with special german characters (umlaut) 

-  Notes
User avatar (092)
Philipp Beckmann (reporter)
2010-02-15 10:07

This problem also exists under Linux. Import works, but all umlauts in the text are replaced by their UTF-8 codes. As a workaround, i encapsulated shell_exec and prepended a 'setlocale' & 'putenv' with the locale "de_DE.UTF-8".
User avatar (114)
oxomichael (reporter)
2010-03-01 09:03
edited on: 2010-03-01 09:06

I have the same problem with character encoding. I'm a french user.

For example, "à" is convert as "?\195?\160"

I work on Linux in utf8
My database is in utf8_general_ci

With mantis 1.1.x i simply add export LANG="fr_FR.UTF-8" at the begining of my post-commit script.

What is the best method to solve this issue ? utf8_encode or setlocale ?

User avatar (115)
Philipp Beckmann (reporter)
2010-03-01 09:23

utf8_encode fixed it on windows. Under Linux, it seems that shell_exec calls svn with a wrong locale (LOCALE="C").
User avatar (175)
obones (reporter)
2010-04-09 15:10

As indicated in 130, you need to add "LANG=de_DE.UTF-8" in front of the call to svn binary. And you also need the utf8_encode calls for the database to accept the values.
User avatar (176)
Philipp Beckmann (reporter)
2010-04-10 00:28

I already patched SourceSVN.php to call a new function svn_exec instead of shell_exec, which looks like this:

    private function svn_exec( $p_cmdline ) {
        $locale = 'de_DE.UTF-8';
        setlocale(LC_ALL, $locale);
        putenv('LC_ALL='.$locale);

        return( shell_exec( $p_cmdline ) );
    }

This works for me.
User avatar (200)
Philipp Beckmann (reporter)
2010-04-19 13:12
edited on: 2010-04-19 13:13

I've rewritten my hacks to combine both ways: setting the locale to en_US.UTF-8 (language isn't important) and encode latin to utf8, if necessary.

Please see attached patchfile. Tested with XAMPP 1.7.3 under Win7x64 and current Gentoo Linux with german UI.

User avatar (206)
Philipp Beckmann (reporter)
2010-04-26 08:04

Omitting 'setlocale' was not a good idea. Following code also works on Suse Linux:

$locale = 'en_US.UTF-8';
setlocale(LC_ALL, $locale);
putenv('LC_ALL='.$locale);
User avatar (207)
obones (reporter)
2010-04-27 05:59

setlocale is not safe under multithreaded web servers and should be avoided.
Prefixing the call to SVN with my proposed value is much safer in this regard and works just as well. This also avoids writing another function that calls the svn executable binary
User avatar (208)
Philipp Beckmann (reporter)
2010-04-27 07:15

Is there a way to prefix the call for windows?
User avatar (209)
obones (reporter)
2010-04-27 07:22

Hum, good point, I don't know of any. But for the windows part, I worked around this by reencoding the CP850 output into UTF-8.
Not sure which is the best method though...
User avatar (210)
Philipp Beckmann (reporter)
2010-04-30 12:37

Any known problems with putenv()? I successfully retested under Win7, Gentoo and Suse with:
    putenv('LANG=en_US.UTF-8');
    putenv('LC_ALL=en_US.UTF-8');
    $t_svnout = shell_exec( $svn );
    
    if( !mb_check_encoding( $t_svnout, 'UTF-8' ) ){
        $t_svnout = utf8_encode($t_svnout);
    }
setlocale is not neccessary, seems Gentoo and Suse listen to different envvars. Putenv works under Windows, it changes the UI of svn.exe to english on my german system.

- Issue History
Date Modified Username Field Change
2010-02-03 11:48 Philipp Beckmann New Issue
2010-02-03 11:48 Philipp Beckmann File Added: svn_output_utf8.patch
2010-02-15 10:07 Philipp Beckmann Note Added: 092
2010-03-01 09:03 oxomichael Note Added: 114
2010-03-01 09:06 oxomichael Note Edited: 114 View Revisions
2010-03-01 09:23 Philipp Beckmann Note Added: 115
2010-04-09 15:10 obones Note Added: 175
2010-04-10 00:28 Philipp Beckmann Note Added: 176
2010-04-19 13:04 Philipp Beckmann File Added: SourceSVN.patch
2010-04-19 13:12 Philipp Beckmann Note Added: 200
2010-04-19 13:13 Philipp Beckmann Note Edited: 200 View Revisions
2010-04-26 08:04 Philipp Beckmann Note Added: 206
2010-04-27 05:59 obones Note Added: 207
2010-04-27 07:15 Philipp Beckmann Note Added: 208
2010-04-27 07:22 obones Note Added: 209
2010-04-30 12:37 Philipp Beckmann Note Added: 210
2011-05-16 10:34 John Reese Relationship added has duplicate 205


Copyright © 2000 - 2012 MantisBT Group
Time: 0.1566 seconds.
memory usage: 8,434 KB
Powered by Mantis Bugtracker

hosted with
Linode