Notice: this is a static mirror for historical purposes.

View Issue Details Jump to Notes ] Related Changesets ] Issue History ] Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
126Source IntegrationWebSVNpublic2010-04-01 08:202010-04-09 07:15
Assigned ToJohn Reese 
PrioritynormalSeverityminorReproducibilityhave not tried
Product Version 
Target VersionFixed in Version0.16 
Summary126: infinite import on merged svn-repository
DescriptionI have merged two SVN-Repositories into one. When I try to import the latest data into mantis I get an "infinite loop".
(fyi: I have installed the plugin as you have described in your blog and it works with other repos.)

In your blog you say it is no problem if there is an timeout on large repositories - just start it again and again till it has finished.
So for me it is a problem, because my repos is merged out of two my revision-log is not sortable by the timestamp-column.

so what happens:
1. I started the import
2. the import had a timeout
3. I started the import again
4. in SourceSVN.php->import_full you select the revisions from changeset-table ordered by timestamp-column
5. you take the first row out of this ordered result and start importing with the following changesets
6. for my repos this is not the latest changeset because the ordering is wrong
7. the import imports changesets it already has imported and gets a timeout again
8. it all starts again at 3. :)

Now I have changed SourceSVN.php. In import_full I replaced the order by timestamp by an order by revision and now it works.

That is what my select looks like now:
$t_max_query = "SELECT revision FROM $t_changeset_table
                        WHERE repo_id=" . db_param() . '
                        ORDER BY CAST( revision AS UNSIGNED ) DESC';
Steps To ReproduceI think it will be hard to reproduce because you need a svn-repository with unordered timestamps for the revisions that is large enough to produce timeouts on import.
But if you need some information please feel free to ask me.
Additional Informationexample how my repos looks like

Revision Timestamp
1 2009-03-01
2 2009-11-24
3 2010-02-15
4 2008-03-15 (this is where I started to merge the 2nd repos in)
5 2009-04-30
6 2010-03-28

(Rev 1-3 are from Repos 1; Rev 4-5 are merged from repos 2 into repos 1; rev 6 was normaly checked-in after merging)
TagsNo tags attached.
Attached Files

- Relationships

-  Notes
User avatar (150)
John Reese (administrator)
2010-04-02 14:11

Will investigate.
User avatar (155)
obones (reporter)
2010-04-08 04:51

I have the exact same behavior, and it's even worse. There's no need to import again, the loop occurs with a very small number of revisions.
In my case, it imports revisions 1 through 201, then starts again at 1 until it errors out on an "out of memory" error. I don't even have to restart it, it loops on its own.
After it crashed, I had more than 100000 rows in the changeset table, all referring to revisions 1 through 201
Applying the proposed fixed above sorted the situation out, all revisions are now imported just fine.

Just for background, the repository has more than 43800 revisions and is the result of the merge of about 10 separate dumps made from another SCM system
User avatar (159)
John Reese (administrator)
2010-04-08 12:16

My biggest concern about the proposed fix is whether that SQL code is portable across implementations. Ie, will it work correctly on MySQL, Postgres, and MSSQL at the very least?
User avatar (162)
obones (reporter)
2010-04-08 12:38

Well, that syntax works under those three as it is part of the SQL standard.
The "unsigned" keyword is MySQL specific though, so it should be better to use "integer" instead.
User avatar (163)
John Reese (administrator)
2010-04-08 12:47

That's exactly the problem: "unsigned" only works in Mysql, but just "integer" doesn't. It seems that even the ANSI standard CAST() isn't portable enough due to differences in data types... =\ Either way, I've committed the fix for Mysql to a separate branch "bug126" that I've pushed to the public repository. At least for the time being, I'll try to keep that in sync with master until I can find a better solution.
User avatar (165)
obones (reporter)
2010-04-08 12:52

According to the following page, Integer should work all by itself: [^]
User avatar (166)
John Reese (administrator)
2010-04-08 12:56

Database query failed. Error received from database was #1064: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTEGER ) DESC LIMIT 1' at line 3 for the query: SELECT revision FROM mantis_plugin_Source_changeset_table
WHERE repo_id='5'
User avatar (167)
obones (reporter)
2010-04-08 13:08

Yeah, you're right, the list of types supported by cast is not the list of types supported for a column type... That sucks.
User avatar (172)
as (reporter)
2010-04-09 06:52

That is what I saw, too. So I just made it work for me for my current installation and that is MySQL as you can see.
But I'm "happy" to see that I'm not the only one with this problem. Sorry obones ;)

Can we fix this with DECIMAL instead of int?

The cast to decimal should work for MySQL ( [^]) and for MSSQL it seems to work also. Postgres I don't know.

I have not tested this with the plugin if decimal would work...
User avatar (173)
John Reese (administrator)
2010-04-09 07:15

Fix committed to master branch at 2010-04-08 19:35:52.

- Related Changesets
Source Integration: master b9ef2021
Timestamp: 2010-04-08 19:35:52
Author: John Reese
Committer: John Reese
Details ] Diff ]
Fix 126: Sort SVN changesets by numeric revision

To better handle merged SVN repositories, it is necessary to retrieve
the latest changeset from the database based on integer revision number
rather than by timestamp. This will prevent an "infinite loop" of
repeatedly importing the same changesets in cases where the latest
revisions were actually committed before earlier changesets.

Using ANSI SQL's `CAST(? AS DECIMAL)` as it seems to be compatible with
the most database platforms.
mod - SourceSVN/SourceSVN.php Diff ] File ]

- Issue History
Date Modified Username Field Change
2010-04-01 08:20 as New Issue
2010-04-02 14:11 John Reese Note Added: 150
2010-04-02 14:11 John Reese Assigned To => John Reese
2010-04-02 14:11 John Reese Status new => acknowledged
2010-04-08 04:51 obones Note Added: 155
2010-04-08 12:16 John Reese Note Added: 159
2010-04-08 12:38 obones Note Added: 162
2010-04-08 12:47 John Reese Note Added: 163
2010-04-08 12:52 obones Note Added: 165
2010-04-08 12:56 John Reese Note Added: 166
2010-04-08 13:08 obones Note Added: 167
2010-04-09 06:52 as Note Added: 172
2010-04-09 07:15 John Reese Note Added: 173
2010-04-09 07:15 John Reese Status acknowledged => resolved
2010-04-09 07:15 John Reese Fixed in Version => 0.16
2010-04-09 07:15 John Reese Resolution open => fixed
2010-04-09 07:15 John Reese Changeset attached master b9ef2021 =>

Copyright © 2000 - 2012 MantisBT Group
Time: 0.1453 seconds.
memory usage: 8,387 KB
Powered by Mantis Bugtracker

hosted with