Richard Jones' Log: (Update) Roundup benchmark time again

Sun, 25 Apr 2004

Update: have added timings for N=100,000 (or thereabouts), which equates to 87,000 or so open issues.

Roundup 0.7 includes some new optimisations, specifically automatic generation of a number of table indexes in the RDBMS backends.

I've added a script "tools/" which loads a tracker with N issues and N/100 users. Random distribution of priorities, statuses and assignetos. Approximately 85% of issues are open (ie. 17 in 20, 170 in 200, etc).

Against a stock classic tracker, loading the standard index page (sort by activity, group by priority, showing batches of 50 issues at a time):

Backend N Index Page Time
anydbm 200 3.5s
sqlite 20,000 1.1s
100,000 4.5s (9.4s)
metakit 20,000 1.9s
100,000 8.4s (18s)
mysql 20,000 1.3s
45,000 2.1s
65,000 3.3s (5.7s)
postgresql 20,000 1.1s
75,000 2.7s
100,000 4.1s (5.7s)

Note:The second (bracketed) values indicate the time for a concurrant index page view (I press reload in a second browser window shortly after requesting the first). The sqlite and metakit backends serialise accesses, so their second access time is understandably doubled.

Comment by James Kew on Tue, 20 Apr 2004

I was thinking to myself "anydbm was much slower than that for me" until I noticed the two-orders-of-magnitude difference in test size between the anydmb hash and the RDBMS tests.

Nice. And I'm gladder still that I migrated to sqlite a while ago...

Comment by Martin on Thu, 29 Apr 2004

Any particular reason why you do not use the same n's for all dbs/except maybe anydbm) ?


Comment by Richard on Thu, 29 Apr 2004

The intention was to do that, but the load time for mysql was just too long (it took over 12 hours to load 65,000). I do intend to finish loading mysql with the 100,000 issues, and I'll update the list then.

Comment by Alex Chekholko on Fri, 30 Apr 2004

Indeed! Our bsddb3 roundup tracker has slowed to a crawl with only ~400 issues(quickly growing). The main index page with about 50 open issues takes about 10s to load. The server is a P3-933 with enough RAM, running nothing else at the moment, so it looks like we'd greatly benefit from switching backends.

Comment by Richard on Wed, 12 May 2004

bsddb3 appears to have the worst performance of all the backends. I'm not sure why yet, but it really, really sucks.

Comment by Eddie Parker on Wed, 26 May 2004

Would be nice if there was a unit time displayed no? IPT / N, so we could get a sense of what was truly fastest?

Also, is there an existing benchmark for 0.6.xx somewhere kicking around, so we can see the speed differences?