Just how many PMs are going around, anyway?

[Jenne]

Ok, just took 35 pages of IMs down to one measly page.

Didja read 'em?  I found myself down this weird, spiralling trip of PD.com Memory Lane. 

Yeah it was kind of depressing, really...  five years of fights mostly.

  But MY pm's to you were GREAT! 

[hooplala]

Ok, just took 35 pages of IMs down to one measly page.

Didja read 'em?  I found myself down this weird, spiralling trip of PD.com Memory Lane. 

Yeah it was kind of depressing, really...  five years of fights mostly.

  But MY pm's to you were GREAT! 

Yes, yours were a ray of sunshine.   

[Jenne]

[Triple Zero]

I was actually thinking of asking people to flush all their old pm's if they have a lot, there area great many people who have sent pm's over the years that have probably built up to a very substantial amount.

Why?  What's the benefit?

With three thousand members and years of pm's built up for many people it could take up a good bit of space.

I don't think it's that much compared to the  number of posts. Three thousand members aren't really all active.

And then, what about the space? I thought slow database queries were the problem? We have enough diskspace, right?

And I don't think the PMs make general forum browsing slower, can you base that on anything but guesswork? It's probably not even the same tables,except for the user tables.

I'm sorry but all this deleting of stuff, while we're not even really sure if it actually helps? Or are we? Damn those people at the SMF support forum for not answering my questions.

[Jenne]

(woot! I see my words in 000's sig thingie!)

...I dunno, seems making room and cleaning house is a good thing for diskspace anyhow.  Especially when folks like me have upwards of 20 pages of stuff...doubled for sent messages, even.  But maybe I just like the idea I'm doing my civic duty for PD.com.

[hooplala]

Shit, I never even thought about my sent messages... 

[Jasper]

Just killed like 9 pages of ancient PM history.

Tell your server it owes me one.

[Dimocritus]

Incidentaly, as soon as I clicked "less than 1" I received multiple PM's. Some one is trying to make a liar out of me...

[Triple Zero]

(woot! I see my words in 000's sig thingie!)

...I dunno, seems making room and cleaning house is a good thing for diskspace anyhow.  Especially when folks like me have upwards of 20 pages of stuff...doubled for sent messages, even.  But maybe I just like the idea I'm doing my civic duty for PD.com.

I just don't like the idea of throwing away stuff. Then again, from a privacy point of view, you totally should, who knows where that database might end up when some crazy spag goes a bit too far and we get subpoena'd

As well as, diskspace is truly not an issue. Last time I looked, there were several backup copies of the database in the root dir of our server account, which are previous backups made by Faust that he only deletes every once in a while [for extra moar redundancy].

The issue why we are deleting and/or pruning big threads is because of database lookup times.

Think of this database like a big stack of cross referenced phonebooks. You can find a number in a phonebook pretty quickly because the names are alphabetical, right? Databases work just like that. Except sometimes you don't want to look up something by its name, but by the street, so you need another phonebook which is sorted by street. So you got a lot of phonebooks, one for every thing you want quick access to. That and hash tables, which are a littlebit like how you can quickly find a CD in your CD collection [that you haven't sorted], because you know the cover is blue. You just scan for all the blue CDs and only check the title of the blue ones. Combine that with caching, which is similar to having a vague recollection of where you most probably put the CD back last time you handled it and checking there first.

So that's kind of how it works. Now the thing is, if you ask the database something complicated, it takes quite a while before it makes an answer, and the server is tied up and the forum becomes slow. This happens mostly when someone does a search query on the forum, which is why we used to have that feature disabled for a while back.

But on average, I'm not really sure, rendering one page of a forum thread takes about 10-20 database queries, not just one.

However, the forum software only uses the tables [phonebooks] related to PMs when someone is loading their Private Message page, or sending a private message. I'm really not sure but if people only get/send 1-5 PMs a week, it seems to me that the load on the server this causes would be negligible.

On the other hand, since the cunts at the SMF support forum never answered my question, this is all just educated guesswork from me as well. Probably a tad biased from the fact that I do not like to throw away information, since datastorage is so fucking cheap and we really got loads of it.

It would be awesome if the database/forum software could somehow figure out that really old threads that nobody ever reads anymore can be stored in a separate phonebook that always gets checked last. This would be just as good as pruning all those threads, except that you can access them if you want. However, I have no idea whether SMF can do this.

Still, the cost of looking something up in a phonebook is related logarithmically to the number of entries in a phonebook. This means that if a phonebook twice the size would cost 0.1 seconds more time, four times would take 0.2, and 32 times as big would only cost 0.5 seconds to search. Now unless there is some kind of extra [linear] cost to a long thread, I don't even see how long threads would produce such a load on the server. And yeah I ploughed through the SMF code quite a lot but it's pretty hard to zoom out and see everything that it does for a single page view.

But if Faust repeatedly observed speedups after pruning big threads, I can't argue with empirical evidence

[the last yatto]

I have this theory that the majority of this forum lives in a barren personal message desert, while a few high-rollers hog all of them. 

i have about 26 pages worth of pms

[Faust]

(woot! I see my words in 000's sig thingie!)

...I dunno, seems making room and cleaning house is a good thing for diskspace anyhow.  Especially when folks like me have upwards of 20 pages of stuff...doubled for sent messages, even.  But maybe I just like the idea I'm doing my civic duty for PD.com.

I just don't like the idea of throwing away stuff. Then again, from a privacy point of view, you totally should, who knows where that database might end up when some crazy spag goes a bit too far and we get subpoena'd

As well as, diskspace is truly not an issue. Last time I looked, there were several backup copies of the database in the root dir of our server account, which are previous backups made by Faust that he only deletes every once in a while [for extra moar redundancy].

The issue why we are deleting and/or pruning big threads is because of database lookup times.

Think of this database like a big stack of cross referenced phonebooks. You can find a number in a phonebook pretty quickly because the names are alphabetical, right? Databases work just like that. Except sometimes you don't want to look up something by its name, but by the street, so you need another phonebook which is sorted by street. So you got a lot of phonebooks, one for every thing you want quick access to. That and hash tables, which are a littlebit like how you can quickly find a CD in your CD collection [that you haven't sorted], because you know the cover is blue. You just scan for all the blue CDs and only check the title of the blue ones. Combine that with caching, which is similar to having a vague recollection of where you most probably put the CD back last time you handled it and checking there first.

So that's kind of how it works. Now the thing is, if you ask the database something complicated, it takes quite a while before it makes an answer, and the server is tied up and the forum becomes slow. This happens mostly when someone does a search query on the forum, which is why we used to have that feature disabled for a while back.

But on average, I'm not really sure, rendering one page of a forum thread takes about 10-20 database queries, not just one.

However, the forum software only uses the tables [phonebooks] related to PMs when someone is loading their Private Message page, or sending a private message. I'm really not sure but if people only get/send 1-5 PMs a week, it seems to me that the load on the server this causes would be negligible.

On the other hand, since the cunts at the SMF support forum never answered my question, this is all just educated guesswork from me as well. Probably a tad biased from the fact that I do not like to throw away information, since datastorage is so fucking cheap and we really got loads of it.

It would be awesome if the database/forum software could somehow figure out that really old threads that nobody ever reads anymore can be stored in a separate phonebook that always gets checked last. This would be just as good as pruning all those threads, except that you can access them if you want. However, I have no idea whether SMF can do this.

Still, the cost of looking something up in a phonebook is related logarithmically to the number of entries in a phonebook. This means that if a phonebook twice the size would cost 0.1 seconds more time, four times would take 0.2, and 32 times as big would only cost 0.5 seconds to search. Now unless there is some kind of extra [linear] cost to a long thread, I don't even see how long threads would produce such a load on the server. And yeah I ploughed through the SMF code quite a lot but it's pretty hard to zoom out and see everything that it does for a single page view.

But if Faust repeatedly observed speedups after pruning big threads, I can't argue with empirical evidence

The indexing works perfectly at. Its the sheer size of the db that causes it to lock to the important stuff slowly, the server hardware isn't amazing and the larger the database is the slower the forum goes.
If you think it won't help then leave it. Its just when I log into the mysql menu for the site, the pm table does show up as open and with half a meg overhead.
Leave it for now so, I will look into it and see. And yes, smf are useless at answering any question related to the db and not some superficial modification for the forum.
As to the linearity of the search times and size of the db.
When the site was 500 erroring on every third or fourth request and I did the huge prune of old threads it stopped happening. And I know for a fact I didn't get rid of more then half the DB.

[Cramulus]

holy god I have 76 pages in my inbox, 47 pages in my outbox

[Jenne]

holy god I have 76 pages in my inbox, 47 pages in my outbox

HIGHROLLINGHOG HIGHROLLINGHOG HIGHROLLINGHOG

Actually, you should see how many I have at EB&G...oy oy oy (sorry Bella and BAI!)

[Cain]

What is wrong with you people?

[Jenne]

   Cram's popular.  Actually, he's in a lot of projects, so I bet he gets a lot of pm's based on that.

(yes, I"m PURELY shooting the shit here)