Hi,
We have a juju2 environment (now running juju 2.1.1) where the juju.txns mongo collection is getting out of control :
juju:PRIMARY> db.txns.count()
34831682 <= nearly 35M
juju:PRIMARY> db.txns.stats()["size"]
16801507852 <= that's 16GB
juju:PRIMARY> db.txns.stats()["storageSize"]
5176877056 <= that's 5GB
juju:PRIMARY> db.txns.stats()["totalIndexSize"]
459661312 <= that's 450MB
On this environment (where HA is enabled, we have 3 controllers), mongodb is regularly killed by Linux' OOM-killer. I have enabled DEBUG logging for juju.txn, and I'm fairly convinced that the oom-kills are happening when all the txns are gathering. In the logs, we see the following :
2017-03-27 10:45:57 INFO juju.txn prune.go:43 txns after last prune: 206688, txns now = 34715286, pruning required: true
2017-03-27 10:45:57 DEBUG juju.txn prune.go:148 loading all completed transactions
But we never see "found %d completed transactions across %d collections", and the code suggests we should (https://github.com/juju/txn/blob/master/prune.go#L148).
According to what's stored in the juju.txns.prune collection, our last successful pruning dates back to March 10th :
juju:PRIMARY> db.txns.prune.find({"_id": "last"}).pretty()
{ "_id" : "last", "id" : ObjectId("58c2b1fbc2b5ab5367292bf5") }
juju:PRIMARY> db.txns.prune.find({"_id": ObjectId("58c2b1fbc2b5ab5367292bf5")}).pretty()
{
"_id" : ObjectId("58c2b1fbc2b5ab5367292bf5"),
"started" : ISODate("2017-03-10T13:59:36.751Z"),
"completed" : ISODate("2017-03-10T14:02:35.747Z"),
"txns-before" : 592869,
"txns-after" : 206688
}
This is rather concerning, as the problem will only get worse.
In the same prune.go file, we can read that (https://github.com/juju/txn/blob/master/prune.go#L137) :
// This set could potentially contain many entries, however even
// 500,000 entries requires only ~44MB of memory. Given that the
// memory hit is short-lived this is probably acceptable.
Which means that 35M entries require 70*44MB = ~3GB of RAM, and we simply don't have this kind of free memory on our controllers (which have 8GB each).
Even after the last prune, there were 206k transactions remaining. Isn't that a bit high for a controller hosting ~60 models ? I took a look at oldest txn object, and it's the transaction that creates the controller model, adds the "admin" user, etc.. In essence, the very first transaction. Isn't it supposed to be purged at some point ?
The controller model also has some very old transactions in its txn-queue, the first of which says to update "available-tools" for the controller model to "2.0.1", so quite old. Is that one supposed to still exist as well ?
To recap my questions :
1) could you make juju / mgo _not_ read every existing transaction in memory ? I think this should be very much avoided.
1a) how do we get our environment to purge transactions ? Do you have a workaround / manual mongodb query we could apply ?
2) shouldn't the juju.txns collection be capped ? As I understand it, it will only get pruned if the number of current transactions is twice the number of transactions after the last purge, allowing for a unlimited (and potentially exponential) growth
3) is it normal that some transactions stay in the mongodb database forever ?
4) what removes transactions from the txn-queue ? I believe some documents are still in the juju.txns collection because they're still referenced in some txn-queue.
Thanks
> 1) could you make juju / mgo _not_ read every existing transaction in memory ? I think this should be very much avoided.
Agreed that the current approach doesn't scale and needs to be fixed.
I think what we should store the working set in a temporary MongoDB collection. This can be done efficiently using MongoDB's aggregation pipeline features (the $out operator is particularly useful) and the bulk update features. Note that although it'll be fairly efficient it does trade memory usage for disk I/O. For this reason transaction pruning is still something that we don't want to do too often.
> 1a) how do we get our environment to purge transactions ? Do you have a workaround / manual mongodb query we could apply ?
One of the things the mgopurge tool does is purge completed transactions. You need to shut down the controllers to run it so it's pretty invasive.
https:/
The better solution is make Juju do a better job of purging itself.
> 2) shouldn't the juju.txns collection be capped ? As I understand it, it will only get pruned if the number of current transactions is twice the number of transactions after the last purge, allowing for a unlimited (and potentially exponential) growth
It can't be capped in the usual sense. As long as a transaction is referenced by a document somewhere.
> 3) is it normal that some transactions stay in the mongodb database forever ?
Yes that is expected. The last transaction to touch a document stays in the document's txn-queue field.
Also, when transaction is "assert only" (doesn't change anything) the txn-queue field is only appended to, with previous completed transactions being left behind. We should fix this behaviour in mgo/txn as it can lead to excessive amounts of transactions being left behind which can't be cleaned up.
I've just filed this: https:/
> 4) what removes transactions from the txn-queue ? I believe some documents are still in the juju.txns collection because they're still referenced in some txn-queue.
Yep that's correct. A previous transaction id is removed from a document's txn-queue when the next data modifying transaction touches it. Some documents are never touched after creation so the creating transaction hangs around. I suspect changing this behaviour in mgo/txn will be non-trivial.
Aside from the above, I think we should make the pruning schedule more aggressive. As you've noted the current approach of only pruning when the transaction count has doubled is not really working. There is a balance to be found though - given that pruning is an expensive operation, we don't want to do it too often.