Changes to Places memory consumption, expiration and performances

As part of the Memshrink effort, we reduced the SQLite page cache used by Places.  This is not a leak, the page cache is actively used by SQLite to improve performances by reducing disk I/O. Originally Places was allowed to use a large cache (maximum 6% of physical memory, per connection), to ensure responsiveness of the awesome bar.

Starting from the 20110901 Nigthly, the following changes have been made to Places:

- the cache size (per connection) is now the minimum between 2% of physical memory and half of the database size, with a minimum value of 5MiB. This is the maximum amount of memory SQLite may use for the cache, but it isn't reserved or used unless needed.

- expiration tries to target a maximum database size (you can check current value through the places.history.expiration.transient_optimal_database_size pref, changing this pref won't have any effect), this is not mandatory, but is used to improve the algorithm. This also takes into account storage with low available space or quota.

- some users, with a really huge (tens of thousands) amount of bookmarks, were constantly losing history due to automatic expiration trying to keep the database to a sane size, from now on history of the last 7 days will not be considered expirable. You can obviously still remove these visits through Clear Recent History, manual deletes (see below) or add-ons.

- some users, who used Weave before native GUIDs were added to Places, may still have obsolete Weave annotations, Richard Newman removed those, measuring in some cases a database size shrink of 8%! Nice work Richard!

There is some work left to do (as always), especially copying over the cache_size pragma to cloned connections, since right now only the main connection cache is correctly limited, but we have 4 connections.

Some add-on was also found inflating the cache size, so, if you notice an extremely high value for places.sqlite cache-used in about:memory, try Safe Mode. If that solves the problem file a bug, reporting the culprit add-on, so we can work to figure out a solution.

As a final note, we identified a regression causing removals from history being painfully slow, it has been fixed and should now be possible to manually cleanup history in a decent amount of time. Unfortunately this still happens on the UI thread, but it is planned to move it to a separate thread asap, to further improve UI responsiveness.

If you notice any misbehavior or regression that may be related to these changes, please report it.

MaK Venerdì 02 Settembre 2011 at 08:45 am | | Mozilla-EN