Places got Async Expiration
A better expiration component is now part of Places module
Last week, on Friday, i've pushed the last pieces of the new Places Expiration component. This was one of the Firefox Projects intended for 1.9.3 branch, you can find more background on the start of this project in its wiki page.
Some background
Originally expiration was managed by History component itself on three major steps: after each visit, during idle, at shutdown. This had various drawbacks. First of all it was making navigation experience laggish, so we moved the after each visit step to be after each sync between memory and disk tables. We also reduced idle expiration and shutdown expiration.
The result was better, but we had other issues: we were not expiring enough pages related to the number of visits, and the sync component was now bloated with non-related functionality (And slower). We were also still doing a bunch of stuff at shutdown.
In bug 516940 i cleaned up the shutdown stuff, while increasing separation between History and Expiration, at that point was easier to split it out of History in a separate component.
So, what's new?
The new component is a JS component, it runs expiration in steps, every 3 minutes, with a simple adaptive algorithm, so that if the last step did not expire enough, the next one will be run later, while if it finds more items than the expired ones, the next step will expire more! This should ensure we don't lag behind with expiration.
It also uses async Storage API, this ensures that we run I/O in a separate thread, so we won't hurt your navigation.
Expiration on idle will run just a single larger step, then it'll stop till you exit idle, this way it won't kill your standby or batteries. Expiration on shutdown runs a larger step, but not too large, in most cases the adapative expiration steps should still ensure we don't expire on shutdown.
What has changed for you?
The new component is able to detect your hardware specs, especially memory size, and adapt expiration to it, this means you don't need anymore to tweak number of days of history, or whatever. For this reason we have removed the number of days field from the preferences panel, you don't need anymore to tell us how much days of history your computer can handle.
What about privacy? Well, we discussed about that obviously and we went to a conclusion that the days field was not giving back any real privacy gain. Sure i could have set it to 6 days, but that would have not protected me since:
- The days pref was an "at least" pref, so for most users it was really a fake-change
- Being expiration async per definition, you can't be sure when pages are phisically expired
- even if you reduce history to 6 days, nobody can ensure you don't have bad entries in these days
Since we have better privacy tools (And we can even build new ones, so feel free to suggest changes and file enh bugs about that) like Clear Recent History, Private Browsing and Forget about this page/site, the choice was pretty clear, we want real privacy, not fake-privacy.
Also hidden expiration preferences have gone, so browser.history_expire_days, browser.history_expire_days_min, browser.history_expire_sites are now replaced by a single places.history.enabled preference. No more need to read preferences manuals just to make the browser feel faster.
What can you tweak? Ideally you don't need to tweak anything, and i suggest you don't touch any pref. Btw, for the sake of information we have two new hidden preferences: places.history.expiration.interval_seconds is number of seconds between each expiration step, while places.history.expiration.max_pages is maximum number of pages that we will retain before expiring. We make our best to have satisfying default values for anyone, current values are built to be pessimistic, we will evaluate how we behave with them, and eventually increase them in future, if we feel that's needed.
What you got finally
Faster, easier and more secure expiration with smoother navigation. All of this at the price of fake-privacy. Win win.
sei commenti
You can read the currently used value in places.history.expiration.transient_max_pages and set an higher value in places.history.expiration.max_pages.
But i discourage that, we will try to increase the value as far as we can, i expect that actually we will already keep much more history than before. So ideally, you won’t need to touch anything.
Two questions.
places.history.expiration.interval_seconds and places.history.expiration.max_pages are both stealth, right? (won’t show up unless you create them manually)
Secondly, is there any other expiration mechanism than max_pages? For example, by age? Has such a mechanism existed sometime in mozilla-central?
I ask this because I had a lot of history (with min expiry set to 9999 days) and I noticed anything older than 6 months was removed even though I wasn’t near places.history.expiration.transient_max_pages at all.
Also, what does “page” mean? A “visit”? An unique page?
Finally I think the limit should be made logarithmic and truncated – as it stands a single core with 512MB of RAM will store less than 8000 pages, and a multicore with 4096MB will store over 125000, this is too much variation IMO. And besides aiming for a blanket 6% of RAM is not reasonable for large RAM amounts (250MB just for history is just way too much, even if you have 4GB of RAM).
Thanks for your work!
yes those prefs are hidden.
No, by age expiration has been removed for the above reasoning, if you see expiration happening when unneded please file a bug.
by pages we mean unique pages, regardless number of visits per each.
The limit will be revised with further tweaks. the 6% of memory has always been in place from Firefox 3, but notice that’s an upper limit. that means that we won’t usually use 6% of memory for history, but that we can’t use more than that. It’s a limit we can’t go over, that’s all.
I have another question of you don’t mind. With the old system visits older than the expiration cutoff were deleted. Once a page had no visits it was removed.
If I understand correctly, with the new system a page and its visits are removed once the page limit is reached. But doesn’t this mean that the number of visits is unbounded? For example if you visited the same 40 pages every day for a long time, nothing would ever be expired (and even if you visit other pages, these 40 along with their visits would stay).Or has the visits thing changed too?
yes visits are unbound. visits don’t take lot of space in the database, and with the size of the web it’s really hard that you are going to visit the same page over and over without visiting any other page.
This allows casual users to avoid losing history if they are not heavy history users.
Also, the old system was removing visits over 90 days only if a page limit was reached, so this is similar, just is like we removed the 180 days limit that many users were already removing by themselves.
But what if we want to keep /more/ history than the heuristic allows?