Places got Async Expiration
A better expiration component is now part of Places module
Last week, on Friday, i've pushed the last pieces of the new Places Expiration component. This was one of the Firefox Projects intended for 1.9.3 branch, you can find more background on the start of this project in its wiki page.
Some background
Originally expiration was managed by History component itself on three major steps: after each visit, during idle, at shutdown. This had various drawbacks. First of all it was making navigation experience laggish, so we moved the after each visit step to be after each sync between memory and disk tables. We also reduced idle expiration and shutdown expiration.
The result was better, but we had other issues: we were not expiring enough pages related to the number of visits, and the sync component was now bloated with non-related functionality (And slower). We were also still doing a bunch of stuff at shutdown.
In bug 516940 i cleaned up the shutdown stuff, while increasing separation between History and Expiration, at that point was easier to split it out of History in a separate component.
So, what's new?
The new component is a JS component, it runs expiration in steps, every 3 minutes, with a simple adaptive algorithm, so that if the last step did not expire enough, the next one will be run later, while if it finds more items than the expired ones, the next step will expire more! This should ensure we don't lag behind with expiration.
It also uses async Storage API, this ensures that we run I/O in a separate thread, so we won't hurt your navigation.
Expiration on idle will run just a single larger step, then it'll stop till you exit idle, this way it won't kill your standby or batteries. Expiration on shutdown runs a larger step, but not too large, in most cases the adapative expiration steps should still ensure we don't expire on shutdown.
What has changed for you?
The new component is able to detect your hardware specs, especially memory size, and adapt expiration to it, this means you don't need anymore to tweak number of days of history, or whatever. For this reason we have removed the number of days field from the preferences panel, you don't need anymore to tell us how much days of history your computer can handle.
What about privacy? Well, we discussed about that obviously and we went to a conclusion that the days field was not giving back any real privacy gain. Sure i could have set it to 6 days, but that would have not protected me since:
- The days pref was an "at least" pref, so for most users it was really a fake-change
- Being expiration async per definition, you can't be sure when pages are phisically expired
- even if you reduce history to 6 days, nobody can ensure you don't have bad entries in these days
Since we have better privacy tools (And we can even build new ones, so feel free to suggest changes and file enh bugs about that) like Clear Recent History, Private Browsing and Forget about this page/site, the choice was pretty clear, we want real privacy, not fake-privacy.
Also hidden expiration preferences have gone, so browser.history_expire_days, browser.history_expire_days_min, browser.history_expire_sites are now replaced by a single places.history.enabled preference. No more need to read preferences manuals just to make the browser feel faster.
What can you tweak? Ideally you don't need to tweak anything, and i suggest you don't touch any pref. Btw, for the sake of information we have two new hidden preferences: places.history.expiration.interval_seconds is number of seconds between each expiration step, while places.history.expiration.max_pages is maximum number of pages that we will retain before expiring. We make our best to have satisfying default values for anyone, current values are built to be pessimistic, we will evaluate how we behave with them, and eventually increase them in future, if we feel that's needed.
What you got finally
Faster, easier and more secure expiration with smoother navigation. All of this at the price of fake-privacy. Win win.
34 commenti
You can read the currently used value in places.history.expiration.transient_max_pages and set an higher value in places.history.expiration.max_pages.
But i discourage that, we will try to increase the value as far as we can, i expect that actually we will already keep much more history than before. So ideally, you won’t need to touch anything.
Two questions.
places.history.expiration.interval_seconds and places.history.expiration.max_pages are both stealth, right? (won’t show up unless you create them manually)
Secondly, is there any other expiration mechanism than max_pages? For example, by age? Has such a mechanism existed sometime in mozilla-central?
I ask this because I had a lot of history (with min expiry set to 9999 days) and I noticed anything older than 6 months was removed even though I wasn’t near places.history.expiration.transient_max_pages at all.
Also, what does “page” mean? A “visit”? An unique page?
Finally I think the limit should be made logarithmic and truncated – as it stands a single core with 512MB of RAM will store less than 8000 pages, and a multicore with 4096MB will store over 125000, this is too much variation IMO. And besides aiming for a blanket 6% of RAM is not reasonable for large RAM amounts (250MB just for history is just way too much, even if you have 4GB of RAM).
Thanks for your work!
yes those prefs are hidden.
No, by age expiration has been removed for the above reasoning, if you see expiration happening when unneded please file a bug.
by pages we mean unique pages, regardless number of visits per each.
The limit will be revised with further tweaks. the 6% of memory has always been in place from Firefox 3, but notice that’s an upper limit. that means that we won’t usually use 6% of memory for history, but that we can’t use more than that. It’s a limit we can’t go over, that’s all.
I have another question of you don’t mind. With the old system visits older than the expiration cutoff were deleted. Once a page had no visits it was removed.
If I understand correctly, with the new system a page and its visits are removed once the page limit is reached. But doesn’t this mean that the number of visits is unbounded? For example if you visited the same 40 pages every day for a long time, nothing would ever be expired (and even if you visit other pages, these 40 along with their visits would stay).Or has the visits thing changed too?
yes visits are unbound. visits don’t take lot of space in the database, and with the size of the web it’s really hard that you are going to visit the same page over and over without visiting any other page.
This allows casual users to avoid losing history if they are not heavy history users.
Also, the old system was removing visits over 90 days only if a page limit was reached, so this is similar, just is like we removed the 180 days limit that many users were already removing by themselves.
What about users who want to keep say a week’s worth of History and then have it deleted not because of hard-disk space or privacy, but just to keep clutter out of the urlbar list? I suspect many users were using this pref for this kind of reason.
Again, Mozilla REMOVING pref UI AND the hidden pref counterpart, for no reason. Fine, make this change be the default, maybe it is better, but I still want a hidden pref, so I can override that behavior. I don’t want Firefox to delete history until I tell it too. This new method NEEDS to be overridable at least by some hidden pref.
“you don’t need anymore to tell us how much days of history your computer can handle.”…I didn’t use it to “tell us how much days” (how many days) my computer could handle, I used to to specify how much history I wanted to keep/see. Now you removed that ability, so if I update, it will likely truncate my history…& later, when you tweak this feature to keep “more” history, it won’t matter, my history will already be truncated by this version.
I’m all about new versions & doing things in better ways (maybe this new algorithm is great!), but removing hidden prefs has to stop! Only advanced users use hidden prefs, so leave them alone!
Also, if you are posting articles in English & comments in English, why is the “chrome” of the website not English? (Example: “Inserisci Commento” “Anteprima Commento”)
Ok. So if I understand correctly, its not possible to auto remove entries by date. Currently I have about 65000 entries. And entries older than a year. (It would be a lot more if I wouldn’t delete some of the useless crap manually).
I have the new places.history.expiration.transient_current_max_pages set to 128801. So that means items will only be deleted after around two years. That feels needless to me. It seems like a loss to me that I can’t set a time limit.
Also. I can’t really understand why I have more than a year old entries in the first place since I have (had? its still there in about:config) expire day set to 300.
While I’m here I’d like to ask why do I have entries with 0 visits? Why are they stored? It seems all of them are download links, so something I really never visited.
if you really need to tweak values you can set places.history.expiration.max_pages to a value less than places.history.expiration.transient_max_pages
The fact you had history set to 300 days but you have older entries is one of the reasons the time limit was mostly useless.
Entries with 0 visits could be stored either because they are bookmarks, or just because they have not yet been expired. if they stay after some time I’d suggest to file a bug.
So, let’s say I have an i7 with 4G of RAM and a smart phone synced via Firefox Sync; FF4 on the smartphone will keep a limited amount of history entries due to its limited resources, but Sync will download the complete history every time, «no problem, just disable history syncing», sure, let’s disable a feature that worked perfectly fine since now because you decided to “solve” a non existent issue.
This is wrong for so many reasons. They were hidden preferences, so what’s the point in removing them? I don’t want an adaptive algorithm to decide for me, I know what’s best for me, don’t go around crippling the browser for everyone because uneducated users can’t bother reading documentation: set a default and let power users tweak whatever they want.
Hi!
On my computer your algorithm set places.history.expiration.transient_current_max_pages to 257676
I think it is way too much and it hunted performance really badly.
I had history way back to February 2009.
I then manually deleted history from 2009 and my FF is significantly faster!
You should consider lowering those numbers…
(if you need any more info please contact me)
We can for sure tweak numbers if we find those are excessive, a large number by itself is not a issue though, so that should be done only if we cannot improve performances.
I’d really be interested in knowing which parts of the browser you feel faster after removing history, to work on performances.
Mainly typing in AwsomeBar.
It was really lagging. I use it for search also with search prefixes (g for google, wiki for wikipedia etc.) and when I for example type ‘g firefox ‘ it sometimes searches for ‘firefo’ (without last few letters). Somehow it registers enter before those letters.
I am very upset by this change in firefox 4
I used to keep all my history (9999 days) incase I forget where is the page, I can always search in the history.
Yes, it will slow down firefox a bit and I acknowledge it, but for me at least it cannot compare the convenience of provide my full browsing history when I want.
Now with this new ff4, my place.sqlite file was trimmed down from 330M to 20M!
And I lost everything more than 1 month!, as long as the remembered password! (password is there but not pop out from the page I need them)
Please at least provide some option for us, don’t act on behave of us without asking. I know better what I want
Is there a way to retrieve my old pages? the profile folder now has a 340M place.sqlite.corrupt file
@offal the current settings are probably the same as your old settings, or with very similar. Your problem is due to a corrupt database that caused us to create a new one, and unfortunately this currently causes history loss, this could happen for filesystem problems or if you touch your database with third party cleaning software.
Please contact support at http://support.mozilla.com/
Why are you putting of some options of Firefox when you make an update ? I remember the time when I could open my home page in each new tab oppened, and when I could choose about my history.
I agree with the fact that it can increase the browser security, but why not just save in this history the LINKS of the sites, and not the cookies where can lies viruses ? It’s safer, and we could enter the number of days we wan’t to keep.
Ok, I’m not looking to invent a suggest, but I think removing this perf was not a good idea, and the Firefox community is waking up about that (see forums)
I cannot agree more :
[i]I am very upset by this change in firefox 4
I used to keep all my history (9999 days) in case I forget where is the page, I can always search in the history….
Is there a way to retrieve my old pages? the profile folder now has a 340M place.sqlite.corrupt file[/i]
Excuse me but this is outrageous – you should at least leave the about:config entries alone for those who want them !!!!
@all Please, read the article before commenting, nobody posted valid use-cases that have not already been answered.
People complaining about the missing option in the Preferences dialog, should realize that options was not an hard limit, we were already retaining more history that what you wanted. So nothing changed.
People complaining about lack of 9999 days of history can still use the new prefs to obtain the same old result, but I think the new settings are largely covering their needs without need of tweaking, indeed we retain far more useful history than before. The situation improved for you.
People complaining they can’t set a lower limit, should really think about their needs, since both privacy and performances have been largely answered and were not improved by the old pref. We just stopped making you think a lower limit would have improved your performance or privacy, both were untrue myths.
No you are not keeping more history than before – I actually lost much of my history on “upgrating” – I realized this too late. And I needed this info ! Of course we read the article before posting – the point is we do not want FF to take us by the hand – we want to be able to keep as much history as we want – 0 to 9999 (9999 for me) days, not pages.
Why don’t you just accept you did a (big) mistake ?
Please explain how can I use the new preferences to achieve this ! ?
I cannot even change places.history.expiration.transient_current_max_pages.
And I don’t want to count in pages – I want to count in days
Who told you FF users want their browser decide for them ? I would use IE if that were the case !
@Luchino: as I said, please read the article, there is clearly a section saying “What can you tweak?”, and you are asking how to change the preferences that is clearly explained there.
I want infinite (unlimited) history for one particular website and heuristic history for everything else. How do I do that? Rexpr URL patterns would be nice.
What is the meaning of the key: browser.history_expire_days.mirror ? and its value: 180?
It isn’t at the Mozillazine Knowledge Base.
I updated from FF3.6 to FF4, and I have the key: browser.history_expire_days_min, value: 2.
Do you recommend to change this value or delete it?
Thanks.
@pduff it’s not possible, an add-on could do something like that though.
@3lsiguienteprog browser.history_expire_days.mirror has no effect, it’s stored for UI reasons. in Firefox 4 browser.history_expire_days_min is unused, you can Reset it and it will disappear.
Since Netscape 2 my basic Web design, used first at work and for about 13 years also at home, has used History as a bookmark for reading. Nothing else: I have no use for storing any Web page on my hard disk. Nor do I care about privacy: I expect that everyone who can access my computer, including myself, to be able to access everything on my hard disk. So for many years I have set History expiration to three (3) days. I can still do that with IE which I attempt never to use. Removing that preference completely destroyed the ability to use History expiration as a bookmark.
What about “places.history.expiration.interval_seconds” that I saw above? Does it work? I added it and set it to 259200 seconds, three days if my calculator is correct. But I wanted to write this comment now instead of waiting three days. If it works, and if no better solution is available, I’ll add appropriate instructions to my home page. Please reply with a copy to my e-mail address. Grazie.
Until recently, I have continued using FF2 (instead of upgrading to more recent versions) because of the “Enhanced History Manager” add-on. It provided me with the same history features as Netscape, such as the ability to scroll through the entire history in a single list with “first visited” and “last visited” fields; and the ability to search either the title or the location with various options such as “starts with” and “doesn’t contain”. I kept the number of days at 9999; and yes, that slowed down things a bit, but being able to re-construct information that I remembered having seen years earlier has been extremely valuable in my job.
Well, I have no choice but to abandon FF2 because too many sites don’t work with it any more. I now see that, in addition to giving up the useful, effective and convenient history interface of FF2, I also have to give up the ability to keep years of history. I am not surprised. This is part of a well-established trend in software: the more things are dumbed down, the more it becomes usable by a broader population, even though you make it less convenient for experienced users who, anyway, have no choice. What alternative do I have, Internet Explorer?!? Plleeeez!!
The dumbing down of software has been pioneered by Microsoft, and, as I said, it’s not surprising that Mozilla is following their lead. I am now using both Windows XP and Windows 7 on different computers. I still haven’t found a single thing that W7 does better than XP, but I have a long list of useful features that have been eliminated. I’ll continue using XP as long as I can, but I am sure that the time will come when I am forced to “upgrade” to W7, as I am now forced to “upgrade” to FF4.
I have read some comments here and also at the mozilla.support.seamonkey newsgroup that some users are upset by the fact that entries in the browser history can no longer be automatically expired according to the age of the entry. That is, we can no longer set a preference to expire any entry that is more than 30 days old.
Bug #660646 at https://bugzilla.mozilla.org/show_bug.cgi?id=660646 requests that this capability be restored. This bug will likely NOT be treated seriously unless more users log onto bugzilla.mozilla.org and vote for this bug. (Nevertheless, votes do not necessarily result in implementation.)
cannot change the value of
places.history.expiration.transient_current_max_pages
In fact you can change it but the browser is going to restore the old value anyway (a value that is way too big to my taste). I think you’ll maybe answer that the old setting was deceiving, anyway, so that the real solution is in fact removing the setting altogether.
You are terrible developers. The privacy you removed wasn’t fake at all, that’s just a bad excuse for the removal of this feature.
If I want a dumbed down browser that offers no options to the user, I’ll use Chromium. Actually, I’ll change to Chromium right now.
It’s fine that you guys want this as the new default settings in have not problem with that, but DO NOT take away the users ability to tweak FF to our liking. To me it’s like having your garbage only picked up every 6 months and them telling you that they think you might want to dig thru it to find something you might want later, BS. I want to be able set my history for 3 days!
I play several online games (loads of repeated visits, like >1000 visits per day) and that seriously degrade my FF ;(
How can I limit say those games site visits, but remember my other (research related) history for ~3months?
For everyone complained about missing expire by days, check http://blog.bonardo.net/2011/08/05/a-cou..
thanks! i am not very tekkie, but i have a question:
is there any other tools option that i might have tweaked that has made it so that now i only have a couple weeks of history? ( i would love to have more like 1 or 2 months to refer back to!)
e.g. would the advanced- network tab- limit cache to —— MB be one of those?
thanks again for your help! :)
But what if we want to keep /more/ history than the heuristic allows?