Conversation

Notices

1. admin de gnusocial.net (administrator@gnusocial.net)'s status on Thursday, 01-Dec-2022 06:58:42 JST admin de gnusocial.net

   in reply to

   • Alexandre Oliva
   @lxo
   
   1) The "trick" of adding "?page=2" not only lets you browse in a paginated way, but also fixes the issue of any page getting stuck.
   
   2) Even with more page loads, most of the content is cached, so it is downloaded only once.
   
   3) I meant what I said: being only you using that method, the server is fine, just banning your IP ocasionally. If there were more massive petitions like yours, it could not hold all the load (by the way, it has happened again today).
   • Alexandre Oliva (lxo@gnusocial.net)'s status on Thursday, 01-Dec-2022 06:58:44 JST Alexandre Oliva
     erhm... I'd love to understand what you mean by that. there are two sources of uncertainty: (i) what I do has changed very significantly over the past week, from (a) using the infinite-scrolling web interface that downloaded an incremental update every 16 posts/threads, which added up to some 50 pages per day; the odds of some of these failing, requiring me to start over and fetch several updates again, or binary-search for the point where I was, was hitting me 2 or 3 times a day, each generating waste of time for me and wasted resources on the server (I'd occasionally load 21 or 42 pages at once after such failures to get back on track; was that ever a problem?); to the failed experiment (b) batch-downloading thousands of posts and merging them into a single web page with a week's worth of posts, that DoSed my browser and the server with requests for avatars and whatnot; and finally to (c) breaking up the downloaded posts into groups of a few hundred, some 20 paginated views every 8h.
     with the current arrangement, I batch the page downloads and the avatar downloads when displaying it, which might hit the server harder for a short time, but it should be more efficient in that the avatars and the pages would be downloaded one way or another, but now I'm actually saving server resources by reducing redundant downloads and possibly even improving cache use in multiple layers, by increasing locality and freshness, like batch-processing usually does