IRC logs for #openttd on OFTC at 2020-10-29
⏴ go to previous day
00:15:20 <glx> <FLHerne> This is also used on string literals that never end up as in-game strings, like grfids and track labels :-/ <-- ah yes looks like parse_string_to_dword() use grfstrings when it should not
08:10:33 *** sla_ro|master has joined #openttd
08:26:10 *** longtomjr has joined #openttd
08:45:07 *** WormnestAndroid has quit IRC
08:45:19 *** WormnestAndroid has joined #openttd
08:56:33 *** andythenorth has joined #openttd
09:21:47 *** HerzogDeXtEr has joined #openttd
10:19:33 *** andythenorth has joined #openttd
10:26:38 *** Progman has joined #openttd
10:45:18 *** andythenorth has joined #openttd
12:02:55 *** HerzogDeXtEr1 has joined #openttd
12:54:37 <TrueBrain> funny .. I think the repository frosch123 created is marged to use LFS, but is not really using it
12:54:54 <TrueBrain> mainly as we would have hit the bandwidth limits by now already, easily .. and I don't have git-lfs installed and I have the files :P
12:55:32 <TrueBrain> well, maybe it is better to not use LFS :P
13:40:03 *** andythenorth has joined #openttd
13:51:45 *** snail_UES_ has joined #openttd
14:16:38 <TrueBrain> maybe it motivates people to fix shit :D
14:52:44 *** supermop_Home has joined #openttd
14:53:11 <supermop_Home> well andy I found a pretty good American Cornish yarg analog
14:53:32 <supermop_Home> well, from vermont
14:59:23 <supermop_Home> its hay wrapped instead of nettle wrapped though
14:59:38 <supermop_Home> don't really have nettles on the east coast for some reason
15:00:08 <supermop_Home> and slightly firmer, but flavor profile is there
15:01:01 <andythenorth> I am told that nettles are fairly uniquely British
15:01:05 <andythenorth> dunno how true that is
15:52:02 <supermop_Home> Wikipedia would suggest no?
16:05:31 <FLHerne> TrueBrain: Is it no longer possible to download grfs directly from the bananas wobsite?
16:05:48 <FLHerne> (or only for some grfs, for some reason?)
16:05:52 <TrueBrain> it is not; well, it hasn't for a few months now :)
16:07:14 <FLHerne> Is that a deliberate policy, or not-implemented-and-who-cares?
16:07:56 <TrueBrain> someone claimed to made a patch for it .. but .. it was only words on IRC
16:08:15 <TrueBrain> I am a bit disappointed that in your spectrum it is either: policy <-> we don't care :P
16:08:20 <TrueBrain> there are things in between there :)
16:09:05 <FLHerne> Well, the 'few months' indicates a fairly low bound on the level of caring ;-)
16:09:15 <TrueBrain> clearly the community doesn't care, indeed :)
16:09:25 <TrueBrain> you are only the second person to notice and ask about it :P
16:09:55 *** ChanServ sets mode: +v tokai
16:10:13 <TrueBrain> we should use one of these "what-feature-to-work-on-next" pages :D
16:10:22 <TrueBrain> (as it is of course a matter of priority, more than anything else)
16:11:46 <FLHerne> I don't know if a new scheme would fix them anyway
16:12:05 <TrueBrain> yeah .. people were never suppose to link to those files directly to start with ..meh
16:12:11 <FLHerne> Since it would presumably use sensible names rather than whatever-the-author-uploaded...
16:12:30 <TrueBrain> what do you mean, sorry?
16:14:31 <FLHerne> I mean, the old links are like `bananas/newgrf/JapanSet_Landscape-3.1.tar.gz`, where the last part is I think just whatever the grf author uploaded it as?
16:14:56 <TrueBrain> no; it is the name of the BaNaNaS entry + the version entered there
16:15:08 <TrueBrain> BaNaNaS always repacks everything users upload, to sanitize stuff .. like the tar name
16:15:16 <TrueBrain> the current BaNaNaS still does exactly the same
16:15:33 <FLHerne> Whereas the new system puts more information into the filenames, so it wouldn't be the same unless explicitly trying to be
16:15:35 <TrueBrain> (in other words, 0 chance on name collisions :D)
16:16:06 <TrueBrain> the new URLs are a bit more clever, as it contains the md5sum too
16:16:14 *** Wormnest has joined #openttd
16:16:18 <TrueBrain> but that is merely an implementation detail :D
16:16:55 <TrueBrain> (in reality, the names are stored in AWS S3 as <md5sum>.tar.gz, and a Lambda@Edge rewrites the URLs to match <md5sum>/<anything>.tar.gz to <md5sum>.tar.gz)
16:17:15 <TrueBrain> hmm, no, that word can be left out completely
16:17:22 <TrueBrain> you get what I mean I hope :)
16:17:47 <TrueBrain> but anyway, these URLs always have been a bit problemetic; in the old BaNaNaS, these URLs only existed as long as it was the latest upload of that GRFID
16:17:58 <TrueBrain> upload a newer version, and the link becomes invalid
16:18:23 <TrueBrain> I hope in the new BaNaNaS we can have an URL pointing to bananas.openttd.org first, which might redirect you to the CDN, or return you a nicer error telling youwhat is going on
16:18:43 <TrueBrain> but it first requires demand to become anywhere near the list of "to implement in 2021" :P
16:18:50 <TrueBrain> I had more requests to re-add download statistics, for example
16:19:58 <FLHerne> Although I feel like those were self-perpetuating :p
16:20:00 <TrueBrain> so I would say, on that Wiki page, the links should be changed to the BaNaNaS URL (the non-download)
16:20:21 <FLHerne> People sorted by 'most downloaded' and downloaded them
16:20:36 <FLHerne> On the basis that those must be the best ones, right?
16:20:39 <TrueBrain> yeah, I never liked the way we implemented the old system
16:20:49 <TrueBrain> I was planning to do it a bit differently this time around
16:20:54 <FLHerne> (and not just the oldest)
16:20:59 <TrueBrain> and only show the last 90 days or something, and "downloads per day" or what-ever
16:21:03 <TrueBrain> still allows for this behaviour
16:21:06 <TrueBrain> but should weaken it a bit
16:21:37 <TrueBrain> and also track per entry, not per version
16:21:46 <TrueBrain> to avoid being punished for uploading a new version
16:23:27 <TrueBrain> it still isn't perfect, but it should help
16:26:23 *** andythenorth_ has joined #openttd
16:49:07 <TrueBrain> right ... lot of bookkeeping more, but TrueWiki can now (locally) track metadata with a cache
16:49:16 <TrueBrain> if a Page changes, it knows what to invalidate and what to rebuild
16:49:18 <TrueBrain> instead of everything
16:49:28 <TrueBrain> means it no longer takes 90+ seconds EVERY startup :P
16:50:16 <TrueBrain> it does how-ever consume a bit more memory .. and I might have made a mistake :P
16:50:40 <TrueBrain> I would rather have made scanning for metadata faster, but .. I have no clue how atm ... it is CPU-bound, but not in Python-land ..
16:56:55 <TrueBrain> haha, no, this invalidation approach does not work :P To easy to make the metadata-cache to become inconsistent
17:06:00 *** andythenorth has joined #openttd
17:08:46 *** sla_ro|master has joined #openttd
17:39:37 *** andythenorth has joined #openttd
17:53:51 *** gelignite has joined #openttd
17:56:52 *** andythenorth has joined #openttd
18:12:09 *** heffer_ has joined #openttd
18:20:24 <milek7_> another wiki moving to git, it seems
18:36:45 <TrueBrain> GitHub even. And that is because it is just a good idea :D
18:51:50 <DorpsGek_III> - Update: Translations from eints (by translators)
19:13:50 *** andythenorth has joined #openttd
19:30:46 *** WormnestAndroid has quit IRC
19:31:31 *** WormnestAndroid has joined #openttd
19:33:01 *** frosch123 has joined #openttd
19:58:06 *** Wormnest has joined #openttd
19:59:04 *** Speeder_ has joined #openttd
20:58:15 <TrueBrain> frosch123: am I right to conclude that you have in fact not used Git LFS for the wiki-data repo? :D
20:59:27 <frosch123> what makes you think so?
20:59:34 <TrueBrain> two things that surprised me: 1), the quota is 1GB per month ... it should be 300MB, and I downloaded it at least 4 times already
20:59:40 <TrueBrain> 2) I do not have Git LFS installed, and I have the files :P
21:00:14 <frosch123> what size is your .git folder?
21:00:47 <TrueBrain> difference is most likely a GC run
21:01:05 <TrueBrain> there are no attributes on the files after a git clone
21:03:52 <TrueBrain> this might not be a bad thing .. it seems if we use Git LFS, we will have to buy some data packs for GitHub
21:03:54 <frosch123> everything on that page says 0
21:04:07 <TrueBrain> yeah, if you have only public repos, most things will be 0
21:04:15 <TrueBrain> from what I understand, Git LFS is not one of them
21:04:21 <TrueBrain> but this is what I was researching
21:04:33 <TrueBrain> when I found out my git-lfs on WSL2 is in fact a Windows binary :P (and cannot be executed)
21:04:52 <TrueBrain> so I tried to find out how I was cloning the files .. to find out they were without LFS markers
21:05:08 <TrueBrain> it is still very unclear to me how this all works exactly tbh :P Hence the question :D
21:05:29 <TrueBrain> 420M Oct 24 23:28 pack-98f6ef7f21d57767b67b72eb4040fc5a69753ac3.pack
21:05:42 <TrueBrain> looking at the size, I receives the files via a git pack .. which I assume means not via LFS :P
21:05:49 <TrueBrain> but again .. not sure that is a bad thing :)
21:06:18 <TrueBrain> git repos can become up to 1GB before GitHub starts to be less friendly, and at 5GB they start to ask you to fix it, from what I understand
21:06:35 <TrueBrain> not sure how big git becomes if you import the history
21:06:54 <TrueBrain> but honestly, we can just prune the history of File/** every year or so, and nobody would ever notice
21:07:14 <frosch123> the images have no deep history
21:07:36 <TrueBrain> so maybe we should just not use LFS :) Well, we cannot have files of 100+MB, but ... I assume we don't want those anyway :D
21:08:09 <frosch123> already saw :) quite nice
21:08:34 <TrueBrain> I am mostly battling the start-up times atm .. not sure what to do with it
21:08:44 <TrueBrain> so I guess I am going to leave it alone for a bit, and first make the "edit" part
21:11:17 <frosch123> TrueBrain: my initial goal with git lfs was to make git log faster
21:11:28 <TrueBrain> that is a good idea indeed
21:11:39 <TrueBrain> I was considering using the GitHub API for it :P
21:11:40 <frosch123> the first repo that contained the complete history took 20 seconds or longer to show the history of a file
21:11:53 <frosch123> somehow git has to uncompress all diffs to figure out whether a file is involved
21:12:12 <frosch123> i assumed that would be faster with lfs
21:12:57 <TrueBrain> never worked with LFS; no clue tbh :)
21:13:10 <TrueBrain> from what I read, the files committed to the git itself are 130 bytes or so
21:13:18 <TrueBrain> so yeah, I can imagine it would make git operations quicker :)
21:13:40 <TrueBrain> I wonder how GitHub does it tbh
21:14:05 <frosch123> they have their own index :)
21:15:05 <TrueBrain> not a terrible thing for us :)
21:16:13 <TrueBrain> anyway, I think I did everything for the view part we should have for a v1
21:16:22 <TrueBrain> well, with the exception of caching headers
21:16:46 <frosch123> do we have something to show folders?
21:17:02 <TrueBrain> did you still want that? Didn't we say Categories should be doing that?
21:17:08 <TrueBrain> (I am fine with both, btw, honest question)
21:17:31 <frosch123> last idea was some Folder: category
21:17:33 <TrueBrain> we can make a /Folder/en/Manual page for example
21:17:55 <TrueBrain> and list there folders / files in a similar way as Categories do
21:18:06 <TrueBrain> hmm .. would be /Folder/Page/en/Manual, I guess
21:18:17 <TrueBrain> as having /Folder/Category/ would also be needed, I guess
21:18:38 <TrueBrain> that is currently fully up to that page
21:18:50 <TrueBrain> in this case of course pressing Manual in the navigation bar there
21:18:55 <TrueBrain> so this page it is relative simple
21:19:09 <frosch123> so, maybe the header could show the path, where you can click upwards, and some "show list of files in current directory"
21:19:20 <TrueBrain> okay, so we talk about two different things here
21:19:27 <TrueBrain> you want a breadcrumb, I am guessing?
21:19:56 <TrueBrain> how I read you, it splits up in 2 feature requests: navigation through the folders, and listing content of folders
21:20:12 <TrueBrain> the navigation part should be solved by the navigation panes, I think
21:20:20 <TrueBrain> not sure there needs more bloating on the page for that
21:20:29 <TrueBrain> sure, we need to revise the navigation panes, as some are terrible
21:21:00 <frosch123> TrueBrain: i was hoping we could get some automatic categories :)
21:21:14 <frosch123> so people do not just dump pages somewhere
21:21:19 <TrueBrain> well, that is the listing part I guess
21:21:28 <TrueBrain> that wiki you show, also has the same information in 2 places
21:21:32 <TrueBrain> breadcrumbs, and navigation pane
21:21:35 <TrueBrain> not sure that is really useful
21:21:42 <TrueBrain> but okay, that is more a minor thing
21:21:47 <TrueBrain> listing what is in a folder sounds like a sane thing
21:21:56 <frosch123> no idea what breadcrumbs are
21:22:11 <TrueBrain> and we can either do that with a new namespace Folder, or with "auto" categories like /Category/Manual, or what-ever
21:22:40 <TrueBrain> basically that thing on the top left in the wiki you linked :)
21:23:04 <frosch123> yeah, i want that :)
21:23:13 <TrueBrain> but why, if we have a navigation pane that does that too?
21:23:42 <frosch123> currently it's not even possible to get to the main page :p
21:23:52 <TrueBrain> yes, it is .. hit the logo :)
21:24:02 <TrueBrain> (holds true for ALL our sites btw, that run on AWS)
21:24:18 <frosch123> you mean the text next to the logo?
21:24:36 <TrueBrain> but okay, I can add a breadcrumb so we can see if it works
21:24:53 <TrueBrain> what I personally would like, is to automate that navigation pane .. but I think we concluded that would be difficult :D
21:24:57 <frosch123> why would it look terrible?
21:25:14 <frosch123> it could even go into the tab line in the header
21:25:19 <TrueBrain> so you go from "3" to "1"
21:25:23 <TrueBrain> it is a terrible UI flow
21:25:36 <TrueBrain> but that happens with the way tabs are added in our wiki :)
21:25:45 <frosch123> Main Page | Manual | Tutorial | Trains | hfill | View Source | View History | Login
21:26:01 <TrueBrain> it is not about the location; it is about the UX :)
21:26:07 <TrueBrain> but we just add it, and debate it then :P
21:26:23 <TrueBrain> pretty sure what-ever we pick, some pages will need restructuring to become sane
21:26:40 <TrueBrain> I already concluded that TrueWiki will be an "opinionated" wiki server :)
21:26:49 <TrueBrain> "3. Trains" -> "1. Starting"
21:27:39 <frosch123> well, the tabs are also on the right, we should just delete them :)
21:28:00 <TrueBrain> what I always really like, is at the bottom of Tutorials: go to next step
21:28:04 <TrueBrain> (like we have here)
21:28:10 <TrueBrain> the tabs themself .. I could do without
21:28:36 <TrueBrain> but okay, breadcrumbs, sure, we add them, debate them
21:28:41 <TrueBrain> but how about listing
21:29:01 <TrueBrain> we can automate it, if we restructure the files even more, but the a-b-c was an issue
21:29:25 <TrueBrain> automate it, as in, generate the navigation panes we see now in yellow
21:29:45 <frosch123> maybe Category:en/Manual/Tutorial is not that bad
21:29:57 <frosch123> add all pages to the category with their folder name by default
21:30:22 <frosch123> then people can still edit the the category page if they want to add some text
21:30:48 <TrueBrain> and just assume nobody wants to make a custom category with that name :D
21:31:02 <frosch123> they should use a better name then
21:31:35 <TrueBrain> currently there are no categories in any folder, so that is good
21:31:43 <TrueBrain> so we can do [[Category:blabla]] for custom categories
21:31:55 <frosch123> they should have languages
21:32:17 <TrueBrain> and have the en/NNN/ for listing of folders
21:32:25 <TrueBrain> so you cannot add pages to there manually
21:32:35 <TrueBrain> not sure .. maybe we should just name them Folder/
21:32:41 <TrueBrain> and add them as if they were a category
21:32:57 <TrueBrain> so you link to there with [[:Folder/en/Manual/Tutorial]]
21:33:14 <TrueBrain> but okay, here too .. I will cook something up, and we can tune it after
21:33:22 <TrueBrain> well, not tonight, but tomorrow :P
21:34:32 <frosch123> "Pushing large files to forks of a repository count against the parent repository's bandwidth and storage quotas, rather than the quotas of the fork owner." <- is it just me, or does that make absolutely no sense?
21:34:50 <TrueBrain> I doesn't make any sense
21:34:53 <TrueBrain> it kinda freaked me out, honestly
21:35:59 <FLHerne> frosch123: Because in github's weird implementation, forks are actually part of the parent repo
21:36:46 <LordAro> and there's nothing they can do about it unless they go munging in the git repo directly
21:36:55 <LordAro> which they'll probably do, tbh
21:38:09 <TrueBrain> frosch123: anything else missing from the wiki?
21:38:34 <frosch123> features or content?
21:38:52 <TrueBrain> For edit I was planning to do the same as bananas-api: edits are stored in the git repo locally first, and push every N minutes to GitHub
21:39:00 <TrueBrain> otherwise, a big textarea, and a "preview" tab
21:39:04 <TrueBrain> bit of Javascript to make that smooth
21:39:17 <TrueBrain> new pages you make to go to that URL, you get a "file not found", and you can hit Create
21:39:23 <TrueBrain> new folder ... no clue yet
21:39:35 <frosch123> i want to check this weekend, which File or User pages to migrate, if any
21:39:36 <TrueBrain> new files .. some simple HTTP upload form I was thinking
21:40:21 <frosch123> TrueBrain: most important edit feature: prevent creating pages/uploads that differ in case only :)
21:40:51 <TrueBrain> which means keeping a lower-case array of all pages, but that is not an issue :)
21:41:04 <TrueBrain> memory is still < 64MB, so we have some room :P
21:41:47 <TrueBrain> the most difficult part for editing will be the invalidation of categories and translations, that it doesn't have to reload 90s every edit :P
21:41:53 <TrueBrain> I failed in that today .. lot of bookkeeping ...
21:42:00 <TrueBrain> I need to find a simpler system there
21:42:41 <frosch123> how long does it take to render all pages?
21:42:53 <TrueBrain> all pages fully, about 9 minutes
21:43:00 <TrueBrain> gathering only the metadata, 90 seconds
21:43:12 <frosch123> is that too long to do at container start?
21:43:29 <TrueBrain> I rather have it faster, but it will have to do for now :)
21:43:37 <TrueBrain> container startup has a deadline of 5 minutes, so there is some room
21:43:44 <frosch123> so some local sqlite?
21:44:13 <frosch123> fill at container start, update when revision changes, but only changed files
21:44:20 <TrueBrain> the biggest trouble I ran into, is invalidating the metadata after "git pull"
21:44:35 <TrueBrain> as what files changed :P
21:44:49 <TrueBrain> but I like the idea of using sqlite or something .. depends on the memory footprint tbh
21:44:53 <frosch123> store the revision you index last?
21:45:01 <TrueBrain> and just "git diff" it .. hmm
21:45:29 <TrueBrain> something to think about a bit more
21:45:49 <TrueBrain> I am also still wondering if we shouldn't just prerender all pages to HTML and upload to S3
21:45:55 <TrueBrain> I am still somewhat tempted to just go that route
21:46:12 <frosch123> ok, i think i missed some "/**" in my lfs setup, that's why it did not work
21:46:14 <TrueBrain> but okay, for editing the dynamic part needs to work anyway, so first build, later optimize :)
21:46:39 <frosch123> but given how the lfs quotas work, it's kind of a no-go, which is silly
21:46:47 <TrueBrain> I like how I butchered: "build first, optimize later"
21:47:09 <TrueBrain> if uploads are ever really an issue, we can always go the BaNaNaS way, and put the uploads only on S3
21:47:17 <TrueBrain> would be a bit of a shame, but .. it is a possibility
21:47:39 <TrueBrain> but given this is 15 years of wiki data ... I think we will be fine
21:49:15 <TrueBrain> frosch123: there is btw 1 user that has some templates on his space
21:49:22 <TrueBrain> not sure if they are important
21:49:31 <TrueBrain> just remember those, as they took a bit of time to render at first :D
21:50:27 <frosch123> yes, that user is a major annoyance :)
21:55:11 <frosch123> hmm, so my transitivity-usage checks still do not work properly
21:55:22 *** Progman has joined #openttd
21:55:30 <frosch123> none of those templates and categories should be present :)
22:02:09 <frosch123> those of that one user
22:02:22 <TrueBrain> ah .. I just noticed it when talking to the live API
22:02:31 <TrueBrain> haven't spotted it in your export yet :)
22:04:25 <TrueBrain> 42 {{SERVER}} and 15 {{fullurl: that will no longer work .. I think I am going to fix them on the current wiki already ... if I feel like it, that is
22:04:37 <TrueBrain> it is just silly that you have to know the server-name tbfh ..
22:04:39 <TrueBrain> such an old concept
22:06:09 <frosch123> lol, they use that to create links to "edit this page"?
22:06:25 <TrueBrain> yeah ... I just noticed that too
22:06:30 <TrueBrain> so I am just going to remove it completely :P
22:07:28 <TrueBrain> 16 places it is used useful, the {{SERVER}}
22:07:36 <TrueBrain> to view a page coming from a variable
22:08:56 <TrueBrain> hmm, maybe I can even remove the "localurl"
22:09:08 <TrueBrain> would be nice .. not having all these weird things people can break stuff with, is a bonus :)
22:12:26 <frosch123> i saw a lot of usages of PAGENAME, which are probably all broken :)
22:13:30 <frosch123> people used it to add [[File:Steel]] to page [[Steel]]. but that does no longer work for File:en/Steel and page en/Manual/Base Set/Cargos/Steel :)
22:14:45 <frosch123> hmm, didn't you mention a list of broken links some time ago?
22:15:01 <TrueBrain> I can generate a list, yes
22:15:07 <TrueBrain> Will do so tomorrow
22:22:33 *** WormnestAndroid has quit IRC
22:22:51 *** WormnestAndroid has joined #openttd
22:58:55 *** andythenorth has joined #openttd
continue to next day ⏵