IRC logs for #openttd on OFTC at 2023-06-15
02:46:57 *** D-HUND has joined #openttd
02:50:20 *** debdog has quit IRC (Ping timeout: 480 seconds)
03:29:01 *** keikoz has joined #openttd
04:33:15 <DorpsGek> [OpenTTD/OpenTTD] marcossouza1232 opened pull request #11012: fix #10995
04:51:19 <DorpsGek> [OpenTTD/OpenTTD] rubidium42 updated pull request #11007: Codechange: make creating temporary StringParameters easier
05:43:35 *** merni has joined #openttd
05:43:35 <merni> In the , the changelog only shows the changes in 13.2.1 and not those in 13.2
06:33:33 *** Flygon has quit IRC (Read error: Connection reset by peer)
06:47:49 <DorpsGek> [OpenTTD/OpenTTD] 2TallTyler commented on pull request #11012: fix #10995
07:16:04 *** HerzogDeXtEr has joined #openttd
07:44:43 *** audunm has joined #openttd
07:44:43 <audunm> Sorry in advance. Testing if I can talk to myself from discord to matrix via IRC.
07:45:02 <audunm[m]> I can!
07:54:35 *** _aD has joined #openttd
08:03:42 <FLHerne> that's neat
08:04:24 <FLHerne> Is there a global OFTC irc-matrix bridge then?
08:11:58 <Rubidium_> no you can't talk to yourself; the discord bridge doesn't support private messages
08:50:26 <DorpsGek> [OpenTTD/OpenTTD] 2TallTyler approved pull request #10687: Codechange: Simplify drawing of timetables
09:07:58 *** _aD is now known as Guest3155
09:08:02 *** _aD has joined #openttd
09:09:58 *** Guest3155 has quit IRC (Ping timeout: 480 seconds)
09:12:49 <TrueBrain> okay; I got the hang of how to scale up and down with this new cluster, and fixed the stupid cgroup bug for now .. after lunch, let's deploy this lovely bananas to production 🙂
09:16:34 *** _aD is now known as Guest3156
09:16:39 *** _aD has joined #openttd
09:17:19 *** Guest3156 has quit IRC (Read error: Connection reset by peer)
09:18:58 <TrueBrain> wow, the wiki is using 337 MB of RAM .. I had it limited to 384, but that is cutting it rather close 😄
09:20:03 <TrueBrain> most of that memory is the metadata, like all categories, files, links, templates etc that exist ..
09:20:19 <TrueBrain> seems it has grown a bit over the years 🙂
09:21:30 <LordAro> TrueBrain: TrueWiki is inefficient!
09:21:39 <TrueBrain> yes, let's go with that 😛
09:22:40 <pickpacket> TrueBrain: isn't that a lot of RAM though? I would've assumed that stuff like that was in a DB
09:22:50 <pickpacket> or cached
09:22:58 <TrueBrain> and a DB would be magic and won't consume RAM?
09:23:01 <TrueBrain> or a cache won't?
09:23:19 <pickpacket> cache as in static files
09:23:31 <TrueBrain> and how do you cache a tree of information in a static file? 😄
09:23:32 <pickpacket> and a DB is usually pretty optimised
09:23:36 <pickpacket> true
09:23:45 <TrueBrain> I dare you to find a DB that would hold this information for less memory 🙂
09:24:05 <pickpacket> Challenge not accepted! 💪
09:24:19 <pickpacket> 😛
09:24:41 <pickpacket> how big is the wiki in total? I think you mentioned the other day but my IRC search is bad
09:25:17 <TrueBrain> ~400MB of plain text
09:25:24 <TrueBrain> before rendering etc
09:32:23 <petern>
09:32:27 <petern> I think that might be lying...
09:32:43 <TrueBrain> at least you know when it is getting up already
09:39:11 <DorpsGek> [OpenTTD/OpenTTD] rubidium42 commented on pull request #11001: Feature: Transparency option for cost and income indicators
09:45:19 <TrueBrain> ah, okay, TrueWiki only actually uses ~150MB RAM, the rest is Python stuff; after rendering a few big pages, it doesn't free the memory directly
09:45:42 <TrueBrain> (rendered pages are cached on disk, and served from disk from there on; but it ofc takes Python a bit of time to go like: owh, I can free this memory now :P)
09:46:09 <TrueBrain> the `git clone` does consume a bit of memory on startup 😄 I should add a fetch-depth, I guess ...
10:42:13 *** _aD has quit IRC (Ping timeout: 480 seconds)
10:44:19 <DorpsGek> [OpenTTD/OpenTTD] orudge updated pull request #11010: Change: [Actions] Use notarytool for notarization instead of gon
10:48:25 <DorpsGek> [OpenTTD/OpenTTD] TrueBrain approved pull request #11010: Change: [Actions] Use notarytool for notarization instead of gon
11:58:36 <TrueBrain> right .. time to flip the switch .. let's think a bit how to do that with the least amount of downtime .. hmm
12:02:43 *** _aD has joined #openttd
12:07:19 <pickpacket> TrueBrain: what could possibly go wrong? Just do it!
12:07:27 <TrueBrain> well, it serves an empty page now
12:07:30 <TrueBrain> clearly that can go wrong 😛
12:07:34 <pickpacket> :D
12:09:06 <petern> I'm broken. nullptr does not work in C#...
12:09:07 <Rubidium_> just pause the RL-hypervisor temporarily :D
12:09:32 <TrueBrain> turns out you cannot change a hostname with Pulumi / Terraform in Cloudflare
12:09:33 <TrueBrain> funny bug
12:11:04 <Rubidium_> petern: what about something like `public static class CXXDev { public const object nullptr = null; }` and in the other files `using static CXXDev;`?
12:11:21 <petern> Yes definitely doing that...
12:13:07 <petern> Hmm, GRF scanning is funny, it goes through the effort to build a sorted linked-list on the fly, and then once it's done that, it copies all the item pointers into a vector, sorts that, and then relinks the linked-list from it.
12:17:40 <TrueBrain> okay, I think everything is still working as it should
12:22:04 <DorpsGek> [OpenTTD/BaNaNaS] TrueBrain opened pull request #137: Fix: [CI] migrate to new backend for reload triggers
12:23:32 <TrueBrain> 13.3 works, master works, I can login, I can upload files .. I think the flipping of the switch went okay 🙂
12:23:43 <DorpsGek> [OpenTTD/BaNaNaS] TrueBrain merged pull request #137: Fix: [CI] migrate to new backend for reload triggers
12:23:54 <TrueBrain> the one thing I cannot test, if uploading new content on the production version actually works correctly .. on preview it does at least 🙂
12:26:35 <DorpsGek> [OpenTTD/BaNaNaS] TrueBrain opened pull request #138: Fix: [CI] use the correct service name for the reload triggers
12:26:46 <DorpsGek> [OpenTTD/BaNaNaS] TrueBrain merged pull request #138: Fix: [CI] use the correct service name for the reload triggers
12:27:25 <glx[d]> Again ?
12:27:30 <TrueBrain> yes
12:27:31 <TrueBrain> and 3 more times to go 😛
12:27:37 <TrueBrain> I am incapable of doing this correct, it turns out
12:27:42 <TrueBrain> but I am happy you point it out 😛 😄 😄
12:28:35 <TrueBrain> hmmm ... one of the secrets on GitHub is wrong .. that is odd
12:33:20 *** _aD has quit IRC (Ping timeout: 480 seconds)
12:35:40 <DorpsGek> [OpenTTD/BaNaNaS] TrueBrain opened pull request #139: Fix: [CI] Use the API token for the API reload
12:35:41 <TrueBrain> it is not my day ...
12:35:53 <DorpsGek> [OpenTTD/BaNaNaS] TrueBrain merged pull request #139: Fix: [CI] Use the API token for the API reload
12:35:59 <pickpacket> TrueBrain: I believe in you ❤️
12:36:45 <TrueBrain> right, now it finally works 🙂
12:37:46 <pickpacket> yay!
12:40:16 <DorpsGek> [OpenTTD/bananas-api] TrueBrain opened pull request #390: Fix: [CI] use the correct service name for a release deployment
12:40:19 <DorpsGek> [OpenTTD/bananas-server] TrueBrain opened pull request #343: Fix: [CI] use the correct service name for a release deployment
12:40:22 <DorpsGek> [OpenTTD/bananas-frontend-web] TrueBrain opened pull request #215: Fix: [CI] use the correct service name for a release deployment
12:40:23 <TrueBrain> at least it was a consistent mistake 😄
12:45:17 <TrueBrain> `ReadableStream.tee() buffer limit exceeded` .. oops, can't serve big files, it seems ..
12:48:41 <petern> Just remove [az]Base 🙂
12:48:50 <TrueBrain> it is yeti that triggered the issue 😛
12:49:57 <petern> Okay, std::forward_list<T> is pretty unusable. I'm still weighing up std::list<T> or std::vector<T *>
12:53:29 <TrueBrain> ah, YETI is 182MB
12:56:00 <petern> Host everything on Steam workshop 😉
12:56:07 <TrueBrain> ghehe
12:56:40 <TrueBrain> it is a bit annoying .. every documentation mentions you should fetch the result, clone it, put one clone into cache, return the other to theu ser
12:56:50 <TrueBrain> but ... the clone copies the whole body, instead of streaming it
12:57:04 <TrueBrain> and the workers are limited to 128MB RAM .. and 182 > 128, in case you didn't know 🙂
12:57:37 <petern> They probably mean large files like 10KB
12:57:54 <TrueBrain> you can store up to 512MB, so it should kinda work honestly
13:06:14 <Rubidium_> petern: the current GRF "list" is essentially a std::forward_list, isn't it?
13:17:56 <petern> Conceptually, however the interface for wrangling with it is different, and different enough from std::list/std::vector to require learning a new set of skills :p
13:25:00 <petern> emplace() doesn't exist, emplace_after() will work but doesn't return the item you've just emplaced, so unless everything can be done in the constructor, it seems to be painful to get back the item you've just added.
13:29:15 <TrueBrain> there, fixed the issue with streaming large files .. as extra bonus, downloads start sooner now 🙂
13:31:14 <Rubidium_> that indeed makes it more annoying to use
13:33:54 <Rubidium_> and then having to keep the iterator to the end, or keep looking for the last element... std::list seems indeed a lot simpler
13:39:03 <Rubidium_> though I'd just go with std::vector
13:39:20 <petern> Yes, that's what I'm leaning to.
13:39:50 <petern> Especially as there's a place in networking where a grflist is either loaded from network packets, or it's the actual live _grfconfig.
13:40:44 <petern> Using either a std::forward_list or std::list would mean having to make a copy of _grfconfig, or use (and worse, manage) pointers to lists...
13:44:49 <TrueBrain> right, seems BaNaNaS is about 5GB an hour atm .. so about 4TB a month, seems about right .. let's see how this works out 🙂
13:47:00 <petern> I remember when I used to host a mirror (back when we did that) but had to remove it as the traffic swamped $employers' real traffic.
13:47:13 <TrueBrain> I remember that too 😄
13:47:17 <TrueBrain> you were not alone in that issue ..
13:47:49 <TrueBrain> several offers of mirrors where I told people the expected bandwidth, they said: no way, lol, you are not doing that bandwidth, demenaded to be connected .. within a month were like: yeah, I can't afford this
13:47:50 <petern> Later I had an OVH host that could've mirrored but it was basically when we were already using OVH so wouldn't've added much.
13:50:24 <TrueBrain> lol, CDN cache hit/miss ratio so far: 1350 misses, 36 hits
13:50:39 <petern> Just warming up?
13:50:43 <TrueBrain> 😄 It might take a while for the cache to become useful .. if things aren't purged every time before it is rerequested 🙂
13:50:47 <TrueBrain> I really hope so 🙂
13:51:01 <TrueBrain> `public, max-age=31536000, immutable` hopefully that hints enough it can be cached, really, it can
13:52:06 <DorpsGek> [OpenTTD/OpenTTD] orudge merged pull request #11010: Change: [Actions] Use notarytool for notarization instead of gon
13:52:13 <DorpsGek> [OpenTTD/bananas-frontend-web] TrueBrain merged pull request #215: Fix: [CI] use the correct service name for a release deployment
13:52:25 <TrueBrain> let's see if releases finally work 🙂
13:57:12 <TrueBrain> it is funny .. we replaced BaNaNaS, what, 3 years ago? 4? And people still try the old URL from time to time 🙂
13:57:30 <TrueBrain> like: don't give it! YOUR DAY MAY COME!
13:58:03 <DorpsGek> [OpenTTD/bananas-server] TrueBrain merged pull request #343: Fix: [CI] use the correct service name for a release deployment
14:01:42 <TrueBrain> `/index.php/photo-gallery-a-movie/a-few-from-summer-2011/summer5-290` .. do I want to know why someone requests that URL from our servers?
14:01:44 <TrueBrain> feels weird 😛
14:06:20 <DorpsGek> [OpenTTD/bananas-api] TrueBrain merged pull request #390: Fix: [CI] use the correct service name for a release deployment
14:09:12 <TrueBrain> okay ... so tomorrow ... we port a few more services over, I guess 🙂 Should go a lot quicker now, given all infra should be there now 😄
14:09:50 <TrueBrain> the release CDN will be interesting to port over .. there is a lot of data in there
14:10:47 <TrueBrain> sadly, AWS S3 doesn't tell you at all .. Cloudflare R2 at least does tell you those basic things which they use to charge you with
14:11:02 <TrueBrain> AWS is really stupid in those regards .. like .. "you will find out when the bill lands"
14:11:38 <TrueBrain> but, on estimate .. 150GB of data on that bucket ... 100k files .. every nightly we ever produced is in there 😛
14:24:12 *** _aD has joined #openttd
14:32:35 *** Kitrana has joined #openttd
14:35:29 *** nielsm has joined #openttd
14:36:29 *** Kitrana1 has quit IRC (Ping timeout: 480 seconds)
14:42:09 *** Kitrana has quit IRC (Ping timeout: 480 seconds)
14:43:22 *** felix_ has joined #openttd
14:45:34 *** felix has quit IRC (Ping timeout: 480 seconds)
14:49:37 *** Kitrana has joined #openttd
14:52:06 *** Kitrana1 has joined #openttd
14:53:09 <DorpsGek> [OpenTTD/OpenTTD] zephyris updated pull request #11001: Feature: Transparency option for cost and income indicators
14:54:05 <DorpsGek> [OpenTTD/OpenTTD] zephyris commented on pull request #11001: Feature: Transparency option for cost and income indicators
14:57:39 *** Kitrana has quit IRC (Ping timeout: 480 seconds)
15:00:13 *** gelignite has joined #openttd
15:13:34 <DorpsGek> [OpenTTD/OpenTTD] rubidium42 opened pull request #11013: Codechange: move StringParameters to strings_internal.h
15:25:32 *** Eddi|zuHause has quit IRC (Read error: Connection reset by peer)
15:26:14 *** Eddi|zuHause has joined #openttd
15:38:46 <petern> > Segmentation fault (core dumped)
15:38:46 <petern> > error MSB3073: The command "webcompiler wwwroot/scss -r -c webcompilerconfiguration.json" exited with code 139.
15:38:50 <petern> Well, that's not what I want 😦
15:40:21 <TallTyler> That does sound less than ideal
15:43:19 <petern> Hmm, worked second time with no changes. That's also not good, but at least it is going through.
15:58:12 <glx[d]> hmm eints is still on old workflows, so a release is needed, but maybe I should wait
15:59:04 <DorpsGek> [OpenTTD/OpenTTD] rubidium42 approved pull request #11001: Feature: Transparency option for cost and income indicators
15:59:50 <DorpsGek> [OpenTTD/OpenTTD] LordAro commented on pull request #11001: Feature: Transparency option for cost and income indicators
16:00:08 <DorpsGek> [OpenTTD/OpenTTD] rubidium42 approved pull request #10890: Feature: Create group of vehicles from manage vehicle list button.
16:34:14 <DorpsGek> [OpenTTD/website] merni-ns opened issue #265: Wrong changelog for 13.2
16:39:53 <DorpsGek> [OpenTTD/website] TrueBrain opened pull request #266: Fix: mention the changelog for both 13.2 and 13.2.1
16:39:58 <TrueBrain> So people actually use those links; surprising 😛
16:40:27 <DorpsGek> [OpenTTD/website] LordAro commented on pull request #266: Fix: mention the changelog for both 13.2 and 13.2.1
16:41:02 <TrueBrain> meh; do we actually care enough to erase it from existence? Fine ....
16:41:53 <DorpsGek> [OpenTTD/website] TrueBrain updated pull request #266: Fix: mention the changelog for both 13.2 and 13.2.1
16:49:39 <TrueBrain> pfff, he can make a comment within a minute, but can't even approve in 8? I am dissapointed LordAro 😛 😛 😛
16:54:16 <TrueBrain>
16:54:16 <TrueBrain> slowly growing, what is cached and what is not 🙂
17:11:48 <LordAro> TrueBrain: i went home :p
17:12:01 <TrueBrain> shitty excuses 😛
17:12:04 <DorpsGek> [OpenTTD/website] LordAro approved pull request #266: Fix: remove any mention of 13.2.1
17:12:07 <TrueBrain> 😄 😄
17:12:08 <DorpsGek> [OpenTTD/website] TrueBrain merged pull request #266: Fix: remove any mention of 13.2.1
17:12:11 <TrueBrain> Tnx 🙂
17:12:14 <DorpsGek> [OpenTTD/website] TrueBrain closed issue #265: Wrong changelog for 13.2
17:12:30 <TrueBrain> haha, Hashicorp (Nomad) is going to send me a t-shirt for fixing that bug I found yesterday 🙂
17:12:34 <LordAro> :D
17:13:08 <TrueBrain> I like how they handle their Open Source as a business; it is nice 🙂
17:13:49 <LordAro> i did a bit of reading up of nomad today
17:15:21 <TrueBrain> it is a more understandable way of kubernetes; and a less shitty way of AWS ECS 😛
17:15:33 <LordAro> definitely interesting
17:15:51 <LordAro> but as previously, i just don't have the need for a) cloud-based things and b) such high availability
17:16:12 <TrueBrain> that is why I use OpenTTD as my garden 😄
17:16:30 <TrueBrain> otherwise I wouldn't have a need for it either; but it is a good way to understand how these technologies are evolving
17:16:58 <TrueBrain> I (strongly) advise companies to not switch to kubernetes, unless they want to invest 2 FTE; which mostly scares them away
17:17:03 <TrueBrain> but the next question they ask: but what else?
17:17:26 <TrueBrain> Azure Containers are pretty decent .. AWS Fargate is .. and now Nomad is pretty high on that list 🙂
17:18:42 <LordAro> i do get the feeling of 'being left behind' from time to time
17:19:08 <DorpsGek> [OpenTTD/OpenTTD] rubidium42 opened pull request #11014: Codechange: remove a number of unneeded c_str() calls
17:19:11 <DorpsGek> [OpenTTD/OpenTTD] rubidium42 opened pull request #11015: Codechange: make SQString::Create that supports std::string and use that
17:19:15 <TrueBrain> that is a pretty normal feeling to have, and one you should have; IT is going so quickly .. if you stand still, you become a specialist in a certain field, and that is it 🙂
17:19:34 <LordAro> all of the containers i have are just "mostly automated"
17:19:40 <TrueBrain> I worked with too many people and teams the last few years where they thought their ideas of 5 years ago were modern ... they weren't
17:21:11 <DorpsGek> [OpenTTD/OpenTTD] TrueBrain approved pull request #11014: Codechange: remove a number of unneeded c_str() calls
17:21:39 <DorpsGek> [OpenTTD/OpenTTD] LordAro commented on pull request #11015: Codechange: make SQString::Create that supports std::string and use that
17:21:57 <LordAro> i'm pretty sure lxd was modern when we started using it 6 years ago
17:22:07 <LordAro> it had bugs in it and everything
17:22:18 <TrueBrain> and by now it is old news 😄
17:22:35 <TrueBrain> runc is the new kid on the block
17:22:47 <TrueBrain> (it isn't new btw; which is the fun part)
17:23:05 <TrueBrain> first release seems to be from 2015 .. lol
17:24:29 <LordAro> interesting
17:24:54 <TrueBrain> what I also find interesting, and what I said 3 years ago already, that Docker is now finally also actually going away
17:24:59 <TrueBrain> we start to call it OCI containers
17:25:02 <LordAro> mm
17:25:13 <TrueBrain> and Docker is mentioned less and less in reference to containerization
17:26:05 <DorpsGek> [OpenTTD/OpenTTD] rubidium42 updated pull request #11015: Codechange: make SQString::Create that supports std::string and use that
17:26:06 <LordAro> i've never been quite sure we've been doing lxd right anyway, tend to just treat them as lightweight VMs, all stateful and just upgrading them in place - some of them still say "image: ubuntu:16.04" or similar in the config
17:26:24 <TrueBrain> oof
17:26:33 <TrueBrain> I know of little companies that use lxd, I have to admit
17:28:38 <TrueBrain> one of the reasons why I kinda like the old and new OpenTTD infra, is because everything, except for a few things like the DorpsGek chat logs, is not actually stored in the cluster
17:28:44 <TrueBrain> that is to say, if the cluster burns down
17:28:46 <TrueBrain> you spin up a new one
17:28:50 <TrueBrain> and you still have everything as you had before
17:28:55 <TrueBrain> nothing is lost .. like .. nothing
17:29:06 <TrueBrain> I really like that kind of container idea
17:29:25 <TrueBrain> but I still see way too many companies build stateful and persistent payloads in their clusters
17:30:14 <LordAro> so all the actual data is stored on some "thing" somewhere else, and all the containers/nodes/whatever just point to the right part of it?
17:30:27 <TrueBrain> yup
17:30:34 <TrueBrain> like BaNaNaS is on GitHub and a S3-compatible bucket
17:30:35 <LordAro> the equivalent in my head is a centralised database server, rather than each container running their own
17:30:38 <LordAro> (which we also do..)
17:30:41 <TrueBrain> so the actual instance has no actual value in terms of content
17:30:59 <TrueBrain> so the containers can crash all night long, but it doesn't matter, like, at all
17:31:09 <LordAro> aye
17:31:11 <DorpsGek> [OpenTTD/OpenTTD] rubidium42 commented on pull request #11015: Codechange: make SQString::Create that supports std::string and use that
17:31:37 <TrueBrain> what I like most of it, is that few things are emergencies .. does a container crash when a user visits a certain URL? I will fix it tomorrow
17:31:51 <TrueBrain> (the wiki had that, for months .. once every week or so someone visited a certain link, crashing the wiki)
17:32:03 <LordAro> the issue i don't really know the solution to is how to achieve that sort of storage solution (without "magical clouds" doing it for you)
17:32:14 <LordAro> for us at work, we're just not big enough to justify something like ceph
17:32:27 <TrueBrain> OpenTTD also isn't big .. but it is being a bit creative 🙂
17:32:52 <LordAro> it's "bigger" in terms of services :)
17:32:53 <TrueBrain> having an NFS shared mount on all hosts could already be a step, for example
17:33:07 <TrueBrain> or having a redis cluster
17:33:12 <LordAro> i've had so many bad experiences with NFS mounts :p
17:33:20 <TrueBrain> NFSv4 is pretty good
17:33:37 <TrueBrain> AWS EFS is basically NFS, and the amount of issues .. I can still count them on my hands
17:33:46 <JGR> With NFS it's very easy to end up in a huge muddle with uid/gid mapping issues
17:34:02 <LordAro> that too
17:34:07 <TrueBrain> yeah ... you need to know what you are doing, that is for sure 🙂
17:34:25 <TrueBrain> but yeah, if you are talking about ceph, you are talking about self-hosted, and that is just ... shitty in 2023
17:34:34 <LordAro> trying to route AD auth through it all has also proven difficult
17:34:41 <LordAro> TrueBrain: aerospace, got to be on-site
17:34:46 <TrueBrain> when I was security consultant I did one too many talks about explaining to companies that self-hosting is terrible for security
17:35:19 <LordAro> are they available anywhere? :D
17:35:36 <TrueBrain> haha, I hope not 😄
17:35:59 <LordAro> got to run, thanks for some interesting ideas
17:36:05 <LordAro> i don't have any time to do anything with any of them, but even so
17:36:07 <TrueBrain> haha, enjoy running 😛
17:36:32 <TrueBrain> (and he just got home!)
17:37:49 <LordAro> cinema time!
17:37:56 <TrueBrain> w00p!
17:39:19 <TrueBrain> lol, reading random tt-forums topic (yes, I know), about issues connecting to the coordinator .. I see we should make our logs a tiny bit better
17:39:29 <TrueBrain> when you only have IPv4
17:39:35 <TrueBrain> it tells you connections over IPv6 fail
17:39:38 <TrueBrain> (to be expected)
17:39:44 <TrueBrain> but it doesn't mentions it was also trying IPv4 🙂
17:40:02 <TrueBrain> (the IPv6 fails with an error, and in this case the IPv4 timed out)
17:40:22 <TrueBrain> so the IPv6 error is printed verbose, and the timeout is handled more graceful .. hihi
17:40:29 <TrueBrain> making logs make sense in most cases is hard!
17:43:00 <DorpsGek> [OpenTTD/bananas-api] TrueBrain updated pull request #373: Fix: bring the image size limit of PIL in line
17:44:00 <TrueBrain> finally I can now test that PR myself, without setting up a local setup 😄
17:47:06 <TrueBrain> lol, I need to assign more memory to this preview instance .. can't even attempt a 10kx10k map 😛
17:47:22 <TrueBrain> something for another day 🙂
18:11:56 *** Flygon has joined #openttd
18:35:12 <TrueBrain> wauw, analyzing a 10kx10k heightmap is ... taking a bit of time. Why did we want to allow such big maps again?
18:35:51 <Rubidium_> because maybe JGRPP supports them?
18:35:59 <Rubidium_> as in, maps of that size
18:36:25 <TrueBrain> but it is just completely unrealistic to process such large heightmaps on the backend
18:37:08 <discord_user_03329cf> Ngl i wanna try 16k x 16k
18:37:23 <discord_user_03329cf> Or at least 8k x 8k
18:37:49 <discord_user_03329cf> 4x4 feels too small to have continents
18:38:43 <TrueBrain> 16kx16k consumes ~1.2GB of RAM on the backend .. and takes over 30s to process .. lol. I wonder if the image actually needs to be completely in memory .. PIL does that, but does it have to? Hmm ..
18:39:08 <JGR> 16k x 16k is not really usefully playable
18:39:31 <discord_user_03329cf> Maybe 8x8 then
18:40:01 <discord_user_03329cf> Made up of stitched together klutz maps
18:40:48 <DorpsGek> [OpenTTD/bananas-api] TrueBrain dismissed a review for pull request #373: Fix: bring the image size limit of PIL in line
18:41:54 <DorpsGek> [OpenTTD/OpenTTD] DorpsGek pushed 1 commits to master
18:41:55 <DorpsGek> - Update: Translations from eints (by translators)
18:42:45 <DorpsGek> [OpenTTD/bananas-api] TrueBrain commented on pull request #373: Fix: bring the image size limit of PIL in line
18:42:59 <TrueBrain> even requesting the image size via PIL triggers the OOM
18:42:59 <TrueBrain> lol
18:44:50 <DorpsGek> [OpenTTD/OpenTTD] rubidium42 updated pull request #10733: Codechange: use standard int types
18:45:50 <TrueBrain> so `open` is suppose to be lazy .. this is weird 🙂
18:46:15 <TrueBrain> owh, md5sum is calculated with an ``
18:46:20 <TrueBrain> yeah .. that is already not smart 🙂
18:46:39 <TrueBrain> should be chunk-based ofc
18:47:11 <TrueBrain> but a 8kx8k image is just 1MB, so that shouldn't actually be a problem .. well, this will be interesting
18:52:11 <DorpsGek> [OpenTTD/bananas-api] TrueBrain commented on pull request #373: Fix: bring the image size limit of PIL in line
18:55:06 <Rubidium_> TrueBrain: how hard would it be to determine the highest difference in orthoganally adjacent pixels in your classifier script? I wonder whether the big heightmaps still have quite high differences between pixels, so they can't be shown correctly. Going even more pixels will only make that problem worse, so I'm not sure that >4kx4k is really necessary
18:55:29 <Rubidium_> since the majority of the information is going to be thrown away anyway to fit within the limits of our maps
18:55:54 <TrueBrain> I really do not know
18:56:12 <TrueBrain> I don't like backends being the limitation of systems; that I do know 🙂
18:57:28 <TrueBrain> we have some 8kx8k uploads on BaNaNaS currently
18:58:17 <TrueBrain> the only thing I can imagine, that people would like a heightmap to have 1-on-1 resolution with the biggest map you can create with JGRPP
18:59:11 <TrueBrain> (which brings us to 256M tiles)
19:00:36 <petern> Urgh, compilation errors that don't show you the line number...
19:04:51 <TrueBrain> meh; Pillow really just loads the image in memory in full
19:04:57 <TrueBrain> such a waste .. I just want to see the bytes once ..
19:07:00 <Ahyangyi> Yeah, if 16k maps cause OOM, I'll give up and submit 8k heightmaps instead
19:07:16 <TrueBrain> 8k even cause OOM 😛 I just need to look into this 🙂
19:07:22 <Ahyangyi> Weird
19:07:27 <TrueBrain> not really
19:07:35 <TrueBrain> we never really used to read the content of heightmaps; we added that recently
19:07:47 <TrueBrain> and .. well .. not many people want to upload very large maps 😄
19:08:27 <discord_user_03329cf> Oom?
19:08:40 <Ahyangyi> discord_user_03329cf: shorthand for "out of memory"
19:09:08 <discord_user_03329cf> Aaa
19:13:51 <TrueBrain> Ahyangyi: I think a library like `pyvips` instead of Pillow might really help
19:13:57 <TrueBrain> it explicitly doesn't keep the whole image in memory
19:15:52 <TrueBrain> suggests a difference of 9x memory usage 😄
19:17:50 <glx[d]> nml could maybe use that too
19:19:46 <TrueBrain> it doesn't have the best documentation, but that happens a lot with these kind of libraries 🙂
19:20:28 <andythenorth> libvips eh
19:20:51 <andythenorth> I did check recently, Pillow isn't as slow as I expected for a lot of common tasks
19:20:56 <andythenorth> I assumed 'eh python is slow'
19:21:05 *** Wolf01 has joined #openttd
19:21:06 <andythenorth> but some benchmarks suggested Pillow isn't
19:21:21 <TrueBrain> speed I did not look into .. memory footprint however is insane 🙂
19:21:27 <TrueBrain> I can't find any way to stream an image
19:21:33 <TrueBrain> although most image backends seem to support it just fine
19:21:42 <TrueBrain> but at some point, all pixels are loaded in a single buffer, it seems
19:22:18 <andythenorth> Horse image generation is now nearly 5s
19:22:23 <andythenorth> and that's using an MP pool
19:23:16 <andythenorth> 24s single threaded
19:23:38 <andythenorth> 2k LOC in my image processor though 😛
19:23:42 <andythenorth> not sure I want to convert
19:25:08 <Ahyangyi> libvips does come with a historgram function
19:25:25 <Ahyangyi> which seems to be the main thing we do with the heightmap in bananas-api?
19:25:31 <TrueBrain> Pillow has one too .. it was rubbish 😛
19:25:40 <TrueBrain> main thing is that we convert RGB in a certain way
19:26:09 <TrueBrain> which is one of the standard ways to convert RGB to a single colour btw
19:26:15 <TrueBrain> so "it depends" 😄
19:26:19 <DorpsGek> [OpenTTD/OpenTTD] JGRennison opened issue #11016: [Bug]: Use after free in network invalid packet error path
19:26:29 <Ahyangyi> I see, that's the hard part (when memory is a concern)
19:26:50 <TrueBrain> JGR: lol .. we fixed more than one bug that looks very similar to what you describe .. the gift that keeps on giving 😄
19:27:03 <TrueBrain> Ahyangyi: well, we just need a stream of pixel-data, honestly
19:27:20 <TrueBrain> it can even be in any order, as long as we have seen all pixels in the end 😛
19:28:08 <andythenorth> wonder if anyone could port my Pillow code 😛
19:28:17 <andythenorth> it's probably just method and parameter replacement
19:28:38 <andythenorth> at first glance, libvips appears to have the majority of methods I'm using
19:28:39 <TrueBrain> pyvips' documentation is not really telling me how to use it ... lol
19:28:59 <andythenorth> it's more obvious if you've done 10 years of PIL 😛
19:29:14 <andythenorth> it probably has an image stream of raw pixel data
19:29:21 <andythenorth> and an Image wrapper around that
19:29:28 <andythenorth> with save / load etc
19:29:35 <andythenorth> it has crop, insert, transpose, rotate
19:29:38 <andythenorth> colour map
19:29:47 <andythenorth> it has a point reader
19:29:54 <andythenorth> it has draw tools, but they come with a performance warning
19:30:14 <TrueBrain> okay, the SourceCustom we most likely need to wrap around our `fp` .. so that at least is possible .. I found out by an example of theirs, ironically 🙂
19:35:36 <TrueBrain> guess one could always just save the image as BMP and use the TargetCustom to capture data .. but I am sure there is some other function to call .. haven't found it yet 😄
19:36:05 <TrueBrain> anyway, I was going to watch a movie .. Ahyangyi if you feel up for looking into this, would be appreciated. Otherwise I will toy with this a bit soon .. it has to be possible to not use an absurd amount of memory to just read through an image once 🙂
19:36:17 <TrueBrain> (and yes, the backend is memory-limited .. that is just what it is 😄 )
19:38:11 <TrueBrain> I also wonder if we actually check the upload is in a format we support in OpenTTD .. like not that someone uploads a TIFF or PDF 😛
19:40:55 <TrueBrain> seems you need to use `crop` to get smaller sections of an image to avoid memory usage
19:45:23 <glx[d]> multiple load+crop ?
19:47:16 <TrueBrain> Seems crop returns a new object.. dunno, needs testing 🙂
19:51:00 <TrueBrain> Owh, sink screen seems promising too, but doesn't seem to have a py wrapper ..
20:02:24 <petern> Hmm
20:02:25 *** gelignite has quit IRC (Quit: Stay safe!)
20:03:59 <petern> Weird, 1) ZeroedMemoryAllocator doesn't seem to have any effect with std::make_shared, and 2) the std::string constructor of GRFConfig doesn't even invoke ZeroedMemoryAllocator() anyway, so how does that work in master...
20:05:35 <petern> Although I'm happy to not use ZeroedMemoryAllocator anyway, as it's bad form when structs contain stl things.
20:07:04 <Rubidium_> yeah, eventually ZeroedMemoryAllocator should go away
20:08:20 <JGR> The point of std::make_shared is to avoid calling the allocator for the new object
20:08:31 <Rubidium_> and ZeroedMemoryAllocator allocates (new) zeroed memory. It doesn't have a constructor zero-ing stuff
20:08:37 <JGR> So that the control block and the data can go in the same allocation
20:09:17 <petern> Ah ha, I see.
20:10:06 <petern> So the fact that the copy constructor DOES include a call to ZeroedMemoryAllocator() is a red herring, and either isn't necessary, or is there to shut up a compiler warning.
20:10:38 <petern> I wasn't sure if make_shared called new behind the scenes, or if that's all implementation specific...
20:10:50 <JGR> It calls placement new
20:11:36 <petern> Fun.
20:11:46 <Rubidium_> petern: yes, that seems to be a red herring
20:12:33 <TrueBrain> Some more googling .. vips has regions which does exactly what we want / need .. it reads a region of pixels .. so seems vips can help reduce memory 🙂
20:14:49 <DorpsGek> [OpenTTD/OpenTTD] rubidium42 merged pull request #11014: Codechange: remove a number of unneeded c_str() calls
20:55:49 *** HerzogDeXtEr has quit IRC (Read error: Connection reset by peer)
20:56:41 *** Wolf01 has quit IRC (Quit: Once again the world is quick to bury me.)
20:56:44 <LordAro> cinema was broken, so we went bowling instead and went to the arcade
20:56:58 <LordAro> i won enough tickets for 7 mini bags of haribo
20:58:05 <TrueBrain> End the night with a sugar high? 😄
21:00:11 <petern> Desserts are like that :/
21:03:12 <petern> Nice, removing ZeroedMemoryAllocator from certain places, and that means alloc_func.hpp is no longer needed...
21:03:17 <petern> At least, no longer directly needed.
21:03:55 <petern> Include hell :/
21:06:12 <petern> alloc_type.hpp rather.
21:22:07 *** keikoz has quit IRC (Ping timeout: 480 seconds)
22:15:35 *** _aD has quit IRC (Quit: leaving)
22:52:49 <DorpsGek> [OpenTTD/OpenTTD] zephyris opened pull request #11017: Fix #10838
23:12:52 *** nielsm has quit IRC (Ping timeout: 480 seconds)
23:17:39 *** Eddi|zuHause has quit IRC (Remote host closed the connection)
23:18:31 *** Eddi|zuHause has joined #openttd
23:23:04 *** Eddi|zuHause has quit IRC (Remote host closed the connection)
23:23:46 *** Eddi|zuHause has joined #openttd
23:33:00 <petern> That's a terrible PR title :p
23:41:02 *** D-HUND is now known as debdog
23:42:13 <petern> How did 00:42 happen... oops.