IRC logs for #opendune on OFTC at 2012-12-21
⏴ go to previous day
02:33:58 *** Sonarpulse has joined #openDune
02:37:28 <Sonarpulse> or any other team member?
02:43:11 *** Sonarpulse has left #openDune
04:26:23 *** planetmaker has joined #openDune
04:26:23 *** ChanServ sets mode: +v planetmaker
06:23:07 *** Sonarpulse has joined #openDune
09:08:02 <Xaroth|Work> depends on your definition of dev
09:08:15 <Xaroth|Work> most of us is in the EU timezone anyhow
10:29:36 * planetmaker has the feeling that real questions get better answers than "anyone here who can answer random questions" ;-)
13:54:50 *** TrueBrain has joined #openDune
14:43:32 <Sonarpulse> I sent a pm to the developers directly on he forums did any of you see it?
15:10:11 <TrueBrain> but most of us got jobs etc ;)
15:10:37 <Xaroth|Work> planetmaker: true
15:11:06 <TrueBrain> Sonarpulse: what I read from what you want, would be nearly impossible
15:11:34 <TrueBrain> the only reason we could do OpenDUNE, was because it uses a very simple CPU instruction set, uses DOS (which is very simplistic from design), uses non-virtual memory and no libraries
15:12:33 <TrueBrain> even then it took months to get it to work (I spent 3 months fulltime on only getting the initial screens to work)
15:12:39 <TrueBrain> so there is the bad news
15:13:03 <TrueBrain> in regards of hoping to compile an open source library and match is byte-per-byte to an existing game, I would say is close to the border of impossible
15:13:13 <TrueBrain> well, maybe better put: highly unlikely
15:13:20 <TrueBrain> but, given enough time, you might manage that :D
15:16:02 <Sonarpulse> can you explain what would make the matching hard?
15:16:27 <Sonarpulse> let's say somebody statically linked to something without changing it
15:16:28 <TrueBrain> you need the exact same compiler under the exact same conditions
15:17:05 <TrueBrain> the game is still under active development, not?
15:17:16 <Sonarpulse> new owners, new engine
15:17:23 <Sonarpulse> this is the older version.
15:17:40 <Sonarpulse> also, the Voxlap engine it uses has one external asm module
15:17:44 <Sonarpulse> how about finding that?
15:17:51 <TrueBrain> in general, do never try to reverse engineer a game that people have financial interest in (just as mroal information, not related to your question really)
15:18:42 <TrueBrain> if I understand you correct, and there are still people around who make money from the game (one way or the other), they will be mad :P
15:19:36 <Sonarpulse> yes, they already are. But it really looks like there is nothing in the original releases telling you not to
15:20:02 <Sonarpulse> the releases then were really informal, and they can't just retroactively through a UELA on you
15:20:02 <TrueBrain> which country are you from?
15:20:10 <TrueBrain> then you are out of luck
15:20:15 <Xaroth|Work> then yer fucked :P
15:20:18 <TrueBrain> US law disallows revese engineering
15:20:23 <TrueBrain> no need for anything anywhere
15:20:30 <TrueBrain> most EU countries do too, btw
15:20:36 <Sonarpulse> this fucking country
15:20:43 <TrueBrain> so, to leave all your questions on the side, lets put it differently
15:20:48 <TrueBrain> say, you manage to reverse engineer
15:20:50 <TrueBrain> figure out all the code
15:20:55 <TrueBrain> make an open source game, like OpenDUNE
15:21:00 <TrueBrain> say you can all get that to work
15:21:12 <TrueBrain> then the original owner of the game comes, sues the fuck out of you, and leave you broke
15:21:26 <TrueBrain> so .. why do it? Isnt it better to:
15:21:42 <TrueBrain> you have the protocol, you have the engine itself .. build the game logic by "clean room principals"
15:22:00 <TrueBrain> this basically means you work with 2 people: 1 writes down what he sees in the original game, he can even use disassembly for this
15:22:04 <TrueBrain> he gives this to you
15:22:09 <TrueBrain> and you write the code for it
15:22:17 <TrueBrain> this, by all current laws, is 100% legit
15:22:20 <Sonarpulse> yeah it wouldn't just be me working on this
15:22:37 <Sonarpulse> probably some europians to, the old version of the game is pretty popular right now
15:22:40 <TrueBrain> who ever the 2 people are; the important thing is that the person writing the code never looked in the code he is "copying from"
15:22:54 <TrueBrain> so you can never be told you stole anything
15:23:20 <TrueBrain> the only reason I took the gamble of reverse engineering OpenDUNE, was because I tried a few months finding the original author
15:23:24 <TrueBrain> or the copyright holder in fact
15:23:27 <TrueBrain> and failed to find it
15:23:32 <TrueBrain> Westwood no longer exists
15:23:37 <Xaroth|Work> yeh, that was a fun battle
15:23:38 <TrueBrain> Virgin Entertainment gives no reply
15:23:47 <TrueBrain> so, I did my atmost to get permission
15:23:51 <TrueBrain> (and I have means to proof that)
15:23:56 <Sonarpulse> ea says no reverse engineering or no they are not copywrite holder?
15:24:00 <TrueBrain> that means they acan never sue me; just ask me to shut down
15:24:06 <TrueBrain> which I will as soon as someone asks me btw
15:24:29 <TrueBrain> EA says they have no copyright claim on the game
15:24:40 <TrueBrain> they basically only have a distribution license
15:24:46 <TrueBrain> even there they were vague
15:25:14 <TrueBrain> what I have seen in the past few months, that people see OpenDUNE and think: I can do that with any game
15:25:18 <TrueBrain> sadly, it is not that easy :P
15:25:28 <TrueBrain> both technical, but also in the domain of the law
15:25:39 <Xaroth|Work> at least Sonarpulse actually wants to talk about it :P
15:25:45 <Xaroth|Work> most just go 'omg why not'
15:25:56 <Xaroth|Work> then go rageface when they realise the answer is more complex than the question
15:25:57 <TrueBrain> :D That dude was just rude ...
15:26:06 <Xaroth|Work> and an absolute IDIOT
15:26:21 <TrueBrain> so, Sonarpulse, if I were you, I would just reinvent the game, rather then reverse engineer (thin line, but meh :P)
15:26:32 <Sonarpulse> haha, someday I want to make VERY advanced optimizing compilers
15:26:35 <TrueBrain> even so, after all my blabla, if you still want to know more, just yell, and I will try to help you out ;)
15:26:39 <Sonarpulse> so maybe that gives me some perspective
15:27:22 <TrueBrain> and it is funny, a few days ago I was raging to Xaroth that the game now was paid
15:27:29 <TrueBrain> IRC is fine, forum is fine, all is fine
15:27:55 <TrueBrain> basic rule of reverse engineering: don't piss in the garden of those who make money from that garden :P
15:28:18 <Xaroth|Work> unless you intend on taking a crap into said garden
15:28:26 <Xaroth|Work> but then don;t look upset when the monkeys fling poo back at you
15:29:56 <TrueBrain> you have the server, the protocol, the engine ... so all you need is the client-side game logic, right?
15:30:35 <Sonarpulse> the enet thing is already portable to
15:30:41 <Sonarpulse> and well voxlap compiles now
15:30:43 <TrueBrain> so a rewrite for that should be "relative easy", not?
15:31:18 <Sonarpulse> Yeah. Really I am mainly interested in a reverse engineering if it could accelerate that process.
15:31:40 <Xaroth|Work> reverse engineering is slow
15:31:44 <TrueBrain> I would have written OpenDUNE in ... 30% of the time it took us now, if done from scratch :D
15:31:52 <Xaroth|Work> and with 80% less bugs :P
15:32:09 <Sonarpulse> well openDUNE needed emu, you had no source
15:32:25 <TrueBrain> you already have so much .. it is basically done :P
15:32:42 <Sonarpulse> RE is always hard, but I was openDUNE would be on the harder end and this would be the easier end
15:33:20 <Sonarpulse> Well then I guess i better bring up the issues in my port
15:33:22 <TrueBrain> 16bit real mode DOS applications are easiest, in the spectrum of RE :D
15:34:07 <Sonarpulse> someday it will fully hit me why
15:34:47 <Sonarpulse> anyways with my port. there is one commit I know where things got srewed up
15:35:07 <TrueBrain> compare it with RE a bike or a modern F1 car ;)
15:35:55 <TrueBrain> which commit would that be?
15:36:32 <Sonarpulse> The C is not mine, it is the original authors.
15:38:07 <Sonarpulse> There were three seperate instances within this massive function where I could convert asm back to C. The first and the third went fine, but the second makes debrie from braking voxels all white. making me think some pointer stuff with the colors is messed up
15:38:48 <TrueBrain> ugh ... assembly ... why do people think they are more clever than compilers *shrug*
15:39:35 <TrueBrain> how many lines in 1 file, lolz :D
15:39:56 <Xaroth|Work> OVER NINE THOUSAND! :P
15:40:57 <Sonarpulse> haha yup. In the original author's defence, MSVC didn't know any SIMD instructions then. So he'd keep things in SSE regesters between function calls, did some self modifying code, and all sorts of stuff
15:41:18 <TrueBrain> counter argument: dont use MSVC :D
15:42:53 <TrueBrain> lolz: if (!((color & 0xFFFF) << 8) & (color & 0xFFFF00))) // R == G == B
15:43:01 <TrueBrain> it is true, but I wonder if it is faster :P
15:43:05 <Sonarpulse> ...wasn't my design choices.
15:43:17 <TrueBrain> Now I want to test that ......
15:43:34 <TrueBrain> 2 cmps over 1 shl, 2 ands and 1 not
15:43:52 <TrueBrain> so basically, 1 cmp against 1shl, 2 ands and 1 not
15:44:19 <TrueBrain> and 4 memory accesses against 2 ..
15:44:42 <TrueBrain> hmm ... curious optimization ;)
15:45:03 <TrueBrain> "premature optimization", as the call it
15:45:08 <TrueBrain> right, sorry, I drifted off a bit :D
15:45:22 <TrueBrain> I am always curious how people try to speed code up, and if it holds true :)
15:45:45 <Sonarpulse> well somewhere there once was a more sane version of this
15:45:46 <TrueBrain> anyway, I know very very little about non-80386 assembly
15:45:51 <TrueBrain> so I am of no use to you :P
15:46:44 <Sonarpulse> oh well. Thanks for looking at it anyways. those should be 64 bit registers btw if you are curious
15:47:16 <TrueBrain> in my world, everythnig is 16bit, or it doesnt exist :D
15:47:58 <Sonarpulse> how long were floats on the early fpus?
15:48:09 <Sonarpulse> or did that come with i386?
15:48:31 <TrueBrain> FPUs exist for a long time
15:48:44 <TrueBrain> but it is an extension instruction set
15:49:05 <TrueBrain> in the old days, sending stuff to the FPU was slow
15:49:09 <TrueBrain> so most games didn't
15:49:27 <TrueBrain> it still is slower, just our math got complex enough to compensate :D
15:52:56 <Sonarpulse> With open dune, you had to boot strap on the original executable for a long time too right?
15:53:55 <TrueBrain> depends on what you mean exactly, but we emulated BIOS + DOS
15:54:10 <TrueBrain> which then loads DUNE2.EXE, and starts the execution
15:54:53 <Sonarpulse> it wasn't tell the very end it could run just off your code?
15:55:28 <TrueBrain> hmm, no, not really
15:55:34 <TrueBrain> very soon DUNE2.EXE was no longer needed for the code
15:55:40 <TrueBrain> except for the parts we didnt know about yet
15:55:47 <TrueBrain> but it still was, and is, needed for the data segments
15:56:03 <TrueBrain> for a long time we did need libemu and others to still give a DOS env to the game
15:56:16 <TrueBrain> those parts have no mostly been removed, meaning no longer there is need for a library like that
15:56:17 <Sonarpulse> does opjdump do a piss-poor job at figuring out code vs data then?
15:56:37 <TrueBrain> objdump works perfectly on modern files
15:56:50 <TrueBrain> in case of DUNE2 there are no code nor data pages
15:57:19 <TrueBrain> also how in the old days the protected against decompilation processes
15:57:44 <Sonarpulse> and you had TONs of self modifying code right?
15:57:53 <TrueBrain> DUNE2 not, but it was normal back then, yes
15:58:04 <TrueBrain> well, DUNE2 does, but it uses a technique called overlays
15:58:09 <TrueBrain> which is not really the same
15:58:21 <TrueBrain> (blocks of code can be replaced by others, depending on what is needed at the moment
15:58:29 <TrueBrain> it was merely there to reduce memory footprint)
15:58:42 <TrueBrain> other games I have seen decompress on startup
15:58:46 <Sonarpulse> well how hard is it to re assemble a modern executable without changing it then?
15:58:59 <TrueBrain> very strong to the side of impossible
15:59:51 <Sonarpulse> just in case it makes a difference, I meant not making any purposefull changes
15:59:57 <TrueBrain> well, hmm .. I guess it depends what you really want tbh :P
16:00:37 <TrueBrain> you can of course make a simple tool that executes the excutable, reads the instruction, writes the instruction as a C function, and runs through any file like that
16:00:45 <TrueBrain> then all you have to do is implement all CPU insturctions in C
16:00:48 <TrueBrain> it would be VERY VERY slow
16:00:51 <TrueBrain> but it would work :P
16:00:59 <TrueBrain> kinda what we did at first for OpenDUNE :P
16:01:09 <Sonarpulse> isn't that's what IDA does?
16:01:21 <TrueBrain> IDA does static decompilation
16:01:33 <TrueBrain> it has no runtime analyzes
16:01:38 <TrueBrain> that is why it is a suggestion of what can happen
16:01:44 <TrueBrain> and you have to fix all kinds of things
16:01:54 <TrueBrain> what I suggest is a runtime decompilation
16:02:02 <TrueBrain> which is 100% correct, but only knows what has been executed
16:02:08 <TrueBrain> pros and cons to both ;)
16:02:12 <Sonarpulse> well if you don't have the overlay things, or real SMC, what are the advantages of runtime decompilation?
16:02:18 <TrueBrain> but IDA can never make any compilable code
16:02:20 <Sonarpulse> i'm a bit murky on that
16:02:40 <TrueBrain> runtime decompilation tells you what really got executed
16:02:46 <TrueBrain> static decompilation gives you an idea of what might be executed
16:03:23 <TrueBrain> you can look on the weather channel to see if it rains outside (static decompilation), or you can just walk outside and see if you hair gets wet (runtime decompilation)
16:03:37 <Sonarpulse> well with no write on exec pages, I'd think static would be easier without much downside
16:03:44 <TrueBrain> the first tells you what might happen, but also what happens 100km away
16:03:51 <TrueBrain> the latter tells you what is happening, but only here and now
16:04:17 <TrueBrain> you would be surprised ;) But IDA is a very good tool; just it will never produce a compilable source. Then again, it is not made to do so, so :)
16:04:57 <Sonarpulse> yeah IDA definitely didn't have that in mind.
16:05:21 <TrueBrain> there are plugins who try
16:05:24 <TrueBrain> but it is fundementally flawed
16:05:26 <TrueBrain> so they never worked
16:06:14 <Sonarpulse> I guess I'll put it this way, if I did do a big objdump, re assembled making sure that the sections were the right length, and then used dd or someting to put back in the data sections, what would go wrong?
16:06:55 <TrueBrain> if it returns identical bytes, nothing ofc
16:06:58 <TrueBrain> as .. you did nothing ;)
16:07:07 <TrueBrain> it is how cracks etc do their job, basically
16:07:46 <Sonarpulse> if i find every single jump instruction, replace it with a symbol, does that give me more freedom?
16:08:09 <TrueBrain> getting to hypotetical
16:08:47 <Sonarpulse> also, the data and bss all get combined linktime right? or do you have more than one?
16:09:07 <TrueBrain> depends on the compiler, but you can have as many .data and .bss as you like
16:15:39 <Sonarpulse> Is it in your mind the fact that IDA is a static dissasembler that makes it fundamentally unable to make something recompiable?
16:15:46 <Sonarpulse> or something else?
16:16:34 <TrueBrain> in the general context it is not possible to make something recompilable without knowing something about the execute env
16:16:49 <TrueBrain> strictly seen you can feed IDA enough knowledge to figure everything out
16:17:03 <TrueBrain> but that would require knowing the compiler and system it operates on in very strong details
16:17:30 <TrueBrain> so yes, IDA cannot make recompilable code for every source, because it is a static decompiler; but again, IDA is not meant for that, and should not be used as such
16:17:41 <TrueBrain> IDA is meant to look in depth in an application, and figure out how it works your self
16:17:52 <TrueBrain> mostly, I guess, as reverse engingering is illegal in many many countries ;)
16:18:05 <TrueBrain> (Well, it is only legal if it is for your own research :P)
16:18:29 <Sonarpulse> ok. when you say executable enviroment what do you mean?
16:18:47 <TrueBrain> a simple example: switches
16:18:48 <Sonarpulse> especially if it's a "generic" win32 binary that's supposed to run anywhere
16:18:59 <TrueBrain> depending on the compiler, switches are solved in different ways
16:19:13 <TrueBrain> if you have C code like: switch(a) { case 1: case 2: case 3 }
16:19:38 <TrueBrain> what a stupid compiler would do: make in the memory a table with pointers, like an array, from 0 to 3 (4 entries)
16:19:55 <TrueBrain> entry [1], [2] and [3] point to a pointer in the code where those cases are
16:19:55 <Sonarpulse> that sounds more like compile enviornment than runtime environment though
16:20:05 <TrueBrain> [0] and an if() before points to code after the swtich
16:20:26 <TrueBrain> this block can be either in the execution segment, in the data segment
16:20:37 <TrueBrain> other compilers optimize, by substracting with 1
16:20:42 <TrueBrain> many variations there
16:21:02 <TrueBrain> so you need to know which compiler, and how it did switches
16:21:06 <TrueBrain> else you will never figure out what goes on here
16:21:28 <TrueBrain> you need to know the system you operate on in very strong details, because you need to know what syscalls do EXACTLY
16:21:31 <TrueBrain> what comes back in the registers
16:21:37 <TrueBrain> and how are things handled
16:21:52 <TrueBrain> the first, strictly seen, can be figured out in a static context
16:21:55 <TrueBrain> if you know which compiler etc
16:21:58 <TrueBrain> IDA has a lot of code for this
16:22:05 <TrueBrain> and figures out most of it ... most of the time .... mostly right
16:22:10 <TrueBrain> the latter is MUCH harder
16:22:15 <TrueBrain> and you need to do some deduction on it
16:22:28 <TrueBrain> basically, you need to execute the code, to figure out what really happens
16:22:35 <TrueBrain> then you leave the static decompilation, and enter the runtime ;)
16:22:36 <Sonarpulse> with the syscalls?
16:22:44 <TrueBrain> syscalls, I/O calls, .....
16:22:52 <TrueBrain> anything external defined by the executing system
16:23:03 <TrueBrain> you will see that IDA also stops here, and just tells you what you can expect
16:23:11 <Sonarpulse> isn't that known stuff with windows?
16:23:25 <TrueBrain> mostly, sure ... by estimation, sure ...
16:23:36 <TrueBrain> wine does an epic job
16:25:02 <TrueBrain> I think it is easiest explained how you see IDA read a file
16:25:03 <Sonarpulse> on windows, libc is dynamic linked too right?
16:25:09 <TrueBrain> it first makes an initial sweep
16:25:14 <TrueBrain> and it guesses where code is, where data is etc
16:25:26 <TrueBrain> then as it goes on, it finds it made bad calls, and corrects those
16:25:37 <TrueBrain> it often runs a few time over, till it figured out all the ins and outs
16:25:50 <TrueBrain> even then it misses stuff
16:26:10 <TrueBrain> I really don't know a lot about Windows ... google will tell you I am sure, if libc is dynamically linked or not :P
16:26:48 <Sonarpulse> well i guess either way brings me to my next point. you usually don
16:27:12 <Sonarpulse> t see to many symobls in executables AFIAK, how does the dynamic linking work?
16:28:11 <TrueBrain> you should see a lot of symbols
16:28:18 <TrueBrain> else it doesnt link dynamically :P
16:28:33 <Sonarpulse> can mingw pick em up?
16:29:20 <TrueBrain> objdump even tells you
16:29:57 <Sonarpulse> hmm i must be doing something wrong then. I know for a fact it dynamically links with msvc c++ runtime
16:31:57 <Sonarpulse> i586-mingw32msvc-objdump -t
16:32:15 <TrueBrain> remember you have imports and exports
16:33:33 <TrueBrain> a dll exports symbols
16:33:36 <TrueBrain> an exe imports symols
16:33:41 <TrueBrain> the latter can be done runtime
16:33:43 <Sonarpulse> all the more reason I should have a symbol table right?
16:34:05 <TrueBrain> on a dll you will always see export symbols, yes
16:34:46 <Sonarpulse> i guess true runtime linking could work differently
16:35:07 <Sonarpulse> calling some linker library directly
16:35:27 <Sonarpulse> the library couldn't really look at smybols in the static executable anyways
16:36:10 <Sonarpulse> well i should probably go
16:36:29 <Sonarpulse> thanks so much more all the information TrueBrain (and Xaroth too)
16:36:42 <TrueBrain> np, and good luck with your project :)
16:36:50 <Sonarpulse> I will let them no that I was wrong about revese engineering, even in this case it is still hard as hell
16:37:15 <Sonarpulse> thank you. hopefully with so much open source we can just right a clone quickly
16:37:40 *** Sonarpulse has left #openDune
continue to next day ⏵