QB Re-compiling Error with Formatted Strings

Discuss thug2 modding, post your own, share your ideas, ask questions.
LeonBlade
Posts: 11
Joined: Fri Apr 17, 2015 10:48 am
Contact:

QB Re-compiling Error with Formatted Strings

Postby LeonBlade » Sat Apr 18, 2015 11:13 am

Hello,

Let me just preface this by saying this isn't just some noobish "pls help me" thread.

I just decompiled and compiled cas_logos.qb only to have it fail without any modifications. I tried looking into the problem, but it's kind of difficult to track down exactly just by searching. The issue in question is when compiling something like this:

Code: Select all

:i :s{
   :i $desc_id$ = $My Cool Texture$ // <--- here is the problem
   :i $frontend_desc$ = %sc(15,"My Cool Texture")
   :i $common_hatlogo_params$
   :i $with$ = %s(30,"textures/logos/My_Cool_Texture")
   :i $sponsor$ = $neversoft$ // idk what relevance this has honestly
:i :s}


The compiler will throw an error expecting the $ to end the desc_id, and instead is met with a space and it might throw errors on other characters, haven't really looked that far into it. So, the obvious solution would to remove all of the spaces and other characters right? Of course, that works, but the problem is that this is a pain in the ass given how many things need to be replaced. I'd rather just not fuck with it and have it be formatted properly the first time. I'm pretty sure it's just copying the frontend_desc and making it as the identifier as well, and so this is a problem as you can imagine.

So, this is my first time really doing any of this, so I don't know if I'm just an idiot and there's some newer compiler that has fixed this (given there are many sites with many links to outdated programs all over the web). I found the source for qb_new qb_old and qb_fast, I'm not sure how recent the revision is compared to something like QB2.1 for example, I haven't compiled against it yet to see if it generates the CRC-32 hashes for you in the table or not (as I had this problem with one compiler). Anyways, there's a text file that describes the opcodes and what's implemented and all, and so right now I'm just figuring out exactly where the desc_id is being created, I found the hash for it in the file and so right now I'm just reading it with my eyes trying to see what's going on here.

My question is, is there something out there already that has resolved this, or is this just one file that people haven't bothered to replace, and it's one of the only files that has this issue, therefore no one has really encountered it? I'm going to install Visual Studio and get this compiled and test it out and try to compile something and see if I can fix the problem myself either way.

Thanks.
Demo
Posts: 494
Joined: Mon Aug 15, 2011 2:20 pm
Contact:

Re: QB Re-compiling Error with Formatted Strings

Postby Demo » Sat Apr 18, 2015 6:17 pm

i doubt generations of thps modders missed that out :P
http://www.mediafire.com/download/dq041 ... oq_CMD.rar

the sources you're talking about are pretty much outdated. i guess by the time it was written not even all codes were found.

for thps related tools you don't have to check anything but this and thmods.
LeonBlade
Posts: 11
Joined: Fri Apr 17, 2015 10:48 am
Contact:

Re: QB Re-compiling Error with Formatted Strings

Postby LeonBlade » Sat Apr 18, 2015 9:14 pm

Thank you, like I said, this is my first time doing anything with THUG2 on the PC side of things with modding.

Edit: I realized what you mean now, I just had woken up when I read your reply. Unfortunately, I do think that this is some sort of oversight as the roq.exe included in what you sent me is just version 2.1 and doesn't change the output in any way.

The sources are out of date, yes, but the qb_new seems to just be revision 2.0. I've found up to date information on the opcodes on this or the other forums, and so I think at this point the best option would be to modify the source myself and perhaps re-implement the features added in 2.1 to it, either that or rely on one version for compression and another for decompression.

I've tried to look up both cas_logos.qb and decks.qb online to see if anyone has said anything related to them, and I only find links to download websites or strange foreign websites that look to just be junk, as well as a forum topic that mentions decks.qb and to modify the hex manually to change something.

Unless you have any other suggestions, it looks like the only solution is to modify the compiler. It seems like I can probably get away with just finding where it's matching the names and allow it to match more loosely than it currently is, and it should compile without needing to change anything substantial that would break something else.
LeonBlade
Posts: 11
Joined: Fri Apr 17, 2015 10:48 am
Contact:

Re: QB Re-compiling Error with Formatted Strings

Postby LeonBlade » Sun Apr 19, 2015 9:23 am

Just posting an update.

I went ahead and edited the qb_new and added the functionality to generate CRC hashes without needing to modify the table file like version 2.1 has and I've also loosened up how it matches the names for them better. It only accepted alphanumerical values not included space, and there are plenty of IDs in at least the two files I worked with that have special characters like single quotes periods and even starting with numbers which the compiler previously didn't like.

This change has allowed me to decompile decks.qb and compile it with no errors. The only thing I need to do now is have it write the table file again so that it's not doing extra hash calculations that it doesn't need to on subsequent compilations and it will be on par with 2.1 without needing to double compile.

Here is proof of it recompiling (not like anyone wouldn't believe me) I just added \c2 in the front of Created Graphic as you can see here. If you don't believe that this is an issue, go ahead and try to decompile and compile decks.qb yourself. If for some reason it works, then please send me your version of qb.exe :D

Image
LeonBlade
Posts: 11
Joined: Fri Apr 17, 2015 10:48 am
Contact:

Re: QB Re-compiling Error with Formatted Strings

Postby LeonBlade » Sun Apr 19, 2015 12:14 pm

Ah, thank you, I was right I guess. I'll release this version of the compiler once I fix the problems from before, I'm working on figuring out how the decks are swizzled, but that's off topic so I won't go into further detail.

Thanks for your help!
WhoElseButMe
Posts: 419
Joined: Tue Aug 04, 2009 12:50 am
Location: FL - USA
Contact:

Re: QB Re-compiling Error with Formatted Strings

Postby WhoElseButMe » Mon Apr 20, 2015 8:59 pm

What demo said.
The QbKey tool used with that compiler doesn't work with non alpha numeric values maybe a couple exceptions like slash will work.
One way around it is to gather all the checksums and modify them to compile, with edits.
Then hex edit the original checksums back into the file by doing a hex search for the modified qbkey.
Make not to edit the table entries checksum and value too else the next decompile will have mismatched key value pairs.
Image
WhoElseButMe on Nov 26, 2009 wrote:It's that lack of respect amongst their peers and ignorance towards modding etiquette that keeps us who know this stuff well from spreading it like wild fire. We do still enjoy playing the game and if you need to cheat to play a game PLAY SOMETHING ELSE YOU DON'T SUCK AT.
LeonBlade
Posts: 11
Joined: Fri Apr 17, 2015 10:48 am
Contact:

Re: QB Re-compiling Error with Formatted Strings

Postby LeonBlade » Mon Apr 20, 2015 10:57 pm

WhoElseButMe wrote:What demo said.
The QbKey tool used with that compiler doesn't work with non alpha numeric values maybe a couple exceptions like slash will work.
One way around it is to gather all the checksums and modify them to compile, with edits.
Then hex edit the original checksums back into the file by doing a hex search for the modified qbkey.
Make not to edit the table entries checksum and value too else the next decompile will have mismatched key value pairs.

Yeah, thanks. I just ended up changing the compiler to treat the keys more loosely and it compiles just fine now.
WhoElseButMe
Posts: 419
Joined: Tue Aug 04, 2009 12:50 am
Location: FL - USA
Contact:

Re: QB Re-compiling Error with Formatted Strings

Postby WhoElseButMe » Tue Apr 21, 2015 4:31 am

Are you sure the qbkeys it creates now are correct?
Here's a qbkey tool written in c# that can generate standard qbkeys or unicode keys (.qs file qbkeys found in GH).
https://copy.com/CZQduEftcbpyzKtY
No obfuscation was used so it can easily be decompiled.
Image
WhoElseButMe on Nov 26, 2009 wrote:It's that lack of respect amongst their peers and ignorance towards modding etiquette that keeps us who know this stuff well from spreading it like wild fire. We do still enjoy playing the game and if you need to cheat to play a game PLAY SOMETHING ELSE YOU DON'T SUCK AT.
LeonBlade
Posts: 11
Joined: Fri Apr 17, 2015 10:48 am
Contact:

Re: QB Re-compiling Error with Formatted Strings

Postby LeonBlade » Tue Apr 21, 2015 9:48 am

WhoElseButMe wrote:Are you sure the qbkeys it creates now are correct?
Here's a qbkey tool written in c# that can generate standard qbkeys or unicode keys (.qs file qbkeys found in GH).
https://copy.com/CZQduEftcbpyzKtY
No obfuscation was used so it can easily be decompiled.


Ah, thank you, I will check it out. I haven't done a complete compile of every source and verified myself, but I'm fairly certain that my changes aren't detrimental to the compiler, basically it's just matching between the $ characters I found that were interfering in deck.qb or cas_logos.qb for example.

Right now I'm focusing on learning the QB source syntax after decompiling every script from qb_scripts.prx (minus the two that create 4 billion tabs because of some strange overflow bug I haven't looked into yet). I'm noticing a lot of inconsistencies with how things are written out and finding that words like "call" and "arguments" are completely fluff and serve no purpose other than to provide "user friendly" syntax to work with. The issue with this, is that with the inconsistencies of this, it makes for reading and understanding this a real headache.

I have to ask, with THUG Pro being worked on, it seems like working with this QB source would be a hinderance. I'm slowly making sense of a lot of things here, but at the same time it seems like working on a mod that changes a lot of functionality would be better suited to be written in something that can be easily more managed with features to make writing the script easier.

I've only seen one other compiler that was written in Java, and I'm not really sure how old it is or if it even works. I was thinking about writing my own compiler from scratch but still have a lot of research ahead of me and a few test scripts to write. I don't really have any ideas for a mod myself, so this is more just for fun than for anything.

Anyways, I'm pretty ignorant to all of this, so maybe it's not as big of a deal as I think it is.

Thanks for your reply.
WhoElseButMe
Posts: 419
Joined: Tue Aug 04, 2009 12:50 am
Location: FL - USA
Contact:

Re: QB Re-compiling Error with Formatted Strings

Postby WhoElseButMe » Tue Apr 21, 2015 12:39 pm

The tool was provided so you can type in a checksum with unique characters and then create a key from the same text with your qbkey tool and make sure they match. If they match you should be perfectly fine.

call arguments was provided to split the function name and its parameters. When a function has parameters they're all on the same line (in bytecode).
This isn't an inconsistency thing, this is very much saying there isn't a new line in bytecode, but i'm adding one on decompile
The call and arguments words don't need to be there, the parameters don't need to be on another line either. However, if they are on another line they don't get the ":i". With that same logic one can make a struct that is all one line in bytecode, but written in code as multiline, which is far easier to read.
:i $something$ = :s{
$somevalue$ = %i(3,00000003)
:s}

Writing a compiler to handle all the gotchas isn't that hard. I wrote one and it still has a few issues compiling just never finished it.
The syntax isn't blubs either and it uses a new tab for each loaded script.
https://www.copy.com/s/ClqAxYyVV9QOUrtP/Captureblah.PNG
Image
WhoElseButMe on Nov 26, 2009 wrote:It's that lack of respect amongst their peers and ignorance towards modding etiquette that keeps us who know this stuff well from spreading it like wild fire. We do still enjoy playing the game and if you need to cheat to play a game PLAY SOMETHING ELSE YOU DON'T SUCK AT.
LeonBlade
Posts: 11
Joined: Fri Apr 17, 2015 10:48 am
Contact:

Re: QB Re-compiling Error with Formatted Strings

Postby LeonBlade » Tue Apr 21, 2015 2:39 pm

WhoElseButMe wrote:The tool was provided so you can type in a checksum with unique characters and then create a key from the same text with your qbkey tool and make sure they match. If they match you should be perfectly fine.

Oh, yeah I know, sorry if I didn't phrase what I was trying to say right. I was just meaning I haven't gotten around to checking outside of when I recompiled decks. Thank you again.

WhoElseButMe wrote:call arguments was provided to split the function name and its parameters. When a function has parameters they're all on the same line (in bytecode).
This isn't an inconsistency thing, this is very much saying there isn't a new line in bytecode, but i'm adding one on decompile
The call and arguments words don't need to be there, the parameters don't need to be on another line either. However, if they are on another line they don't get the ":i". With that same logic one can make a struct that is all one line in bytecode, but written in code as multiline, which is far easier to read.
:i $something$ = :s{
$somevalue$ = %i(3,00000003)
:s}

Sorry, I didn't really explain what I meant here, here is an example from particle_scripts.qb

Code: Select all

:i if call $ObjectExists$ arguments
   $id$ = %GLOBAL%$name$
   :i if call $StructureContains$ arguments
         $structure$ = %GLOBAL%$params_script$$EmitDuration$
      :i %GLOBAL%$name$.call $Obj_SpawnScript$ arguments
         $EmitFastParticles$$Params$ = :s{$wait_seconds$ =  ( (%GLOBAL%$params_script$) ->$EmitDuration$) :s}

Code: Select all

:i if call $ObjectExists$ arguments
   $id$ = %GLOBAL%$name$

This line makes sense, it's calling $ObjectExists$ and passing in something named $name$ in the current scope to it.

Code: Select all

:i %GLOBAL%$name$.call $Obj_SpawnScript$ arguments
   $EmitFastParticles$$Params$ = :s{$wait_seconds$ =  ( (%GLOBAL%$params_script$) ->$EmitDuration$) :s}

This is also calling a function but call comes after %GLOBAL%$name$ and after the period, which is confusing because the period signifies that you are accessing something on that name, but call is just placed in there. Then you have entire next line which is assumingly calling the function $EmitFastParticles$ with params or it's adding it to a call stack, I don't actually know as I haven't really experimented with it. If it was being added to a call stack it would make sense as to why call and arguments aren't present, I don't know what $Obj_SpawnScript$ is doing exactly so I don't know.

Code: Select all

:i $alloc_fast_particle$:s{$params_script$ =  (%GLOBAL%$params_script$) $name$ = %GLOBAL%call $mangled_id$ arguments
      $attachObjId$ = %GLOBAL%$attachObjId$
   :i $emit_script$ = %GLOBAL%call $emit_script$ arguments
      $emit_params$ = %GLOBAL%$emit_params$:s}

And there's this in the same file (this is an instance where $name$ is passed to the example above).
There's a function call to $alloc_fast_particle$ that doesn't even have call, unless it somehow made its way after %GLOBAL%, because I don't know what call is doing there given that $mandled_id$ is just a reference to a value on the scope and not a function.

This is what I meant by inconsistencies, after I looked into it more, it DOES look like references to call and arguments are made in the right places, but sometimes call is completely in the wrong area which made things at a first glance look like they weren't consistent.

WhoElseButMe wrote:Writing a compiler to handle all the gotchas isn't that hard. I wrote one and it still has a few issues compiling just never finished it.
The syntax isn't blubs either and it uses a new tab for each loaded script.
https://www.copy.com/s/ClqAxYyVV9QOUrtP/Captureblah.PNG

That's really cool. Yeah I mean that's one of the main things I'd like to address is a lot of the gotchas. There's still a lot of things I don't know about, like if $GetArraySize$ or $MangleChecksums$ or $Obj_SpawnScript are functions that live in the EXE and inaccessible or if they're in another PRX somewhere that I haven't unpacked... things like that.

All of this stuff really interests me, it's sad to see that the community isn't so big when it comes to this stuff, at least not so much on the forums that I can see.

Sorry if I made myself look like an idiot here, I'm still new to this stuff like I said.

Thank you.
WhoElseButMe
Posts: 419
Joined: Tue Aug 04, 2009 12:50 am
Location: FL - USA
Contact:

Re: QB Re-compiling Error with Formatted Strings

Postby WhoElseButMe » Tue Apr 21, 2015 9:20 pm

It's calling $Obj_SpawnScript$ with parameters. This is the function being called on by the global object $name$. The parameters of $Obj_SpawnScript$ contain a script name to spawn. In bytecode there isn't an endofline (0x01) after the $Obj_SpawnScript$ function which triggers the call/arguments text to appear by blubs tool.

Everything being passed to a function is placed inside a structure and passed as a structure of parameters.

Code: Select all

:i if call $ObjectExists$ arguments
   $id$ = %GLOBAL%$name$

Code: Select all

:i if $ObjectExists$ :s{$id$ = %GLOBAL%$name$:s}

The above code has the same meaning to the game. Note there isn't an equals sign because $ObjectExists$ doesn't equal the structure. Instead the structure contains the parameters being passed to the function.

So this

Code: Select all

$name$ = %GLOBAL%call $mangled_id$ arguments
      $attachObjId$ = %GLOBAL%$attachObjId$

is creating a $mangled_id$ with parameters and setting $name$ to its result before passing it into $alloc_fast_particle$ as a parameter.
Similarly, Math.Add((2*56), (56+11)); is actually going to pass 112 and 67.

Typical rule of thumb is if the function name has an underscore separating words of the function name then it exist in scripts else it is in the executable.
ObjectExists is an executable function
alloc_fast_particle is a script function
At the same time EmitFastParticles is a script function. So you can't always rely on this.

Modders come and go. Many people just want to make a mod and don't really care the how to's and the basic knowledge. Others grow up or lose interest, it happens. You didn't make yourself look like an idiot at all. Its a learning process.
Image
WhoElseButMe on Nov 26, 2009 wrote:It's that lack of respect amongst their peers and ignorance towards modding etiquette that keeps us who know this stuff well from spreading it like wild fire. We do still enjoy playing the game and if you need to cheat to play a game PLAY SOMETHING ELSE YOU DON'T SUCK AT.
LeonBlade
Posts: 11
Joined: Fri Apr 17, 2015 10:48 am
Contact:

Re: QB Re-compiling Error with Formatted Strings

Postby LeonBlade » Wed Apr 22, 2015 12:07 am

WhoElseButMe wrote:Modders come and go. Many people just want to make a mod and don't really care the how to's and the basic knowledge. Others grow up or lose interest, it happens. You didn't make yourself look like an idiot at all. Its a learning process.


Thank you for taking the time to explain this all to me, I greatly appreciate it. The information about the naming convention for the function names is very helpful. I'll post on the forums again if I have any questions or anything to share.

Thank you.

Return to “THUG2 modding”

Who is online

Users browsing this forum: No registered users and 19 guests