port System::Call and related functions to GCC

initial working patch

fixes lvalue FTBFS and System.nsi crash

This version of the patch fixes the invalid lvalue FTBFS properly and fixes the crash in Contrib/System/System.nsi.

TODO list for this patch:

Testing the scons changes on Windows - not sure about the .asm -> .spp
change.

Integrating the test.py into scons test stuff.
File Added: gcc-system-call-2.patch

I'm informed that test.py isn't really needed.

Posted in the forums asking for some testing of the patch on Windows:

http://forums.winamp.com/showthread.php?s=&goto=lastpost&threadid=227304

Updated patch that fixes the DLL entry point for the system plugin, allowing it to be built and work on Windows again.
File Added: gcc-system-call-3.patch

fixes DLL entry point, now builds and works on Windows

Updated patch that fixes the warnings about the use of #pragma once by only using #pragma once with MSVC and adding #include guards too.
File Added: gcc-system-call-4.patch

only use #pragma once with msvc, add include guards

Another change required for SCons to avoid conversion warnings is the following line in SCons/Config/ms:

defenv.Append(ASFLAGS = ['/coff'])

avoid some conversion warnings

Updated the patch to include the ms ASFLAGS change. Hopefully I did it right.
File Added: gcc-system-call-5.patch

I'm currently reviewing the assembly code itself. I'm already done with _CallProc and now looking at _RealCallBack. It has an issue with zero-parameters callback functions. On line 749, if there are no parameters, it jumps to cb_params_loop_done and skips line 792 which sets [ebp-12] (args size). This causes line 826 to fill bogus values into proc->ArgsSize and in turn causes _CallBack to improperly clean up the stack.

To reproduce this, I've created a simple DLL that just gets a callback function and calls it and then used the following script.

SetPluginUnload alwaysoff
System::Get ()i.s
Pop $0
System::Call scb::show(kr0)
Pop $R0
System::Call $0
SetPluginUnload manual
System::Free $0

Found another one in the same function. Large parameters are parsed in wrong order and so 1 becomes 0x100000000 and vice versa. Lines 794 (cb_params_loop) to 825 (cb_params_loop_done).

In _CallBack, it's assumed _GetValueOffsetParam and _Get_valueOffsetParam don't change ecx and edx. That might not always be the case and so eax should be pushed on the stack just like in _RealCallBack and _CallProc.

asm file with issues fixed

I've uploaded a new version of Call.S which should fix all the issues I've mentioned. Let me know if this new version is fine with you and I'll upload it.

I've notified Thomas of your review and changes.

I've committed the changes for the next version (2.42). Thank you both very much for this impressive patch. Let me know if you find any problems with the version I've committed. The next release is in about a month.

Found another issue where ebp was never popped from the stack in CallProc which caused register corruption and eventual crashes or other weird behaviors. See bug #2318670.

Another one. Don't know what causes it yet or how exactly to reproduce it, but esp is corrupted. You need Firefox for this one.

SetOutPath "$PROGRAMFILES\Mozilla Firefox"
StrCpy $R0 $TEMP\nss
System::Call 'nss3::NSS_Initialize(t R0, t "", t "", t "secmod.db", i 0) i .r0'

Solved that too. It was all due to `push ebp` being in the wrong place. It should only have been saved and restored when stack generation was required. In the original code it was always saved but restored only for POPT_GENSTACK. Then I added `pop` on the bottom and so it was popped twice for POPT_GENSTACK. Now it pushes once and pops once and only for POPT_GENSTACK.