On Software Reverse Engineering
So we have two processes: checkout in cmath.exe and
keygen in lmcrypt.exe or makekey.exe. They
both produce the same correct license codes, but the two processes are not
identical. We have analyzed the first process in some depth in previous
paragraphs, let’s list the important call chains in chronological order.
1.
lc_new_job() ® l_n36_buf() ®
l_x77_buf()
2.
lc_new_job() ® lc_init() ®
l_init() ® l_sg() ® l_key() ®
l_zinit()
3.
lc_set_attr() ® l_set_attr() ® l_set_license_path()
®
l_flush_config() ® l_init_file() ®
l_allfeat() ®
l_parse_feature_line() ® oldkey() ® l_crypt_private()
®
real_crypt() ®
l_string_key()
4.
lc_checkout() ® l_checkout() ®
lm_start_real() ® l_good_lic_key() ® l_xorname()
5.
lc_checkout() ® l_checkout() ®
lm_start_real() ® l_good_lic_key() ® l_sg() ®
l_n36_buff()
6.
lc_checkout() ® l_checkout() ®
lm_start_real() ® l_good_lic_key() ® l_crypt_private()
®
real_crypt() ® l_string_key()
An interesting question is, why does l_sg() call l_key() in the
2nd chain but l_n36_buff() in the 5th? Examining the code
excerpts we see the answer is LM_OPTFLAG_CUSTOM_KEY5 and L_UNIQ_KEY5_FUNC. The
latter is set by l_x77_buf() (i.e. L_SET_KEY5_FUNC) in the
first chain so in both calls l_n36_buff is not null. Then the reason is
LM_OPT_FLAG_CUSTOM_KEY5: it is
switched on after calling lc_init(), that’s
why l_key() is invoked in the 2nd chain. The funny thing is, l_key() is a
useless subroutine in modern FLEXlm versions (it’s for earlier versions before l_n36_buff() is
introduced). In addition, it is utterly unnecessary to call l_sg(), which
decodes encryption seeds, in the initialization stage; that should be done only
at checkout time.
lm_njob.c:
int lc_new_job(oldjob, l_new_job, vcode, newjobp)
{
... ...
(*L_NEW_JOB)(vendor_name,
vcode, 0, 0, 0, &sign_level);
(*L_NEW_JOB)(0, 0,
0, 0, 0, 0);
if (!(ret = lc_init(oldjob, vendor_name, vcode,
newjobp)))
{
(*newjobp)->options->flags
|= LM_OPTFLAG_CUSTOM_KEY5;
... ...
}
return ret;
}
lm_ckout.c:
void l_sg(LM_HANDLE* job, char* vendor_id, VENDORCODE* key)
{
... ...
unsigned long x =
0x6f7330b8; /* v8.x */
if ((
job->options->flags & LM_OPTFLAG_CUSTOM_KEY5) &&
L_UNIQ_KEY5_FUNC)
{
(*L_UNIQ_KEY5_FUNC)(job,
vendor_id, key);
return;
}
l_key(vendor_id,
&(key->keys[0]), keys, 4); /* Pre v6.1 style
*/
... ... /* same xor operations in VKEY5() */
}
lm_init.c:
void (*L_UNIQ_KEY5_FUNC)() = 0;
void L_SET_KEY5_FUNC( void (*f)())
{
if (!L_UNIQ_KEY5_FUNC)
L_UNIQ_KEY5_FUNC = f;
}
Parallel to the checkout process, we also have the call
chains in the keygen process. We shall use lmcrypt.exe for
analysis because it’s more straightforward than makekey.exe (they
two perform the same job).
1.
lmcrypt.c!main() ® lc_init() ®
l_init() ® l_sg() ® l_key() ®
l_zinit()
2.
lmcrypt.c!main() ® dofilecrypt() ®
dofpcrypt() ® lm_crstr.c!lc_cryptstr() ®
parsefeaturelist()
® l_parse_feature_line() ®
oldkey() ® l_crypt_private() ® real_crypt() ®
l_string_key()
3.
lmcrypt.c!main() ® dofilecrypt() ®
dofpcrypt() ® lm_crstr.c!lc_cryptstr() ®
cryptfeaturelist()
® docryptfeat() ® lc_crypt() =
l_crypt_private() ® real_crypt() ® l_string_key()
Note cmath.exe calls lc_new_job(), which
in turn calls lc_init(), for vendor & job
initialization but lmcrypt.exe calls lc_init() directly
because vendor keys, seeds and name are already included in lmcrypt.exe (put
together in vendor structure by macros) so it only needs to initialize job. In
both processes there are two calls to l_string_key() and in
both situations the first one returns 21D5B6E8572E, the insignificant number
for oldkey(), and only the second call matters. The two
processes calls l_string_key() in slightly different ways,
basically checkout needs to provide user license key for checksum comparison
but keygen doesn’t need that input. However the part for calculating the true
hash are the same.
int idx = (*job->vendor) % XOR_SEEDS_ARRAY_SIZ; /* idx = V % 20 = 86 % 20 = 6 */
... ...
memset(y, 0,
L_STRKEY_BLOCKSIZE); /*
L_STRKEY_BLOCKSIZE = 8, in lmachdep.h */
length = (inputlen) /
L_STRKEY_BLOCKSIZE;
XOR_SEEDS_INIT_ARRAY(xor_arr) /* substitution table defined in l_strkey.h */
... ... /* memcpy() from
input to newinput, and other stuff */
p = newinput;
for (i = 0; i < length; i++)
{
XOR(p, y, y);/* XOR and L_MOVELONG
defined in l_strkey.h */
if (i == 0)
{
if
(!user_crypt_filter && !user_crypt_filter_gen
&&
(job->flags & LM_FLAG_MAKE_OLD_KEY))
{
q = y; /* SEEDS_XOR = mem_ptr2_bytes
defined in l_privat.h */
L_MOVELONG(code->data[0]
^((long)(job->SEEDS_XOR[xor_arr[idx][0]])<<0)
^((long)(job->SEEDS_XOR[xor_arr[idx][1]])<<8)
^((long)(job->SEEDS_XOR[xor_arr[idx][2]])<<16)
^((long)(job->SEEDS_XOR[xor_arr[idx][3]])<<24),
q)
L_MOVELONG(code->data[1]
^((long)(job->SEEDS_XOR[xor_arr[idx][0]])<<0)
^((long)(job->SEEDS_XOR[xor_arr[idx][1]])<<8)
^((long)(job->SEEDS_XOR[xor_arr[idx][2]])<<16)
^((long)(job->SEEDS_XOR[xor_arr[idx][3]])<<24),
q)
}
... ...
}
if (!(job->flags
& LM_FLAG_MAKE_OLD_KEY) && !demo)
our_encrypt2(y);
else
our_encrypt(y); /* our_encrypt() does not involve code or job */
p += L_STRKEY_BLOCKSIZE;
}
if (len == L_SECLEN_SHORT) /* L_SECLEN_SHORT = 0x66D8B337 in l_privat.h */
{
... ...
y[6] = y[7] = 0;
}
Since they share the same code, to compute the same hash
string (otherwise what’s the point?) the arguments passed in must be the same.
Tracing result in practice is:
cmath.exe:
job
= 00887630, input = 0012e170, inputlen = 0x16, code = 0012ee20, len = 66d8b337,
license_key = 6d5c...
[00887630]
- 00000066 f... int type;
[00887634]
- 0089008e .... char
*mem_ptr2;
[00887638]
- a06aa84e N.j. unsigned char
mem_ptr2_bytes[12]; (12 is decimal)
[0088763C]
- 00c3a047 G...
[00887640]
- 00660000 ..f.
[00887644]
- 00000000 ....
[00887648]
- 00000000 ....
[0088764C]
- 00000000 ....
[00887650]
- 00000000 ....
[00887654]
- 54414d43 CMAT
[00887658]
- 00000048 H...
[0012E170]
- ab370fd2 ..7. input string
to be hashed, relies on FEATURE line info
[0012E174]
- 414d4300 .CMA
[0012E178]
- 88054854 TH..
[0012E17C]
- 6a000113 ...j
[0012E180]
- c5876e61 an..
[0012E184]
- 000073d0 .s.. total length
= 0x16 = 22, ends at 73
[0012E188]
- 00000000 ....
[0012EE20]
- 00000004 .... int type;
[0012EE24]
- 52ed15b8 ...R 52xxxxb8,
xxxx is random, different at each run
[0012EE28]
- 75cf780f .x.u 75yyyy0f,
yyyy is random, different at each run
[0012EE2C]
- 7c2adb6a j.*| VENDOR_KEY1
[0012EE30]
- b927f5a9 ..'. VENDOR_KEY2
[0012EE34]
- 9cf311f8 .... VENDOR_KEY3
[0012EE38]
- 0dbf7621 !v.. VENDOR_KEY4
[0012EE3C]
- 00020009 .... FLEXlm
version (here is 9.2)
[0012EE40]
- 39300020 .09
[0012EE44]
- 0000302e .0..
[0012EE48]
- 00000000 ....
lmcrypt.exe:
job = 008C49E8, input = 0012D8D4,
inputlen = 0x16, code = 004D7B48, len = 66D8B337
0x008C49E8 66 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 f...............
0x008C49F8 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 ................
0x008C4A08 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 ................
0x008C4A18 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 ................
0x0012D8D4 d2 0f 37 ab 00 43 4d 41 54 48 05 88 13 01 00
6a Ò.7«.CMATH.....j
0x0012D8E4 61 6e 87 c5 d0 73 00 00 00 00 00 00 00 00 00
00 an.ÅÐs..........
0x0012D8F4 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 ................
0x0012D904 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 ................
0x004D7B48 04 00 00 00 b8 39 c1 52 0f 54 e3 75 6a db 2a
7c ....¸9ÁR.TãujÛ*|
0x004D7B58 a9 f5 27 b9 f8 11 f3 9c 21 76 bf 0d 09 00 02
00 ©õ'¹ø.ó.!v¿.....
0x004D7B68 20 00 30 38 2e 30 00 00 c3 80 f4 83 2c c0 1c
77 .08.0..Ã.ô.,À.w
0x004D7B78 00 00 00 00 01 00 00 00 00 00 00 00 00 00 00
00 ................
Note here we are dealing with non-CRO short keys such as
6D5C01FD71C9, it’s composed of 12 ASCII characters representing 6 hex bytes,
that’s why we see “y[6] = y[7] = 0;” in above code. Except for
VENDORCODE and job structure all other arguments are equal[9],
and in these two structs not all components are important. It’s easy to figure
out that code->data[], code->keys[] and job->mem_ptr2_bytes[] are
those participated in hashing the input string.
Immediately we copy the four vendor keys revealed in the
checkout process to lm_code.h (and confirm that in memory
dump), but we are still short of the encryption seeds and the mysterious
VENDOR_KEY5. Now look closer to the keygen process, it differs from the
checkout in code->data[] and job->mem_ptr2_bytes[] (its job
struct is virtually empty). Why it that?
#include
"lmprikey.h"
#include
"lmclient.h"
#include
"lm_code.h"
#include
"lmseeds.h"
... ...
/* set
site_code.data = {ENCRYPTION_SEED1 ^ VENDOR_KEY5,
ENCRYPTION_SEED2 ^ VENDOR_KEY5}
site_code.keys = {VENDOR_KEY1, VENDOR_KEY2, VENDOR_KEY3, VENDOR_KEY4} */
LM_CODE(site_code,
ENCRYPTION_SEED1, ENCRYPTION_SEED2, VENDOR_KEY1,
VENDOR_KEY2,
VENDOR_KEY3, VENDOR_KEY4, VENDOR_KEY5);
... ...
int main(int argc, char
**argv)
{
... ...
/* set site_code.data =
{ENCRYPTION_SEED1, ENCRYPTION_SEED2} */
LM_CODE_GEN_INIT(&site_code);
if (lc_init((LM_HANDLE *)0, VENDOR_NAME, &site_code,
&lm_job))
{
lc_perror(lm_job,
"lc_init failed");
exit(-1);
}
... ...
/* call chain dofilecrypt() ->
dofpcrypt() -> lc_cryptstr() */
estat |= dofilecrypt(infilename, outfilename, &site_code);
return 0;
}
This is the concise source of lmcrypt.c and it
should explain itself. Notice carefully the two macros LM_CODE and LM_CODE_GEN_INIT defined
in lmclient.h: the former initializes site_code.data to be
encryption seeds xored with VENDOR_KEY5 (in accordance with lmcode.c), but
the latter soon reverses it back to the original encryption seeds. Did I tell
you FLEXlm has lousy coding style?
Anyway raw encryption seeds and zero job->mem_ptr2_bytes are used
in keygen, which is different from checkout. The natural thing to do is copy code->data from the
checkout process to lmseeds.h as encryption seeds 1 & 2
and recompile lmcrypt.exe. But it did not work, for code->data is
clearly random in the form 52xxxxB8 and 75yyyy0F where xxxx and yyyy change at
each run. The same is also true for job->mem_ptr2_bytes. We
conclude that the encryption seeds must be obfuscated and stored in two places,
code->data[] and job->mem_ptr2_bytes[] (they
two should be closely coupled somehow), in vendor software because it is
shipped to end users and raw seeds need to be protected. In contrast, lmcrypt.exe is only
available to vendors so encryption seeds can appear in plain form.
[9] Beware
that W32dasm and Visual Studio show memory differently due to little
endianness.
