On Software Reverse Engineering
Uncover Keys and Seeds
From a pragmatic point of view we may stop now, for we can
use the target freely as licensed copies. However, the ultimate goal of software
cracking is to reverse engineer all relevant algorithms and recreate them as if
we were the original authors. In our case, that is minimally to uncover VNI’s 5
vendor keys and 3 random seeds. Needless to say, this requires more careful
perusing of the FLEXlm SDK source code, so we briefly review how it is
organized. The list below shows the folders that contain the most crucial
files.
src
source files for lmgr.lib or lmgr9a.dll
app
source files for lmgras.lib
server
source files for lmgrs.lib
master
source files for lmgrd.exe
utils
source files for utilities
machind
machine independent files (source & header)
i86_n3
final binary files for x86 platform
h
header files
certicom
library and header files from Certicom Corp.
ulite
ultralite version of FLEXlm
Moreover, files and functions are assigned to different
prefixes according on their roles. For instance, files in directories app, server and master all
start with ls_ while in utils it’s the
lm prefix.
l_
license, for internal functions
lc_
license client, for client APIs
ls_
license server, for server APIs
lm_
license manager, for utilities and general
stuff
We now go back to documentations [2] and [3] for
directions on how vendors use FLEXlm SDK. Basically vendors need to integrate much
of the FLEXlm binaries with their own products and create a number of utilities
– some for internal use, some for end user distribution. There are very close
ties among these vendor-generated files and here is the dependency table for
them:
File
Generated by
#include
lmrand1.exe
lmrand1.c + lmgr.lib
lmcode.c, lsrvend.c
lmrand1.exe + lsvendor.c
lm_code.h
lmappfil.c, lmkeyfil.c
lmrand1.exe
lmnewgen.exe
lmnewgen.c + lmcode.c + lmgr.lib
lm_code.h
lm_new.c
lmnewgen.exe
seeds & pubkey
lmseeds.h
lmnewgen.exe
lmprikey.h, lmpubkey.h
lmnewgen.exe
lmcrypt.exe
lmcrypt.c + lmgr.lib
lm_code.h, lmseeds.h, lmprikey.h
makekey.exe
makekey.c + lmgr.lib
lm_code.h, lmseeds.h, lmprikey.h
vendor application
vendor code + lm_new.obj + lmgr.lib
+
libsb.lib + libcrvs.lib
seeds & pubkey
vendor daemon
lsvendor.obj + lm_new.obj + lmgr.lib +
lmgras.lib + lmgrs.lib + libsb.lib +
libcrvs.lib
lm_code.h, lsserver.h, seeds & pubkey
Notice that not all above files are important to us at
this time. lmappfil.c and lmkeyfil.c are
vendor specific filters that are additional security measures but normally left
unused. Similarly lmprikey.h and lmpubkey.h are for
CRO keys only, which we can safely ignore for our case. Vendors can also
customize their daemons by editing lsrvend.c or lsvendor.c but
rarely do people do that.
One thing worth talking about is the Certicom product,
namely libsb.lib and libcrvs.lib (I
believe sb stands for “soft bus” and crvs stands
for “elliptic curves”). They cover almost the entire encryption field from SHA
to DSA, and the APIs employed by FLEXlm are mostly ECC (Elliptic Curve
Cryptography) and RNG (Random Number Generation). ECC is used for
public/private key pairs in CRO, which is turned off in our target; but RNG is
used for vendor seeds transformation, which we cannot neglect.
As stated earlier, vendor selects 3 random seeds
(LM_SEED1, LM_SEED2, LM_SEED3), but they are not used directly. Instead lmnewgen.c!main()®l_prikey.c!l_genrand()®libsb.lib!sb_rngFIPS186Session() produces
4 new seeds[8] (ENCRYPTION_SEED1,
… ENCRYPTION_SEED4), which are output to lmseeds.h, based
on them. Seed 3 and 4 are exclusively for public/private key pairs, thus we
only need to worry about seed 1 and 2.
Although FLEXlm has its own RNG in l_rand.h and lm_rand3.c, here it
uses the algorithm specified in DSS standard (c.f. [8]). This algorithm is
deterministic (without reading time, register value, etc.), so ENCRYTION_SEED
are fixed for given LM_SEED. It is also a one-way function, meaning that it’s
practically impossible to solve LM_SEED from known ENCRYPTION_SEED. Since it is
ENCRYPTION_SEED that are built into the vendor releases and LM_SEED appear
nowhere else than l_genrand(), the best we can do is
recovering the former. However, for exactly the same reason, we deem that
equally good as recovering LM_SEED.
The inner structure of FLEXlm is essentially C/S model,
where integrated vendor application is the client and vendor daemon (vni.exe here) is
the server. Note vendor daemon is different from the FLEXlm license manager lmgrd.exe. The
latter simply redirects client requests to corresponding vendor daemons, which
do the real work. Another factor is the license file format. FLEXlm offers rich
licensing options and an important one is counted vs. uncounted. Uncounted license
has no restrictions on number of checkouts and does not need vendor daemon –
the validation takes place inside the vendor software that acts as client and
server simultaneously. On the other hand, counted license requires vendor
daemon and lmgrd.exe running along with vendor
software to serve the client. Counted license usually floats on a network
(floating license must be counted), which is a common scene at large companies:
a central node running the servers and all workstations connect to it for license
checkout/checkin. Hence we can infer that vendor daemon’s task is mainly to
count, manage and coordinate the use of limited licenses.
Look at our license, it is “permanent uncounted
HOSTID=ANY”, showing that it has no conditions whatsoever. This is definitely
every hacker’s dream (having no daemon process also makes tracing easier). In
fact the “SERVER” and “DAEMON” lines in the license are unneeded, two “FEATURE”
lines alone are enough to guarantee cmath.exe running
unlimitedly. We will come to the topic of license types later.
Now we have cmath.exe as both
client and server with vendor keys and encryption seeds built in (of course
shadowed and hidden). The good news is that CRO is disabled so there is only
one set of keys. If we can dig them out, then we can make any license file by lmcrypt.exe or makekey.exe the same
way as Visual Numerics. Before we set out to do that, we summarize some key
data in key files.
lm_code.h: /* \machind\lm_code.h and \h\lm_code.h must be identical */
VENDOR_KEY1, VENDOR_KEY2,
VENDOR_KEY3, VENDOR_KEY4, VENDOR_KEY5
LM_SEED1,
LM_SEED2, LM_SEED3
lmcode.c:
#include
"lm_code.h"
#include
"lmclient.h"
VENDORCODE
vendorkeys[] = { /* bad name, should use
vendorcodes[] instead */
{ VENDORCODE_7,
ENCRYPTION_SEED1 ^ VENDOR_KEY5,
ENCRYPTION_SEED2 ^ VENDOR_KEY5,
VENDOR_KEY1, VENDOR_KEY2, VENDOR_KEY3,
VENDOR_KEY4,
FLEXLM_VERSION, FLEXLM_REVISION,
FLEXLM_PATCH,
LM_BEHAVIOR_CURRENT, {CRO_KEY1, CRO_KEY2},
LM_STRENGTH, LM_SIGN_LEVEL, 0
},
};
lmseeds.h:
ENCRYPTION_SEED1,
ENCRYPTION_SEED2, ENCRYPTION_SEED3, ENCRYPTION_SEED4
lm_new.c:
x = 0x3d73db2e; /* random number key5_uniqx generated in lmnewgen.c */
VENDORCODE.data /* obfuscated by key5() */
VENDORCODE.keys /* obfuscated by l_xorname() */
l_getattr.c:
#define
VENDORMAGIC_V7 0x08BC0EF8 /* insignificant, just for
xor canceling */
lmclient.h:
#define
L_NEW_JOB l_n36_buf
l_privat.h:
#define
L_UNIQ_KEY5_FUNC l_n36_buff
#define
L_SET_KEY5_FUNC l_x77_buf
#define
SEEDS_XOR mem_ptr2_bytes /* where seeds are hidden in job */
#define
SEEDS_XOR_NUM 12
Besides, there are three numbers that are hard-coded into
the binaries (just like VENDORMAGIC_V7) but seems never used (at
least in the case of cmath.exe). Maybe they are solely for
certain versions or license types, we’ll see that when we visit those
functions.
lm_ckout.c!l_sg():
x = 0x6f7330b8; /* v8.x */
lmnewgen.c!VKEY5():
x = 0x6f7330b8; /* v8.x */
l_key.c!l_zinit():
z = crokey_flag
? 0x62586954 : 0x72346B53; /* v9.x */
[8] Three
more seeds are generated for CRO.
