SimsPolska
New Member
ok
TSO-SE Discussion Thread
- Thread starter Blayer98
- Start date
francot514
Well-Known Member
Just for knowledge niotso documentation is updated??? There are not so much references about the dat files...
I resumed working on TSO-SE protocol research again about a month ago.
Since December 2014, the most important problem we were facing was determining the DataServiceWrapperPDU response format; and just now we have solved it.
Note that this post is longer than usual, so if you just want to see the screenshots, that's fine, just scroll down to the bottom.
DataServiceWrapperPDU is similar to DBRequestWrapperPDU. It appears that the only distinction between the two is that DBRequestWrapperPDU is used for transactions with the server's persistent database (such as getting or setting the lot's name, buying or selling objects, etc.), while DataServiceWrapperPDU tends to be used to synchronize certain volatile game variables (such as which lots are online, which clothing each person is currently changed into, etc.).
The client sends a number of DataServiceWrapperPDU requests after it receives HostOnlinePDU:
Thus, in order to do almost anything in the map view, including joining a lot (possibly by enabling the "Where is my property?" button in a Sim's Sim Page), we really have to learn the DataServiceWrapperPDU response format.
Back in the summer/fall of 2013 when we were trying to determine the LoadAvatarByID response format, we noticed that the file TSOData_datadefinition.dat contained strings related to the avatar: Avatar_Age, Avatar_Appearance, etc. Thus, I mapped the structure of TSOData_datadefinition.dat and put it on the wiki in October 2013.
The file defines four lists:
Note that EA-Land's RTTI for TSODataServiceClientD.dll mimics the contents of this file quite closely.
The AvatarSkills second-level struct type helped with some of the fields in LoadAvatarByID; it was noted that 12 shorts appear in succession in the LoadAvatarByID format, which appeared to correspond to the AvatarSkills struct in TSOData_datadefinition.dat which also consists of exactly 12 shorts in succession.
In June 2014, we produced a valid LoadAvatarByID response packet, which was sufficiently valid to get into the map view. It's here that we learned about DataServiceWrapperPDU and its appearance in many places in the map view.
In December 2014, Darren pointed out that the then-unknown 4-byte field in DataServiceWrapperPDU (which we're now calling the Proxy ID) corresponds to an ID defined in the Strings section (the fourth list) at the bottom of TSOData_datadefinition.dat. Suddenly we knew the names of all of these DataServiceWrapperPDUs that the client was sending to us, and further, their formats were all given (rough) descriptions by TSOData_datadefinition.dat. Thus, thanks to TSOData_datadefinition.dat, if we can figure out the response format for any DataServiceWrapperPDU, then we know it for all of them.
So at this point, it was known that you respond to a DBRequestWrapperPDU request with another DBRequestWrapperPDU. When the client sends a DBRequestWrapperPDU request to the server, it uses the 0x125194E5 (cTSONetMessageStandard) body and, inside that, it attaches a Request/Response Body of type <x>_Request (for some <x>), according to the RTTI in EA-Land's version of DBAppServiceClientD.dll. The server is simply supposed to respond with another DBRequestWrapperPDU with the <x>_Response body attached in place of the _Request body.
With DataServiceWrapperPDU, the client also uses cTSONetMessageStandard. Thus, it's a reasonable guess that you also respond to a DataServiceWrapperPDU with another DataServiceWrapperPDU.
With DataServiceWrapperPDU, however, the client does *not* attach a body inside the cTSONetMessageStandard in its request (Bit 5 in Flags is unset). Thus, we're unable to take the client's request, read the 4-byte Request Type field, search for it in the game dlls, and find the _Response version of that field, like we were able to for DBRequestWrapperPDU.
So we have to make another guess. In DBRequestWrapperPDU, all of the Request/Response Body Type clsids are registered by DBAppServiceClientD.dll, and the v-table addresses for the classes associated with these clsids therefore all belong to DBAppServiceClientD.dll. For DataServiceWrapperPDU, it is probably reasonable to assume that the Response clsids belong to TSODataServiceClientD.dll.
In New & Improved trial version, the code that looks at the Request/Response Type field is located at TSONetServiceSimClientD_base+0x4f2b9. The game reads this field and creates an object using the cGZCOM "create new object from (clsid, iid) pair" function (TSOClient_base+0x7501) using clsid=<value of this field>, iid=0xE4FDA3D4, and it then calls the "read body" function located at v-table offset +0x10 in this object.
We added code in the dll to brute-force this function call for all clsids, from 0x00000000 through 0xFFFFFFFF, inside OllyDbg with iid=0xE4FDA3D4 and log which clsids cause the function to return success. Unfortunately, there are 283 clsids in the resulting list, which is quite large. However, 175 of them belong to DBRequestWrapperPDU. If we assume the clsids we want do not belong to that dll, then that leaves us with 108 clsids.
Further, only 4 of the clsids belong to TSODataServiceClientD.dll, which is on the small side. EA-Land's RTTI for TSODataServiceClientD.dll provides the names for all 4 clsids:
In addition, there are 44 total clsids defined in TSODataServiceClientD.dll, regardless of the IIDs they accept. EA-Land's RTTI provides names for all of them (names unmangled by __unDName):
Here, I was slightly disappointed. As mentioned earlier, EA-Land's RTTI for TSODataServiceClientD.dll "mimics" TSOData_datadefinition.dat and shows that there are a large number of classes such as cTSOProxy_MyAvatar, cTSOProperty_AvatarAppearance, cTSODataHandler_CityVec, etc. Thus, if we could find the clsids of them, and they were accepted for cTSONetMessageStandard's Request/Response Body Type field, then we would (essentially) know for certain that we're supposed to use those clsids for the response, just like we do for DBRequestWrapperPDU.
However, none of those cTSOProxy, cTSOProperty, or cTSODataHandler classes can be directly created from a cGZCOM clsid. Instead, it appeared that there was possibly one clsid that you use for all DataServiceWrapperPDUs responses. It might be that we specify the "string ID" of the struct type, and then the game knows what fields to read next based on that ID; or it might be that we construct the response manually, by specifying the number of fields that follow, and then for each field, we specify the ID of the field and the value of that field.
Since December 2014, the most important problem we were facing was determining the DataServiceWrapperPDU response format; and just now we have solved it.
Note that this post is longer than usual, so if you just want to see the screenshots, that's fine, just scroll down to the bottom.
DataServiceWrapperPDU is similar to DBRequestWrapperPDU. It appears that the only distinction between the two is that DBRequestWrapperPDU is used for transactions with the server's persistent database (such as getting or setting the lot's name, buying or selling objects, etc.), while DataServiceWrapperPDU tends to be used to synchronize certain volatile game variables (such as which lots are online, which clothing each person is currently changed into, etc.).
The client sends a number of DataServiceWrapperPDU requests after it receives HostOnlinePDU:
- MyAvatar
- FriendshipWeb_Avatar
- Neighbor_Avatar
- CurrentCity
- CurrentCityTopTenNeighborhoods
- SimPage_Main
Thus, in order to do almost anything in the map view, including joining a lot (possibly by enabling the "Where is my property?" button in a Sim's Sim Page), we really have to learn the DataServiceWrapperPDU response format.
Back in the summer/fall of 2013 when we were trying to determine the LoadAvatarByID response format, we noticed that the file TSOData_datadefinition.dat contained strings related to the avatar: Avatar_Age, Avatar_Appearance, etc. Thus, I mapped the structure of TSOData_datadefinition.dat and put it on the wiki in October 2013.
The file defines four lists:
- The list of all second-level struct definitions, which may include fields of type "Uint32", or "string", or "List of Uint32s", or "Associative array of Uint32->Uint32 pairs", etc.
- The list of all top-level struct definitions, which may include fields whose type is not only an ordinary variable but also a second-level struct, or a list of second-level structs, etc.
- The list of all instances of top-level structs
- The list providing the names to all of the structs, struct fields, and struct instances defined above in the file
Note that EA-Land's RTTI for TSODataServiceClientD.dll mimics the contents of this file quite closely.
The AvatarSkills second-level struct type helped with some of the fields in LoadAvatarByID; it was noted that 12 shorts appear in succession in the LoadAvatarByID format, which appeared to correspond to the AvatarSkills struct in TSOData_datadefinition.dat which also consists of exactly 12 shorts in succession.
In June 2014, we produced a valid LoadAvatarByID response packet, which was sufficiently valid to get into the map view. It's here that we learned about DataServiceWrapperPDU and its appearance in many places in the map view.
In December 2014, Darren pointed out that the then-unknown 4-byte field in DataServiceWrapperPDU (which we're now calling the Proxy ID) corresponds to an ID defined in the Strings section (the fourth list) at the bottom of TSOData_datadefinition.dat. Suddenly we knew the names of all of these DataServiceWrapperPDUs that the client was sending to us, and further, their formats were all given (rough) descriptions by TSOData_datadefinition.dat. Thus, thanks to TSOData_datadefinition.dat, if we can figure out the response format for any DataServiceWrapperPDU, then we know it for all of them.
So at this point, it was known that you respond to a DBRequestWrapperPDU request with another DBRequestWrapperPDU. When the client sends a DBRequestWrapperPDU request to the server, it uses the 0x125194E5 (cTSONetMessageStandard) body and, inside that, it attaches a Request/Response Body of type <x>_Request (for some <x>), according to the RTTI in EA-Land's version of DBAppServiceClientD.dll. The server is simply supposed to respond with another DBRequestWrapperPDU with the <x>_Response body attached in place of the _Request body.
With DataServiceWrapperPDU, the client also uses cTSONetMessageStandard. Thus, it's a reasonable guess that you also respond to a DataServiceWrapperPDU with another DataServiceWrapperPDU.
With DataServiceWrapperPDU, however, the client does *not* attach a body inside the cTSONetMessageStandard in its request (Bit 5 in Flags is unset). Thus, we're unable to take the client's request, read the 4-byte Request Type field, search for it in the game dlls, and find the _Response version of that field, like we were able to for DBRequestWrapperPDU.
So we have to make another guess. In DBRequestWrapperPDU, all of the Request/Response Body Type clsids are registered by DBAppServiceClientD.dll, and the v-table addresses for the classes associated with these clsids therefore all belong to DBAppServiceClientD.dll. For DataServiceWrapperPDU, it is probably reasonable to assume that the Response clsids belong to TSODataServiceClientD.dll.
In New & Improved trial version, the code that looks at the Request/Response Type field is located at TSONetServiceSimClientD_base+0x4f2b9. The game reads this field and creates an object using the cGZCOM "create new object from (clsid, iid) pair" function (TSOClient_base+0x7501) using clsid=<value of this field>, iid=0xE4FDA3D4, and it then calls the "read body" function located at v-table offset +0x10 in this object.
We added code in the dll to brute-force this function call for all clsids, from 0x00000000 through 0xFFFFFFFF, inside OllyDbg with iid=0xE4FDA3D4 and log which clsids cause the function to return success. Unfortunately, there are 283 clsids in the resulting list, which is quite large. However, 175 of them belong to DBRequestWrapperPDU. If we assume the clsids we want do not belong to that dll, then that leaves us with 108 clsids.
Further, only 4 of the clsids belong to TSODataServiceClientD.dll, which is on the small side. EA-Land's RTTI for TSODataServiceClientD.dll provides the names for all 4 clsids:
Code:
0x09736027: cTSOTopicUpdateMessage
0x0A2C6585: cTSODataTransportBuffer
0x2A404946: cTSOTopicUpdateErrorMessage
0x496A78BC: cTSODataDefinitionContainerIn addition, there are 44 total clsids defined in TSODataServiceClientD.dll, regardless of the IIDs they accept. EA-Land's RTTI provides names for all of them (names unmangled by __unDName):
Code:
0x09736027: cTSOTopicUpdateMessage
0x097608AB: cTSOValueVector<unsigned char>
0x097608AF: cTSOValueVector<signed char>
0x097608B3: cTSOValueVector<unsigned short>
0x097608B6: cTSOValueVector<short>
0x0A2C6585: cTSODataTransportBuffer
0x29739B14: cTSOTopic
0x2A404946: cTSOTopicUpdateErrorMessage
0x496A78BC: cTSODataDefinitionContainer
0x696D1183: cTSOValue<bool>
0x696D1189: cTSOValue<unsigned long>
0x69D3E3DB: cTSOValue<unsigned __int64>
0x896D1196: cTSOValue<long>
0x896D11A2: cTSOValueMap<class cRZAutoRefCount<class cITSOProperty> >
0x896D1688: cTSOValue<class cRZAutoRefCount<class cIGZString> >
0x89738492: cTSOValueBVector
0x89738496: cTSOValueVector<unsigned long>
0x8973849A: cTSOValueVector<long>
0x8973849E: cTSOValueVector<class cRZAutoRefCount<class cIGZString> >
0x89739A79: cTSOProperty
0x89D3E3EF: cTSOValue<__int64>
0x89D3E40E: cTSOValueMap<unsigned __int64>
0xA96E38A0: cTSOValueMap<unsigned long>
0xA96E38A8: cTSOValueMap<long>
0xA96E38AC: cTSOValueMap<class cRZAutoRefCount<class cIGZString> >
0xA96E7E5B: cTSOValue<class cRZAutoRefCount<class cITSOProperty> >
0xA97353EE: cTSODataService
0xA97384A3: cTSOValueVector<class cRZAutoRefCount<class cITSOProperty> >
0xA975FA6E: cTSODataServiceClient
0xA99AF3A8: cTSOValueMap<class cRZAutoRefCount<class cIGZUnknown> >
0xA99AF3AC: cTSOValueVector<class cRZAutoRefCount<class cIGZUnknown> >
0xA99AF3B7: cTSOValue<class cRZAutoRefCount<class cIGZUnknown> >
0xA9D3E412: cTSOValueMap<__int64>
0xA9D3E428: cTSOValueVector<unsigned __int64>
0xA9D3E42D: cTSOValueVector<__int64>
0xC976087C: cTSOValue<unsigned char>
0xC97757F5: cTSOValueMap<bool>
0xE976088A: cTSOValue<signed char>
0xE9760891: cTSOValue<unsigned short>
0xE9760897: cTSOValue<short>
0xE976089F: cTSOValueMap<unsigned char>
0xE97608A2: cTSOValueMap<signed char>
0xE97608A5: cTSOValueMap<unsigned short>
0xE97608A8: cTSOValueMap<short>Here, I was slightly disappointed. As mentioned earlier, EA-Land's RTTI for TSODataServiceClientD.dll "mimics" TSOData_datadefinition.dat and shows that there are a large number of classes such as cTSOProxy_MyAvatar, cTSOProperty_AvatarAppearance, cTSODataHandler_CityVec, etc. Thus, if we could find the clsids of them, and they were accepted for cTSONetMessageStandard's Request/Response Body Type field, then we would (essentially) know for certain that we're supposed to use those clsids for the response, just like we do for DBRequestWrapperPDU.
However, none of those cTSOProxy, cTSOProperty, or cTSODataHandler classes can be directly created from a cGZCOM clsid. Instead, it appeared that there was possibly one clsid that you use for all DataServiceWrapperPDUs responses. It might be that we specify the "string ID" of the struct type, and then the game knows what fields to read next based on that ID; or it might be that we construct the response manually, by specifying the number of fields that follow, and then for each field, we specify the ID of the field and the value of that field.
If you do a ctrl+F for the constant 0x696D1189 in TSODataServiceClientD.dll, you can see that it's used as arg2 for a lot of function calls where arg1 is always set to the string ID of a field of type UInt32 defined in TSOData_datadefinition.dat. In fact, if you do a ctrl+F for the string ID of any field, you will find the clsid, from the above list, associated with the *type* of that field (Uint32, List of Uint32s, etc.). Thus, for our second conjecture, in case we are asked for the type of each field, we would probably specify either the string ID of that type or the clsid associated with it from the above list.
Since there are only 4 clsids, I determined the structure of each of them. They are as follows:
The game actually uses cTSODataDefinitionContainer in order to read TSOData_datadefinition.dat (at TSODataServiceClientD_base+0x15fbd), and the cTSODataDefinitionContainer body exactly matches that of TSOData_datadefinition.dat (4 lists and all). So it would be extremely weird if the game also uses this format for the DataServiceWrapperPDU body, but I suppose it's possible. However, the other clsids are more likely to be interesting.
Additionally, it seems there's no reason to use cTSOTopicUpdateErrorMessage when you can use cTSOTopicUpdateMessage instead.
So that essentially leaves us with two clsids: cTSOTopicUpdateMessage and cTSODataTransportBuffer. However, cTSOTopicUpdateMessage makes no sense: there's a list of dwords and then a cTSOValue which could provide the *value* of a *single* field, whereas we should be given the option to specify a *list* of fields and their names, not just their values.
Further, cTSODataTransportBuffer could be plausible, but also weird: maybe the game uses another binary format for specifying the DataServiceWrapperPDU body, and you send the binary data inside a cTSODataTransportBuffer.
So I searched through the game dlls further to see what this "Topic", "TSOData", "Property", and "Proxy" lexicon mean.
Probably the most helpful debugging string was the one located at TSOServiceClientD_base+0x20271:
City_ReservedLotInfo is a field inside the City struct, and it appears inside the CurrentCity variable (and in no other variables). This showed that cTSOTopicUpdateMessage is related to the "updating" of a struct field.
But I still did not really know what to take from this information. We were looking at possibly 2 clsids sifted from possibly 175 clsids sifted from possibly 283 clsids, under the assumption that we respond to a DataServiceWrapperPDU with a DataServiceWrapperPDU with the response body attached in the Request/Response Body field of cTSONetMessageStandard.
It would far more useful to get the client to respond to a request made by the *server*, just to confirm any one of these assumptions.
So we took a DataServiceWrapperPDU request made by the client (the MyAvatar request) and sent it back to the client verbatim; unfortunately, the client does not respond to the packet when you do this. But it turns out that there is also a "ServerMyAvatar" request, so we changed the string ID to ServerMyAvatar and resent the packet. But the client still did not respond.
There are a number of potentially relevant message IDs that we could try from the Regulators section of the wiki. We gathered together the following list:
We had to remove 0xFBF7001B (kConnectionFailed) since it caused the client to disconnect from the server.
We tried the packet for all 16 message IDs, but the client still did not respond. We then put all 16 versions of the DataServiceWrapperPDU into a single Aries packet, and we created a Perl script to create copies of this packet, with each copy having a different string ID, for all struct instance string IDs in the game, from the third list in TSOData_datadefinition.dat (there are 94 of them in New & Improved).
Unfortunately, it remained the same: the client did not respond to any of these requests by the server.
But one month ago, now, Blayer pointed out that TSO Play Test Rating Pending version in -NoNetwork mode sets the lot price to $2000 when you hover the cursor over a lot, and so it probably responds to the MapView_RollOverInfo_Lot or MapView_RollOverInfo_Lot_Price requests. Play Test Rating Pending is the only version of the game that shows a lot price in -NoNetwork mode; the later versions that support -NoNetwork mode, from Play Test Rated T onwards, all show "Retrieving..." like we were currently seeing with TSO-SE since we could not respond to the _Lot or _Lot_Price DataServiceWrapperPDU requests.
So I did a ctrl+F for the 4-byte constant 0x000007d0 (2000 in decimal) in the game dlls in Play Test Rating Pending, and we found a function (TSODataServiceClientD_base+0xeb55) which references the constants 0x000007d0, 0xEC8D5C5C (Lot_Price), 0x696D1189 (cTSOValue<unsigned long>), and 0x0C0CB1E6 (MapView_RollOverInfo_Lot). By placing a breakpoint on this function, we found that this function gets reached whenever the user hovers over a lot, and further, by changing the 4-byte constant from 2000 to 2001, the lot price ingame subsequently changed from $2000 to $2001, which confirmed that this code was indeed responding to MapView_RollOverInfo_Lot.
This function obtains a cTSOProxy object (at TSODataServiceClientD_base+0xeb71), calls AddRef on the cTSOProxy object (+0xeb81), creates a cTSODataHandler object and associates it with the cTSOProxy object passed in arg3 (+0xebad), passes the cTSODataHandler object in ecx and 2000 value in arg1 to an interesting function (+0xebc4), and then releases the cTSODataHandler object (+0xebd4) and cTSOProxy object (+0xebe3).
That "interesting function" (+0x3ac67) calls another interesting function, TSODataServiceClientD_base+0x3ad82 (+0x3aca6). This new function is interesting because, after placing breakpoints on the cTSOTopicUpdateMessage constructors (+0x1fffb and +0x25b0a) and cTSODataTransportBuffer constructors (+0x26091 and +0x260c1), we found that this function directly calls the first cTSOTopicUpdateMessage constructor (+0x25b0a) and subsequently fills out various fields in the object, one of which is the 4-byte constant 0xA97360C5, which is a message ID recognized by DataServiceRegulator and which the Pre-Alpha constants table calls GZMSGID_cITSOApprovedTopicUpdate. Further, if we place breakpoints on TSODataServiceClientD.dll's "run guard function" (+0x14e0f and +0x1f6b1) and "run process function" (+0x14e37 and +0x1f6c0) functions, we see that the 0xA97360C5 message is being posted, with the cTSOTopicUpdateMessage object being passed as arg1 to the guard and process functions.
We created a DataServiceWrapperPDU with the string ID set to SimPage_Main, and inside the cTSONetMessageStandard, we set the message ID to 0xA97360C5, set the Parameter to 0x00000539 (our avatar ID 1337, the same as what the client sends us in the request), and set the Request/Response Body to cTSOTopicUpdateMessage, and we set the fields of the cTSOTopicUpdateMessage to the temporary values 0x11121314, 0x15161718, etc. We opened TSO New & Improved Trial, sent the packet to the client, and traced what the client did with the fields.
In TSO New & Improved Trial, the cTSOTopicUpdateMessage body handler is located at TSODataServiceClientD_base+0x2ce56. The first 3 dwords are (in order) stored in the cTSOTopicUpdateMessage object (passed as thisptr in ecx) at thisptr+0x18, +0x1c, and +0x20 respectively. The "count" field is read into temporary storage on the stack and is not saved in the cTSOTopicUpdateMessage object. The <count> dwords are stored in a resizable vector pointed to by the dword at thisptr+0x08. (In this vector object, the first dword points to the contiguous array data.) The cTSOValue object is allocated dynamically, and its address is stored in thisptr+0x10. The "pascalvlc" (Pascal-style string with variable-length coding for the length), as I'm calling it, string object is allocated dynamically, and its address is stored at thisptr+0x24. I placed memory breakpoints on all of these fields, and also the array data and string data, and pressed F9 and noted which code looked at this data.
Since there are only 4 clsids, I determined the structure of each of them. They are as follows:
Code:
0x09736027: cTSOTopicUpdateMessage
* dword unknown
* dword unknown
* dword unknown
* dword unknown1_count
* unknown1s - for each unknown1:
* dword
* dword Body clsid (iid=0xE96D0B02: the cTSOValue clsids only)
* Body
* pascalvlc unknown
0x0A2C6585: cTSODataTransportBuffer
* dword Body size
* Body
0x2A404946: cTSOTopicUpdateErrorMessage
* dword unknown
* dword unknown
* dword unknown1 count
* unknown1s - for each unknown1:
* dword
* dword Body clsid (iid=0xE96D0B02: the cTSOValue clsids only)
* Body
* pascalvlc unknown
0x496A78BC: cTSODataDefinitionContainer
<body matches that of TSOData_datadefinition.dat; see TSODataServiceClientD_base+0x6c959>The game actually uses cTSODataDefinitionContainer in order to read TSOData_datadefinition.dat (at TSODataServiceClientD_base+0x15fbd), and the cTSODataDefinitionContainer body exactly matches that of TSOData_datadefinition.dat (4 lists and all). So it would be extremely weird if the game also uses this format for the DataServiceWrapperPDU body, but I suppose it's possible. However, the other clsids are more likely to be interesting.
Additionally, it seems there's no reason to use cTSOTopicUpdateErrorMessage when you can use cTSOTopicUpdateMessage instead.
So that essentially leaves us with two clsids: cTSOTopicUpdateMessage and cTSODataTransportBuffer. However, cTSOTopicUpdateMessage makes no sense: there's a list of dwords and then a cTSOValue which could provide the *value* of a *single* field, whereas we should be given the option to specify a *list* of fields and their names, not just their values.
Further, cTSODataTransportBuffer could be plausible, but also weird: maybe the game uses another binary format for specifying the DataServiceWrapperPDU body, and you send the binary data inside a cTSODataTransportBuffer.
So I searched through the game dlls further to see what this "Topic", "TSOData", "Property", and "Proxy" lexicon mean.
Probably the most helpful debugging string was the one located at TSOServiceClientD_base+0x20271:
Code:
CPU Disasm
Address Hex dump Command Comments
030B0270 53 PUSH EBX
030B0271 68 A02C1203 PUSH OFFSET 03122CA0 ; ASCII "cTSOMapViewRegulator::HandleApprovedTopicUpdate, City_ReservedLotInfo"
030B0276 E8 45730400 CALL 030F75C0City_ReservedLotInfo is a field inside the City struct, and it appears inside the CurrentCity variable (and in no other variables). This showed that cTSOTopicUpdateMessage is related to the "updating" of a struct field.
But I still did not really know what to take from this information. We were looking at possibly 2 clsids sifted from possibly 175 clsids sifted from possibly 283 clsids, under the assumption that we respond to a DataServiceWrapperPDU with a DataServiceWrapperPDU with the response body attached in the Request/Response Body field of cTSONetMessageStandard.
It would far more useful to get the client to respond to a request made by the *server*, just to confirm any one of these assumptions.
So we took a DataServiceWrapperPDU request made by the client (the MyAvatar request) and sent it back to the client verbatim; unfortunately, the client does not respond to the packet when you do this. But it turns out that there is also a "ServerMyAvatar" request, so we changed the string ID to ServerMyAvatar and resent the packet. But the client still did not respond.
There are a number of potentially relevant message IDs that we could try from the Regulators section of the wiki. We gathered together the following list:
Code:
DataServiceRegulator: 16 entries at TSODataServiceClientD_base+0x9d6d0
0xA97360C5
0x8A2726E0
0x89FF5197
0xEA6C8D8F
0x89FF519C
0x7EA33D4D
0x3BF7000A
0xEA6E1298
0xFBF7001B
InterestRegulator: 2 entries at TSODataServiceClientD_base+0x9e1c8
0x09C83484
0x29C8348F
NetMessagePDURegulator: 2 entries at TSONetServiceSimClientD_base+0xa8e28
0x4ABA95B3
0xDBF301A9
(No regulator, but they are sent by the client and they appear in TSOServiceClientD.dll):
0x3BF82D4E
0x897360BF
0x49736C62
0xC9736CE8We had to remove 0xFBF7001B (kConnectionFailed) since it caused the client to disconnect from the server.
We tried the packet for all 16 message IDs, but the client still did not respond. We then put all 16 versions of the DataServiceWrapperPDU into a single Aries packet, and we created a Perl script to create copies of this packet, with each copy having a different string ID, for all struct instance string IDs in the game, from the third list in TSOData_datadefinition.dat (there are 94 of them in New & Improved).
Unfortunately, it remained the same: the client did not respond to any of these requests by the server.
But one month ago, now, Blayer pointed out that TSO Play Test Rating Pending version in -NoNetwork mode sets the lot price to $2000 when you hover the cursor over a lot, and so it probably responds to the MapView_RollOverInfo_Lot or MapView_RollOverInfo_Lot_Price requests. Play Test Rating Pending is the only version of the game that shows a lot price in -NoNetwork mode; the later versions that support -NoNetwork mode, from Play Test Rated T onwards, all show "Retrieving..." like we were currently seeing with TSO-SE since we could not respond to the _Lot or _Lot_Price DataServiceWrapperPDU requests.
So I did a ctrl+F for the 4-byte constant 0x000007d0 (2000 in decimal) in the game dlls in Play Test Rating Pending, and we found a function (TSODataServiceClientD_base+0xeb55) which references the constants 0x000007d0, 0xEC8D5C5C (Lot_Price), 0x696D1189 (cTSOValue<unsigned long>), and 0x0C0CB1E6 (MapView_RollOverInfo_Lot). By placing a breakpoint on this function, we found that this function gets reached whenever the user hovers over a lot, and further, by changing the 4-byte constant from 2000 to 2001, the lot price ingame subsequently changed from $2000 to $2001, which confirmed that this code was indeed responding to MapView_RollOverInfo_Lot.
This function obtains a cTSOProxy object (at TSODataServiceClientD_base+0xeb71), calls AddRef on the cTSOProxy object (+0xeb81), creates a cTSODataHandler object and associates it with the cTSOProxy object passed in arg3 (+0xebad), passes the cTSODataHandler object in ecx and 2000 value in arg1 to an interesting function (+0xebc4), and then releases the cTSODataHandler object (+0xebd4) and cTSOProxy object (+0xebe3).
That "interesting function" (+0x3ac67) calls another interesting function, TSODataServiceClientD_base+0x3ad82 (+0x3aca6). This new function is interesting because, after placing breakpoints on the cTSOTopicUpdateMessage constructors (+0x1fffb and +0x25b0a) and cTSODataTransportBuffer constructors (+0x26091 and +0x260c1), we found that this function directly calls the first cTSOTopicUpdateMessage constructor (+0x25b0a) and subsequently fills out various fields in the object, one of which is the 4-byte constant 0xA97360C5, which is a message ID recognized by DataServiceRegulator and which the Pre-Alpha constants table calls GZMSGID_cITSOApprovedTopicUpdate. Further, if we place breakpoints on TSODataServiceClientD.dll's "run guard function" (+0x14e0f and +0x1f6b1) and "run process function" (+0x14e37 and +0x1f6c0) functions, we see that the 0xA97360C5 message is being posted, with the cTSOTopicUpdateMessage object being passed as arg1 to the guard and process functions.
We created a DataServiceWrapperPDU with the string ID set to SimPage_Main, and inside the cTSONetMessageStandard, we set the message ID to 0xA97360C5, set the Parameter to 0x00000539 (our avatar ID 1337, the same as what the client sends us in the request), and set the Request/Response Body to cTSOTopicUpdateMessage, and we set the fields of the cTSOTopicUpdateMessage to the temporary values 0x11121314, 0x15161718, etc. We opened TSO New & Improved Trial, sent the packet to the client, and traced what the client did with the fields.
In TSO New & Improved Trial, the cTSOTopicUpdateMessage body handler is located at TSODataServiceClientD_base+0x2ce56. The first 3 dwords are (in order) stored in the cTSOTopicUpdateMessage object (passed as thisptr in ecx) at thisptr+0x18, +0x1c, and +0x20 respectively. The "count" field is read into temporary storage on the stack and is not saved in the cTSOTopicUpdateMessage object. The <count> dwords are stored in a resizable vector pointed to by the dword at thisptr+0x08. (In this vector object, the first dword points to the contiguous array data.) The cTSOValue object is allocated dynamically, and its address is stored in thisptr+0x10. The "pascalvlc" (Pascal-style string with variable-length coding for the length), as I'm calling it, string object is allocated dynamically, and its address is stored at thisptr+0x24. I placed memory breakpoints on all of these fields, and also the array data and string data, and pressed F9 and noted which code looked at this data.
Earlier, I had placed a breakpoint at the top of the "cTSOMapViewRegulator::HandleApprovedTopicUpdate, City_ReservedLotInfo" function (TSOServiceClientD_base+0x200c2). This function was interesting because it showed that the message ID has to be 0xA97360C5, and some unknown field has to be 0xA2AE3282 (the string ID for City_ReservedLotInfo), in order for this debugging string to be reached. This breakpoint at the top of the function was reached. Since the message ID was 0xA97360C5, the jump at +0x200ea was taken, taking us to +0x201e2. Here, the code placed the cTSONetMessageStandard object into ecx (again, we can recognize it as such because its constructor is referenced by the cGZCOM function registered to handle 0x125194E5), and it attempted to call the QueryInterface function (v-table offset +0x00) with IID=0xE973601F. This failed, and this caused the function to instantly return. (If we had sent City_ReservedLotInfo, that function would have to succeed in order for the client to reach +0x2021c and take the jump to handle City_ReservedLotInfo.)
The Pre-Alpha constants table says that 0xE973601F means "GZIID_cITSOTopicUpdateMessage". If you look at the QueryInterface function for cTSOTopicUpdateMessage (+0x2ceae), you can confirm that it accepts this IID.
Thus, it appeared that rather than attaching the cTSOTopicUpdateMessage as the body inside cTSONetMessageStandard, we're actually supposed to use cTSOTopicUpdateMessage *in place of* cTSONetMessageStandard (admittedly they both have "message" in the name), and one of the 3 unknown dwords at the top of cTSOTopicUpdateMessage is probably a Message ID field. We prepared a new packet, with 0x09736027 (cTSOTopicUpdateMessage) used in place of 0x125194E5 (cTSONetMessageStandard) and restarted the client, and we placed hardware breakpoints on those 3 unknown fields. We instantly found that the code was comparing the second dword against a few common message IDs (such as 0x0A411AC7 and 0xDBF301A9). So the second dword was most likely the Message ID field, and we changed that field to 0xA97360C5.
With our new packet, the HandleApprovedTopicUpdate function was again reached, and this time, that function in question returned success.
Afterwards, the function makes two function calls:
The first function reads the second dword from the vector provided in cTSOTopicUpdateMessage. The second function reads the third dword. The code then immediately compares the third dword against various struct field string IDs such as 0xA2AE3282 (City_ReservedLotInfo) and 0xC40B51E2 (Avatar_Top100ListFilter). Thus, the third dword in the vector is likely to be a string ID. It appears that more string IDs could follow in the vector, and the game reads possibly 2 or 3 dwords at a time, with the second or third dword being a string ID each time.
We closed the client and switched back to TSO Play Test Rating Pending. We went back to the function that sets the lot price to $2000 (+0xeb55) and reached the function that creates the cTSOTopicUpdateMessage object (+0x3ad82). Compared to New & Improved Trial, all of the fields are in the same offsets inside the cTSOTopicUpdateMessage object.
As expected, the client sets the second field overall in the body to the message ID 0xA97360C5. The first dword in this case is 0, and the third dword turns out to be another counter field which increments for each request/response (similar to the Counter field in the 0x125194E5 body, but in this case, the second-least-significant-byte of the counter starts at 0x80 rather than 0x00).
The client sets the size of the vector to 3 dwords. The first dword in the vector is the struct type (in this case, 0x7A6CECEE for Lot), the second dword is the 4-byte Parameter (in this case, 0x00??00?? for the lot coordinates), and the third dword is, as expected, the field ID (0xEC8D5C5C for Lot_Price). The Body is set to 0x696D1189 (cTSOValue<unsigned long>) with value set to 0x000007d0 (2000). The string field is set to the empty string.
And suddenly (at least to me, it took me this long), it made sense: cTSOTopicUpdateMessage only "updates" one field. If you want to update all of the fields in a struct instance, you need to send *multiple* DataServiceWrapperPDUs, one for each field.
By taking the client's packet and changing MapView_RollOverInfo_Lot to SimPage_Main, changing Lot to Avatar, changing the lot coordinates to the avatar ID, changing Lot_Price to Avatar_Name, and changing the value from cTSOValue<unsigned long> to cTSOValue<class cRZAutoRefCount<class cIGZString> >, ... it worked.
(Screenshot taken today.)
Thus, we have:
We quickly learned the Uint32, String, etc. cTSOValue types, followed by the cTSOValueVector and cTSOValueMap types, followed by cTSOValueBVector. However, we still needed to know how to respond to fields such as Avatar_Appearance, which is of type AvatarAppearance (rather than type Uint32, etc.), and Lot_Thumbnail, which is of type iunknown.
Avatar_Appearance is made difficult by the fact that its value is itself a struct consisting of multiple values. Now the question is: do we send the entire struct at once inside the cTSOValue field, or do we only set one field at a time?
By doing a search for 0xF1B92C99 (Avatar_Appearance) in TSODataServiceClientD.dll, you can see that it's associated with the value 0xA96E7E5B (cTSOValue<class cRZAutoRefCount<class cITSOProperty> >). I set the cTSOValue field to 0xA96E7E5B, and the following was determined (refer to TSODataServiceClientD_base+0x6dd0c):
If we are supposed to specify the entire struct at once, then we can guess:
And this guess was correct; the packet worked.
(Screenshot taken today. In fact, all screenshots in this post are taken today, sorry.)
Now that we know what "property" means, let's define the lexicon:
Most interesting so far is that you can add a lot to the map using City_ReservedLotInfo:
You can add spotlights to lots with City_SpotlightsVector:
Here is the Property Page (see p.33 of the TSO New & Improved manual):
Currently, we have not yet determined the iunknown format. iunknown is only used for the Lot_Thumbnail and Top100List_Thumbnail fields. In the above screenshot, we have added a thumbnail to this lot by modifying userdata/HouseThumbnails/City1/thumbinfo.bin. (Referring to thumbinfo.bin in our copy of EA-Land helped us achieve this.)
Interestingly, it appears we could have done this without using -NoNetwork mode because the client actually already sends a DataServiceWrapperPDU with the cTSOTopicUpdateMessage body when the player applies a map filter; in other words, the client updates a topic on the server without the server requesting it. But we might want to use -NoNetwork mode again when we determine the body for the 0x9B7ED631 (HouseData) message: the message that got us into a lot in the previous post (link).
The Pre-Alpha constants table says that 0xE973601F means "GZIID_cITSOTopicUpdateMessage". If you look at the QueryInterface function for cTSOTopicUpdateMessage (+0x2ceae), you can confirm that it accepts this IID.
Thus, it appeared that rather than attaching the cTSOTopicUpdateMessage as the body inside cTSONetMessageStandard, we're actually supposed to use cTSOTopicUpdateMessage *in place of* cTSONetMessageStandard (admittedly they both have "message" in the name), and one of the 3 unknown dwords at the top of cTSOTopicUpdateMessage is probably a Message ID field. We prepared a new packet, with 0x09736027 (cTSOTopicUpdateMessage) used in place of 0x125194E5 (cTSONetMessageStandard) and restarted the client, and we placed hardware breakpoints on those 3 unknown fields. We instantly found that the code was comparing the second dword against a few common message IDs (such as 0x0A411AC7 and 0xDBF301A9). So the second dword was most likely the Message ID field, and we changed that field to 0xA97360C5.
With our new packet, the HandleApprovedTopicUpdate function was again reached, and this time, that function in question returned success.
Afterwards, the function makes two function calls:
Code:
CPU Disasm
Address Hex dump Command Comments
032C0205 8B4D 08 MOV ECX,DWORD PTR SS:[EBP+8]
032C0208 8B01 MOV EAX,DWORD PTR DS:[ECX]
032C020A FF50 10 CALL DWORD PTR DS:[EAX+10] ; TSODataServiceClientD_base+0x252ed
032C020D 8B4D 08 MOV ECX,DWORD PTR SS:[EBP+8]
032C0210 8BF8 MOV EDI,EAX
032C0212 8B01 MOV EAX,DWORD PTR DS:[ECX]
032C0214 FF50 14 CALL DWORD PTR DS:[EAX+14] ; TSODataServiceClientD_base+0x252f4The first function reads the second dword from the vector provided in cTSOTopicUpdateMessage. The second function reads the third dword. The code then immediately compares the third dword against various struct field string IDs such as 0xA2AE3282 (City_ReservedLotInfo) and 0xC40B51E2 (Avatar_Top100ListFilter). Thus, the third dword in the vector is likely to be a string ID. It appears that more string IDs could follow in the vector, and the game reads possibly 2 or 3 dwords at a time, with the second or third dword being a string ID each time.
We closed the client and switched back to TSO Play Test Rating Pending. We went back to the function that sets the lot price to $2000 (+0xeb55) and reached the function that creates the cTSOTopicUpdateMessage object (+0x3ad82). Compared to New & Improved Trial, all of the fields are in the same offsets inside the cTSOTopicUpdateMessage object.
As expected, the client sets the second field overall in the body to the message ID 0xA97360C5. The first dword in this case is 0, and the third dword turns out to be another counter field which increments for each request/response (similar to the Counter field in the 0x125194E5 body, but in this case, the second-least-significant-byte of the counter starts at 0x80 rather than 0x00).
The client sets the size of the vector to 3 dwords. The first dword in the vector is the struct type (in this case, 0x7A6CECEE for Lot), the second dword is the 4-byte Parameter (in this case, 0x00??00?? for the lot coordinates), and the third dword is, as expected, the field ID (0xEC8D5C5C for Lot_Price). The Body is set to 0x696D1189 (cTSOValue<unsigned long>) with value set to 0x000007d0 (2000). The string field is set to the empty string.
And suddenly (at least to me, it took me this long), it made sense: cTSOTopicUpdateMessage only "updates" one field. If you want to update all of the fields in a struct instance, you need to send *multiple* DataServiceWrapperPDUs, one for each field.
By taking the client's packet and changing MapView_RollOverInfo_Lot to SimPage_Main, changing Lot to Avatar, changing the lot coordinates to the avatar ID, changing Lot_Price to Avatar_Name, and changing the value from cTSOValue<unsigned long> to cTSOValue<class cRZAutoRefCount<class cIGZString> >, ... it worked.
(Screenshot taken today.)
Thus, we have:
Code:
cTSOTopicUpdateMessage body:
* dword Update counter
* dword Message ID - should be 0xA97360C5
* dword Unknown (Probably m_statusCode)
* dword vector_size - should be 3
* vector:
* dword
* dword cTSOValue clsid (iid=0xE96D0B02)
* cTSOValue body
* pascalvlc Unknown (Probably m_reasonText)We quickly learned the Uint32, String, etc. cTSOValue types, followed by the cTSOValueVector and cTSOValueMap types, followed by cTSOValueBVector. However, we still needed to know how to respond to fields such as Avatar_Appearance, which is of type AvatarAppearance (rather than type Uint32, etc.), and Lot_Thumbnail, which is of type iunknown.
Avatar_Appearance is made difficult by the fact that its value is itself a struct consisting of multiple values. Now the question is: do we send the entire struct at once inside the cTSOValue field, or do we only set one field at a time?
By doing a search for 0xF1B92C99 (Avatar_Appearance) in TSODataServiceClientD.dll, you can see that it's associated with the value 0xA96E7E5B (cTSOValue<class cRZAutoRefCount<class cITSOProperty> >). I set the cTSOValue field to 0xA96E7E5B, and the following was determined (refer to TSODataServiceClientD_base+0x6dd0c):
Code:
cTSOValue<class cRZAutoRefCount<class cITSOProperty> > body:
* dword Body clsid (iid=896E3E90 or "GZIID_cITSOProperty"; clsid should be 0x89739A79 for cTSOProperty)
* dword Body
* dword unknown
* dword unknown1_count
* unknown1s - for each unknown1:
* dword unknown
* dword cTSOValue clsid
* cTSOValue bodyIf we are supposed to specify the entire struct at once, then we can guess:
Code:
cTSOValue<class cRZAutoRefCount<class cITSOProperty> > body:
* dword Body clsid (iid=896E3E90 or "GZIID_cITSOProperty"; clsid should be 0x89739A79 for cTSOProperty)
* dword Body
* dword Struct type (e.g. 0x3B0430BF for AvatarAppearance)
* dword Field count
* Fields - for each field:
* dword Field name (e.g. 0x1D530275 for AvatarAppearance_BodyOutfitID)
* dword cTSOValue clsid
* cTSOValue bodyAnd this guess was correct; the packet worked.
(Screenshot taken today. In fact, all screenshots in this post are taken today, sorry.)
Now that we know what "property" means, let's define the lexicon:
- "Topic" means struct field
- "Property" means struct
- "Proxy" means struct instance
Most interesting so far is that you can add a lot to the map using City_ReservedLotInfo:
You can add spotlights to lots with City_SpotlightsVector:
Here is the Property Page (see p.33 of the TSO New & Improved manual):
Currently, we have not yet determined the iunknown format. iunknown is only used for the Lot_Thumbnail and Top100List_Thumbnail fields. In the above screenshot, we have added a thumbnail to this lot by modifying userdata/HouseThumbnails/City1/thumbinfo.bin. (Referring to thumbinfo.bin in our copy of EA-Land helped us achieve this.)
Interestingly, it appears we could have done this without using -NoNetwork mode because the client actually already sends a DataServiceWrapperPDU with the cTSOTopicUpdateMessage body when the player applies a map filter; in other words, the client updates a topic on the server without the server requesting it. But we might want to use -NoNetwork mode again when we determine the body for the 0x9B7ED631 (HouseData) message: the message that got us into a lot in the previous post (link).
Okay, so I got a packet which states that the lot is offline, now, we can see dialog boxes we didn't see!
I'll paste some screenshots down below.
The game waits 60 seconds before allowing any changes to the category, keep in mind, starting from when you sent the packet.
I'll paste some screenshots down below.
The game waits 60 seconds before allowing any changes to the category, keep in mind, starting from when you sent the packet.
Last edited:
That's a stunningly huge progress you guys made!
So if I'm getting this right, cracking this packet would let you implement all the map view features (or at least most of them) in your server emulator? That's really awesome. Keep it up~!
So if I'm getting this right, cracking this packet would let you implement all the map view features (or at least most of them) in your server emulator? That's really awesome. Keep it up~!
-BloodFighter-
Active Member
Awesome to see how much progress you made. Additionally, I guess that this means that we are a step closer to finally being able to acess the map view and to buy lots and to the gameplay of TSO in general.
Well, not just yet.
We still need to figure out getting INTO a lot. We'll focus on the Packets for objects and movements later.
Last edited:
francot514
Well-Known Member
What is the difference between Neighborhood and cities in TSO???Blayer98 said:
From where will be you able to load the lots, from files, or from server??Blayer98 said:
Neighborhoods only work if there's 3 Lots with the same Neighborhood name and ID as the person's lot who created the Neighborhood.
I can't really say about the last question though, we're still using a server emulator that needs files to be in the same place as it's being contained.
-BloodFighter-
Active Member
So why exactly isnt this bringing us a step further to lot buying and gameplay if I may ask? I do not get ur logicWell, not just yet.
We still need to figure out getting INTO a lot. We'll focus on the Packets for objects and movements later.
Well, all the pdus that we've currently figured out at the moment only cover the Avatar and the Map View, nothing with the lot yet apart from broken ones that crash the game, sorry.
PLEASE NOTE THAT THIS TAKES TIME. SORRY BUT I NEED TO GET THIS NEXT BIT OFF MY CHEST.
I've been on NIOTSO since 2013, and ever since I've joined, me and Fatbag both gotten a lot of progress done ever since we got into the map-view, since I'm willing to give up my free time to do so, and I also have a passion for making games, and this project actually means a lot to me, since I never got to play it as a kid, and I also want to learn more about coding, which I AM.
I'm going to say this now, just because these projects started in 2011/2012, doesn't mean that the game will be finished next year or whatever... Anyway, lot buying does work to an extent, but we can't buy a property since the Lot Name requests a PDU which we still need to crack.
Gameplay isn't available, only in Pre-Alpha at the moment, since the house05.dat packet we own, loads a glitch lot with no name that hides all the info on it (until you set it of course, but you can't set the lot, it defaults to defaulthouse.dat/nonetworkhouse.dat or whatever...)
-BloodFighter-
Active Member
I dont get ur logic.. all I was saying is that this is getting us a step closer to the lot buying and gameplay with the progress you and Fatbag made. I am NOT saying that lot or property buying and the gameplay itself is ready or finished yet And I am VERY aware of the fact that this takes time, afterall I am not a little kiddy as Xezno is.
And I am VERY aware of the fact that this takes time, afterall I am not a little kiddy as Xezno is.
LetsRaceBwoi
Well-Known Member
I can hear you!I dont get ur logic.. all I was saying is that this is getting us a step closer to the lot buying and gameplay with the progress you and Fatbag made. I am NOT saying that lot or property buying and the gameplay itself is ready or finished yet
-BloodFighter-
Active Member
Atleast finally one who knows what I mean haha