Investigating an issue about creating string literals in multiple threads in Unity

Alan Liu
7 min readJul 10, 2023

--

Table of Contents

· Problem
· Investigation
· Workaround
· Link

Problem

During an internal test, I found there was an exception occurred on an iOS device:

NullReferenceException: A null value was found where an object instance was required.
Common.WeaponSkinManager.Init ()
ConsoleInitializer.Start ()

Since the exception occurred during the game was initializing, it caused the game couldn’t work correctly. And the exception couldn’t be reproduced again.

Investigation

The following code is from WeaponSkinManager in our game:

public class WeaponSkinManager
{
public Dictionary<int, int> SkinWeaponDict = new Dictionary<int, int>();
public Dictionary<int, List<int>> WeaponSkinsDict = new Dictionary<int, List<int>>();

public void Init()
{
SkinWeaponDict.Clear();
WeaponSkinsDict.Clear();
foreach (var id in GameEntityManager.Instance.WeaponSkinIdList)
{
var info = GameEntityManager.Instance.GetEntityInfo(id) as WeaponSkinInfo;
SkinWeaponDict.Add(id, info.WeaponClassId);

if (!WeaponSkinsDict.ContainsKey(info.WeaponClassId))
WeaponSkinsDict.Add(info.WeaponClassId, new List<int>());

WeaponSkinsDict[info.WeaponClassId].Add(id);
}
}

// ...
}

It looks like that only info is possible to be null and there are two possibilities:

  1. GameEntityManager.Instance.GetEntityInfo(id) returns a null object
  2. GameEntityManager.Instance.GetEntityInfo(id) returns a non-null object, but its type is not WeaponSkinInfo

According to the code related to WeaponSkinIdList and GetEntityInfo, the value that GetEntityInfo returned was coming from GameEntityCatalogParser.WeaponSkinIdList, which was written in GameEntityCatalogParser.PostParse:

protected override object PostParse(PackedCatalogCache<EntityInfo> cache)
{
foreach (EntityInfo info in cache.Configs)
{
if (info == null)
continue;

EntityInfoDict[info.ClassId] = info;

switch (info.EntityType)
{
// ...

case EntityType.WeaponSkin:
WeaponSkinIdList.Add(info.ClassId);
break;

// ...

default:
break;
}
}

return null;
}

And the value in cache.Configs was coming from GameEntityCatalogParser.ParseChildFile:

protected override EntityInfo ParseChildFile(PackedCatalogCache<EntityInfo> cache, PackedCatalogData packedData, int index)
{
EntityInfo info = null;

string file = cache.Files[index];
string subFolder = file.Substring(0, file.LastIndexOf('.')) + "/";

string content = packedData.GetText(file);

if (file.StartsWith(EntityTypeName.WeaponWithBullet + "_"))
{
WeaponWithBulletInfo wInfo = JsonUtils.FromJsonFast<WeaponWithBulletInfo>(content);
info = wInfo;
}
else if (file.StartsWith(EntityTypeName.WeaponSkin + "_"))
{
WeaponSkinInfo wInfo = JsonUtils.FromJsonFast<WeaponSkinInfo>(content);
info = wInfo;
}
// ...
else
LogUtils.Error(string.Format("unsupported file: {0}", file));

return info;
}

private void ParseChildFile(C cache, PackedCatalogData packedData, int start, int count)
{
ConfigManager.QueueUserWorkItem(_ =>
{
for (int i = start; i < start + count; ++i)
{
try
{
cache.Configs[i] = ParseChildFile(cache, packedData, i);
}
catch (Exception e)
{
Log.Error(e, "Failed to parse file: {0}", cache.Files[i]);
}
}

if (Interlocked.Add(ref cache.ParsingCount, -count) == 0)
EndParsing(cache);
}, null);
}

From these code, I couldn’t find any reason why the exception occurred. Since GameEntityCatalogParser was using multiple threads to parse files, I suspected there were bugs related to race condition in the code.

During the investigation, I found another issue: exceptions and errors occurred before users logged in weren’t uploaded to our server. After fixing this issue, I found there were some strange errors when the game was initializing and none of them was reproducible:

// Error 1: Sometimes the code ran into the last branch
SkillTriggerInfo ToSkillTriggerInfo(PackedCatalogData packedData, Dictionary<string, string> dict, string folder)
{
// SkillTriggerTypeName.XXX are string literals

string file = dict["file"];
string content = packedData.GetText(folder + file);
SkillTriggerInfo triggerInfo = null;
if (file.Contains(SkillTriggerTypeName.HitDefaultPick))
triggerInfo = JsonUtils.FromJsonFast<HitInfoDefaultPick>(content);
else if (file.Contains(SkillTriggerTypeName.FollowSource))
triggerInfo = JsonUtils.FromJsonFast<HitInfoFollowSource>(content);
// ...
else
LogUtils.Error(string.Format("unsupported hitInfo file: {0}", file));
}

// Error 2: Sometimes the content of a string literal became incorrect
// For example, for the code "content = packedData.GetText(subFolder + "trigger_list.json");"
// GetText() threw an exception: File doesn't exist: Skill_12004/使用占星师,对敌人累计造成4000点伤害
// According to the implementation of GetText(), the text after "File doesn't exist: " should be Skill_12004/trigger_list.json
protected override EntityInfo ParseChildFile(PackedCatalogCache<EntityInfo> cache, PackedCatalogData packedData, int index)
{
// EntityTypeName.XXX are string literals

EntityInfo info = null;

string file = cache.Files[index];
string subFolder = file.Substring(0, file.LastIndexOf('.')) + "/";

string content = packedData.GetText(file);

if (file.StartsWith(EntityTypeName.WeaponWithBullet + "_"))
{
WeaponWithBulletInfo wInfo = JsonUtils.FromJsonFast<WeaponWithBulletInfo>(content);
info = wInfo;
}
else if (file.StartsWith(EntityTypeName.WeaponSkin + "_"))
{
WeaponSkinInfo wInfo = JsonUtils.FromJsonFast<WeaponSkinInfo>(content);
info = wInfo;
}
// ...
else if (file.StartsWith(EntityTypeName.Skill + "_"))
{
SkillInfo sInfo = JsonUtils.FromJsonFast<SkillInfo>(content);
info = sInfo;
sInfo.TriggerList = new List<SkillTriggerInfo>();

content = packedData.GetText(subFolder + "trigger_list.json");
List<Dictionary<string, string>> triggerList = JsonUtils.FromJsonFast<MyListDict>(content).ToList();
for (int j = 0; j < triggerList.Count; j++)
{
Dictionary<string, string> dict = triggerList[j];

SkillTriggerInfo triggerInfo = ToSkillTriggerInfo(packedData, dict, subFolder);
sInfo.TriggerList.Add(triggerInfo);
}
}
// ...
else
LogUtils.Error(string.Format("unsupported file: {0}", file));
}

public string GetText(string name)
{
string text;
if (!TryGetText(name, out text))
throw new Exception("File doesn't exist: " + name);

return text;
}

The second error indicated the content of a string became incorrect clearly. And the first error could be explained by the same reason. For example, suppose the value of SkillTriggerTypeName.HitDefaultPick becomes another string, when the prefix of file is "HitDefaultPick_", the code runs into the last branch.

The exception mentioned at the beginning could be explained by the content of a string literal becoming incorrect too: for a JSON file that’s supposed to be deserialized to WeaponSkinInfo type, if the code runs into an incorrect branch, it would be deserialized to another type, but its EntityType would be WeaponSkin. In GameEntityCatalogParser.PostParse, this object would be saved into WeaponSkinIdList and caused GameEntityManager.Instance.GetEntityInfo(id) as WeaponSkinInfo produced a null object finally.

So the question became why the content of a string literal became incorrect? In order to investigate it, we need to look into the c++ code generated by IL2CPP:

// Bulk_Assembly-CSharp_15.cpp
// The code was generated from the modified C# code during the investigation, but it doesn't affect the analysis.

// Common.EntityInfo Common.GameEntityCatalogParser::ParseChildFile(Common.PackedCatalogCache`1<Common.EntityInfo>,Common.PackedCatalogData,System.String)
extern "C" IL2CPP_METHOD_ATTR EntityInfo_t3640658339 * GameEntityCatalogParser_ParseChildFile_m1727472549 (GameEntityCatalogParser_t2257004175 * __this, PackedCatalogCache_1_t1683540284 * ___cache0, PackedCatalogData_t1718875397 * ___packedData1, String_t* ___file2, const RuntimeMethod* method)
{
static bool s_Il2CppMethodInitialized;
if (!s_Il2CppMethodInitialized)
{
il2cpp_codegen_initialize_method (GameEntityCatalogParser_ParseChildFile_m1727472549_MetadataUsageId);
s_Il2CppMethodInitialized = true;
}
//...
{
V_0 = (EntityInfo_t3640658339 *)NULL;
String_t* L_0 = ___file2;
String_t* L_1 = ___file2;
NullCheck(L_1);
int32_t L_2 = String_LastIndexOf_m3451222878(L_1, ((int32_t)46), /*hidden argument*/NULL);
NullCheck(L_0);
String_t* L_3 = String_Substring_m1610150815(L_0, 0, L_2, /*hidden argument*/NULL);
IL2CPP_RUNTIME_CLASS_INIT(String_t_il2cpp_TypeInfo_var);
String_t* L_4 = String_Concat_m3937257545(NULL /*static, unused*/, L_3, _stringLiteral3452614529, /*hidden argument*/NULL);
V_1 = L_4;
PackedCatalogData_t1718875397 * L_5 = ___packedData1;
String_t* L_6 = ___file2;
NullCheck(L_5);
String_t* L_7 = PackedCatalogData_GetText_m1965185178(L_5, L_6, /*hidden argument*/NULL);
V_2 = L_7;
String_t* L_8 = ___file2;
bool L_9 = CustomStringUtils_CustomStartsWith_m1044240037(NULL /*static, unused*/, L_8, _stringLiteral3495210533, /*hidden argument*/NULL);
if (!L_9)
{
goto IL_0042;
}
}
//...
}

As you can see, a string literal in C# became a variable _stringLiteral3495210533 in the generated C++ code. Where does it come from?

// Il2CppMetadataUsage.cpp
String_t* _stringLiteral3495210533;

extern void** const g_MetadataUsages[73910] =
{
// ...
(void**)(&_stringLiteral3495210533),
// ...
};

// Il2CppMetadataRegistration.cpp
extern void** const g_MetadataUsages[];
extern const Il2CppMetadataRegistration g_MetadataRegistration =
{
33686,
s_Il2CppGenericTypes,
9827,
g_Il2CppGenericInstTable,
77179,
s_Il2CppGenericMethodFunctions,
74908,
g_Il2CppTypeTable,
80321,
g_Il2CppMethodSpecTable,
14575,
g_FieldOffsetTable,
14575,
g_Il2CppTypeDefinitionSizesTable,
73137,
g_MetadataUsages,
};

// Il2CppCodeRegistration.cpp
void s_Il2CppCodegenRegistration()
{
il2cpp_codegen_register (&g_CodeRegistration, &g_MetadataRegistration, &s_Il2CppCodeGenOptions);
}
#if RUNTIME_IL2CPP
static il2cpp::utils::RegisterRuntimeInitializeAndCleanup s_Il2CppCodegenRegistrationVariable (&s_Il2CppCodegenRegistration, NULL);
#endif

The address of _stringLiteral3495210533 is saved into g_MetadataUsagesand passed into IL2CPP. So it seems that string literals are created inside IL2CPP.

Back to the c++ code generated for ParseChildFile, note there are some initialization code at the beginning of the method:

extern "C" IL2CPP_METHOD_ATTR EntityInfo_t3640658339 * GameEntityCatalogParser_ParseChildFile_m1727472549 (GameEntityCatalogParser_t2257004175 * __this, PackedCatalogCache_1_t1683540284 * ___cache0, PackedCatalogData_t1718875397 * ___packedData1, String_t* ___file2, const RuntimeMethod* method)
{
static bool s_Il2CppMethodInitialized;
if (!s_Il2CppMethodInitialized)
{
il2cpp_codegen_initialize_method (GameEntityCatalogParser_ParseChildFile_m1727472549_MetadataUsageId);
s_Il2CppMethodInitialized = true;
}

// ...
}

When the method is called at the first time, it calls il2cpp_codegen_initialize_method. Let's see what the function does:

// Unity\Hub\Editor\2017.4.40f1\Editor\Data\il2cpp\libil2cpp\codegen\il2cpp-codegen-il2cpp.h
inline void il2cpp_codegen_initialize_method(uint32_t index)
{
il2cpp::vm::MetadataCache::InitializeMethodMetadata(index);
}

// Unity\Hub\Editor\2017.4.40f1\Editor\Data\il2cpp\libil2cpp\vm\MetadataCache.cpp
void MetadataCache::InitializeMethodMetadata(uint32_t index)
{
// ...

for (uint32_t i = 0; i < count; i++)
{
// ...
switch (usage)
{
// ...
case kIl2CppMetadataUsageStringLiteral:
*s_Il2CppMetadataRegistration->metadataUsages[destinationIndex] = GetStringLiteralFromIndex(decodedIndex);
break;
default:
NOT_IMPLEMENTED(MetadataCache::InitializeMethodMetadata);
break;
}
}
}

Il2CppString* MetadataCache::GetStringLiteralFromIndex(StringLiteralIndex index)
{
if (index == kStringLiteralIndexInvalid)
return NULL;

if (s_StringLiteralTable[index])
return s_StringLiteralTable[index];

const Il2CppStringLiteral* stringLiteral = (const Il2CppStringLiteral*)((const char*)s_GlobalMetadata + s_GlobalMetadataHeader->stringLiteralOffset) + index;
s_StringLiteralTable[index] = String::NewLen((const char*)s_GlobalMetadata + s_GlobalMetadataHeader->stringLiteralDataOffset + stringLiteral->dataIndex, stringLiteral->length);

return s_StringLiteralTable[index];
}

void MetadataCache::InitializeStringLiteralTable()
{
s_StringLiteralTable = (Il2CppString**)GarbageCollector::AllocateFixed(s_GlobalMetadataHeader->stringLiteralCount / sizeof(Il2CppStringLiteral) * sizeof(Il2CppString*), NULL);
}

From the code above, we can know how a string literal is created: when a c# method is called at the first time, if a string literal is used in it, il2cpp_codegen_initialize_method calls GetStringLiteralFromIndex to create a string literal and save it into s_StringLiteralTable. Then, the string literal is assigned to metadataUsages, so that the generated c++ code can access it by a global variable, like _stringLiteral3495210533.

And there is an important point to keep it working correctly: s_StringLiteralTable is allocated by GarbageCollector::AllocateFixed, which means the string literals saved into it won't be garbage collected.

Based on the all information above, I thought there was a possible situation that a string literal that’s still in use could be garbage collected:

  • Since ParseChildFile can be called in multiple threads simultaneously, which means il2cpp_codegen_initialize_method and GetStringLiteralFromIndex can be called in multiple threads simultaneously too.
  • Suppose there are two threads, both are calling GetStringLiteralFromIndex for a same string literal which hasn't been created yet.
  • Thread 1 creates a string object and save it into s_StringLiteralTable, then Thread 2 also creates a string object and save it into s_StringLiteralTable. So the string object created by Thread 2 is saved into s_StringLiteralTable.
  • Two threads both return to InitializeMethodMetadata and are assigning a string object to metadataUsages. Suppose Thread 2 assigns first and then Thread 1 assigns. So the string object created by Thread 1 is saved into metadataUsages.
  • But the string object created by Thread 1 wasn’t saved into s_StringLiteralTable, during the next garbage collection, it will be garbage collected. And the content of it may become some unexpected values, like another string literal.

After thinking about this possibility, I created a small project to reproduce it successfully. The project has been submitted to Unity.

Workaround

Since I can’t modify the code of IL2CPP, I modified the code of GameEntityCatalogParser to not use multiple threads to do parsing.

Link

--

--