Save format changes and issues

I'm looking to help out a little bit, and I figured I could investigate the save file issues mentioned on github. I was curious as to what kind of load and save times you've been seeing? I noticed one save file I pulled down from a github issue was a whopping 50 MB of XML text!! I was kind of stunned even with it compressed down to about of 1MB.

Anyway, I've had some experience with RapidJSON, and I figured I could try and build in support and test performance. Downside is that it will probably be a single file, since JSON doesn't support include files. I could try YAML which does support file inclusion, but from what I've seen of benchmarks for YAML, I'm not sure it will be much faster than some XML parsers.

I think I know where in the code most of the changes will go (serialize.cpp/.h) and the serialization tool which looks like it will come in handy.

Is there any other formats you might want me to investigate if I can get this up and running? MsgPack?

I assume you want to strive for human readable format if possible, correct?

 

Yes, I'm familiar with OpenXcom's YAML file format which is a format I prefer for configuration files. It's not as verbose as XML, but has some more powerful language features over JSON like the ability to add comments or include other YAML files. However, I did some research on processing speed benchmarks, and YAML doesn't hold up as well in comparison to some other serialization techniques. It's not without merit though. Long term, I would like to try several serialization methods and do some benchmarks on the OpenApoc save data for comparison. Right now, I've almost completed the RapidJSON writer implementation (I think) inside the Serialization classes. I still need to work on the reader for deserialization. I hope to have something testable in a couple of weeks (got to do some taxes this weekend).

 

Just a quick update. I haven't made much progress on RapidJSON, partly because I've been distracted with a OXCE+ lately and after talking to JohnnyH, it sounds like he's already run tests with RapidJSON and didn't see much benefit. So, I'll be turning my efforts toward FlatBuffers as an experiment and see how far I get.