1. Forum moved (you can use login and pass from old forum)
  2. Many discussions moved to the bugtracker

Save format changes and issues

Discussion in 'Coding' started by jgatkinsn, Mar 1, 2018.

  1. jgatkinsn

    jgatkinsn Member Programmer

    Joined:
    Nov 22, 2015
    Messages:
    10
    Likes Received:
    8
    Best Answers:
    0
    I'm looking to help out a little bit, and I figured I could investigate the save file issues mentioned on github. I was curious as to what kind of load and save times you've been seeing? I noticed one save file I pulled down from a github issue was a whopping 50 MB of XML text!! I was kind of stunned even with it compressed down to about of 1MB.

    Anyway, I've had some experience with RapidJSON, and I figured I could try and build in support and test performance. Downside is that it will probably be a single file, since JSON doesn't support include files. I could try YAML which does support file inclusion, but from what I've seen of benchmarks for YAML, I'm not sure it will be much faster than some XML parsers.

    I think I know where in the code most of the changes will go (serialize.cpp/.h) and the serialization tool which looks like it will come in handy.

    Is there any other formats you might want me to investigate if I can get this up and running? MsgPack?

    I assume you want to strive for human readable format if possible, correct?
     
    makus and Patrick Sweetman like this.
  2. makus

    makus Designer, forum admin Administrator Designer/Artist

    Joined:
    Sep 24, 2014
    Messages:
    250
    Likes Received:
    43
    Best Answers:
    2
    Hey jgatkinsn its great that you join to project =] :treason::play:
    im not programmer, but i heard, devs speak for YAML like in OpenXcom for save system
     
  3. jgatkinsn

    jgatkinsn Member Programmer

    Joined:
    Nov 22, 2015
    Messages:
    10
    Likes Received:
    8
    Best Answers:
    0
    Yes, I'm familiar with OpenXcom's YAML file format which is a format I prefer for configuration files. It's not as verbose as XML, but has some more powerful language features over JSON like the ability to add comments or include other YAML files. However, I did some research on processing speed benchmarks, and YAML doesn't hold up as well in comparison to some other serialization techniques. It's not without merit though. Long term, I would like to try several serialization methods and do some benchmarks on the OpenApoc save data for comparison. Right now, I've almost completed the RapidJSON writer implementation (I think) inside the Serialization classes. I still need to work on the reader for deserialization. I hope to have something testable in a couple of weeks (got to do some taxes this weekend).
     
    Patrick Sweetman likes this.
  4. makus

    makus Designer, forum admin Administrator Designer/Artist

    Joined:
    Sep 24, 2014
    Messages:
    250
    Likes Received:
    43
    Best Answers:
    2
    yep they also mention some other language wich YAML )
     
  5. jgatkinsn

    jgatkinsn Member Programmer

    Joined:
    Nov 22, 2015
    Messages:
    10
    Likes Received:
    8
    Best Answers:
    0
    Just a quick update. I haven't made much progress on RapidJSON, partly because I've been distracted with a OXCE+ lately and after talking to JohnnyH, it sounds like he's already run tests with RapidJSON and didn't see much benefit. So, I'll be turning my efforts toward FlatBuffers as an experiment and see how far I get.
     
  6. makus

    makus Designer, forum admin Administrator Designer/Artist

    Joined:
    Sep 24, 2014
    Messages:
    250
    Likes Received:
    43
    Best Answers:
    2
    well you can speak wich supsuper or openxcom team sure they also make some experiments beore doing own openxcom
     

Share This Page