1. Forum moved (you can use login and pass from old forum)
  2. Many discussions moved to the bugtracker

Suggestion OpenApoc needs and problems

Discussion in 'Coding' started by Istrebitel, Sep 14, 2017.

Thread Status:
Not open for further replies.
  1. Istrebitel

    Istrebitel Well-Known Member Official Developer Administrator
    64/112

    Joined:
    Aug 8, 2016
    Messages:
    189
    Likes Received:
    80
    Best Answers:
    1
    This is mostly dedicated to people like pmprog and JonnyH who know about programming and the framework of OpenApoc more than me. Currently I'm actually limited in time I can contribute to the project, but since a lot of the game is already "done" in my head, I spend all spare time just implementing everything I already thought of. I could try tackle the problems I'll list here, but that would be very inefficient allocation of my time, as I can do much more by just doing more of the crucial features I already have thought through. And I've seen at least pmprog saying he wants to do something useful. So, I'm going to list stuff here that hopefully someone else can implement.

    So, what we currently need is:

    1) Bug tracker

    Using Github Issues now.


    2) Tooltips for UI

    This game uses them heavilly.


    3) Proper rendering

    This one I might tackle eventually but if someone can figure out an exact algorithm that would be wonderful. I did a post on this http://openapoc.org/threads/task-rendering-algorithm.211/ and got an advice to draw stuff diagonally, but that answer did not account for 2x2x2 units moving through tiles.


    4) Parallel computing

    The way we do it now is we do everything one after another, while it obviously can be done at once.

    Examples of what we could compute parallely:

    a) projectiles cannot collide, so we could just move and calculate collision for every projectile, and then apply every projectile's effect

    b) units's ai could think for every unit at once, as all that AI does is issue orders to units, and AI does not need
    the information on what are unit's orders to function

    c) rendering? I mean gpus are parallel now and they somehow do their work properly in terms of z-buffer, right? like they somehow manage to render multiple stuff at once and have it properly overlap based on distance to camera, we can do the same with sprites? or must sprite renderers all be single-core?

    d) updating map parts and units, units do not directly work with map parts, they work with parameters stored in tiles, parameters like "what's the highest object I can stand on, or can I pop my head there, or will it support my legs if I turn off jetpack". We can update map parts all at once and then just update those parameters all at once, this would allow us to do both units and map parts at once on different cores

    e) Map parts looking for support, this is huge and makes game hang for a big when you blow up something enormous, parallel would help a lot here.​

    I remember JonnyH tried to do parallel for projectiles but for some reason disabled that because something wasn't thread safe. Can we go back and try to re-implement that? What was the problem?


    5) Review and rework of code guidelines.

    Someone good at programming should review what's going on, look for what code guidelines we're missing, rework them, and then I would go and rename /rewrite stuff to fit that.

    What I'm talking about is for example in function "updateFalling" I'm checking if the unit has fallen into another unit, and if so, checking if is a brainsucker and if so, checking if the unit it fell into has a suckable head, if so, attach to it. Should this be forbidden, should we have a guideline like "function must contain only simple logic, and any complex logic must be separated into simple blocks", and thus this function would be separated into "updateFalling", "processFallingIntoOtherUnit", "attemptBrainsuck"?

    Or for example are some classes forbidden to do some things? For example, should BattleUnit play sounds, or should it send some event of "emit sound" kind, which the UI processes and plays sounds? Is BattleUnit supposed to access framework directly?

    I don't think we have that in guidelines now, and some coding has gotten a bit complicated due to that. I think a review and rework is in order, so that this (now quite huge) project stays easy to learn for new programmers.

    Oh and when we do that, I'll also rework all the variable names with "_" in them we have left behind from before camelCase was introduced.


    6) Fix form borders

    Some forms have a minsize of 640x480 but size of 646x484 and have a border of 2x2. Some don't. This looks good but what if a user wants to play fullscreen 640x480? It will still have a border! I propose some way to change all forms to be 640x480, and add border in some other way (like, display a border under form if some tag is present on the form?)


    7) Implement a storage class for categorised externalised engine variables.

    What I'm talking about is externalising things like max speed item can travel along XY when thrown, ticks it takes battleunit to move 1 voxel, projectile speed modifier, how many attempts pathfinding does before giving up, ticks between AI updates, AI priority multipliers for certain actions etc. All this should not be hardcoded but should be read from a file. And it should go in categories so there is a tree structure, at least 1 level deep. So it looks like

    AI Consts
    - AI priority multiplier for action: throw grenade
    - AI priority multiplier for action: fire weapon
    ...

    BattleUnit consts
    - Ticks per frame travelled
    - Ticks per frame turned
    - Ticks per debuff application
    ...

    UI Icons
    - Icon for squad arrow #1 selected
    - Icon for squad arrow #1 unselected
    ...​


    8) Figure out a better format than Xml for some (or all) our data files

    Currently saving battlescape takes an unacceptable amount of time because it saves arrays of 0's and 1's, and resulting file takes like 50 megabytes. Xml is not a good format for anything except small packs of data. It's definetly not to be used for:
    - game saves (those containing things like map part lists, arrays of data etc.)
    - game maps / tilesets / sectors
    - voxel maps​

    We should figure out a new way to store stuff so that save/load times become acceptable at the very least


    9) Figure out a better way of handling data to account for modding

    Currently several things hamper implementing modding:

    a) There is no way to delete something.​

    How do I remove personal disruptor shields from the game? All I can do is make them useless, unproducable, name them "broken disruptor shield" and expect player to sell the stock. But I should be able to just have them vanish once player installed the mod.

    b) We don't separate types and templates from actual game data

    Everything is loaded on game start and saved in one place. There is no real difference between changing an AgentType (which should never happen in the game) or changing City's Vehicles (which is done constantly).

    Data should be separated, some data should load from data files and never change in game, some should be loaded changed and saved. This would allow us to mod data without it getting stored in the saves. Like, we would be able to change AgentType and have it affect the save that is loaded on next run. This would both allow us to have saves take less space, and have way easier time using mods (even turning on/off mid-playthrough which EVERYBODY will want).

    One other crucial example is dependencies and completeness of research. We change that directly in the data itself! What the hell? Shouldn't it be some other place where we track what's developed and what's complete?


    10) I don't remember what exactly, but something about dependencies doesn't work in research and manufacture. Need to sort that out.


    11) We need better lists.

    They should support mouse scroll, they should not immediately select item when you click on scrollbar, they should know how to scroll towards selected item (like when you open inventory for last agent in list of 30, list should scroll to make that agent the bottommost visible item). Also, we need multi-select lists for things like agent inventory


    12) We need text input controls

    For renaming agents and bases and such.


    13) We need to speed up rendering of objects tinted with full-alpha channel.


    I assume right now it multiplies colors while instead it should just fill the area. If I'm wrong and it already checks that then ignore.


    14) We need to speed up rendering of primitives.

    Game uses a lot of lines and brackets and it seems to take a long time to render, can this be sped up? I mean it's just coloring some pixels, is that so resource heavy?

    I'm going to update this post with new stuff.
     
    Last edited: Sep 15, 2017
    makus likes this.
  2. makus

    makus Designer, forum admin Administrator Designer/Artist
    64/112

    Joined:
    Sep 24, 2014
    Messages:
    206
    Likes Received:
    31
    Best Answers:
    2
    one is clear - we need more C++ programmers =]
     
  3. JonnyH

    JonnyH Well-Known Member Official Developer Administrator
    64/112

    Joined:
    Jul 17, 2014
    Messages:
    181
    Likes Received:
    31
    Best Answers:
    0
    My responses in red

    1) Bug tracker


    Is Github Issues sufficient for now? https://github.com/openapoc/openapoc/issues


    4) Parallel computing
    ...
    a) projectiles cannot collide, so we could just move and calculate collision for every projectile, and then apply every projectile's effect

    I already had a go at this for the citymap, it turned out to crash constantly on windows - likely due to some non-const data being modified in the collision detection - the commit was https://github.com/OpenApoc/OpenApoc/commit/ce0fe60b757472a45df5271a6964c749657c02b1 - note the comment is slightly wrong, the sp<> seems OK, it's some other data used in Projectile::checkProjectileCollision()

    I never really bottomed this out, it might be possible to simple revert that change and try to debug - or the issue may have 'solved itself' due to the broken code being changed. One possible path is to try to mark the checkProjectileCollision function and it's TileMap argument as 'const', and follow all the call paths making them const too, as that would effectively guarantee it's threadsafe (as non-modified data can be accessed between multiple threads with no issues - right?)


    c) rendering? I mean gpus are parallel now and they somehow do their work properly in terms of z-buffer, right? like they somehow manage to render multiple stuff at once and have it properly overlap based on distance to camera, we can do the same with sprites? or must sprite renderers all be single-core?

    The renderer is already asynchronous - it simple dispatches the command list of (DRAW_OBJECT_1, DRAW_OBJECT_2, ... ,DRAW_OBJECT_N) to the gpu hardware, where it's processed asynchronously. The creation of that command list (IE the glWhatever() calls) is still done in the one thread, however. It can collect up multiple sprite calls (ie Renderer::draw() that takes an Image) and defer the actual command list creation and submission. This ends up being extremely efficient at drawing sprites, and profiling suggests there's not much improvement available here. The issue here is switching rendering modes - IE having drawin a set of sprites, drawing a coloured quad (a line or rect) or switching render target (RenderSurface in OpenApoc) causes this to be flushed. Worse case is something that goes:
    while (LOTS_OF_TIMES) {
    drawSingleImage();
    drawRect();
    }

    That is an opportunity for improvement.




    13) We need to speed up rendering of objects tinted with full-alpha channel.


    I assume right now it multiplies colors while instead it should just fill the area. If I'm wrong and it already checks that then ignore.

    The way modern GPUs work, the cost of getting the data lined up and ready is often more than the cost of the arithmetic, especially in simple shaders (and the shaders used in openapoc are /very/ simple). This means there's no real advantage in GPU use attempting to optimise a "some pixels are 100% transparent, and some are 100% opaque" compared to "blend with a multiply" - as by the time the GPU can decide if the pixel is transparent or not, it already has to have both possible inputs (the new sprite pixel colour, and the 'previous' colour in case it was transparent). The actual calculation multiplication to possibly blend the two is lost in the noise at this point.

    One optimization we could do here is to stop the non-sprite draws (IE the coloured lines/rects mentioned above) being a barrrier to the sprite batching, as not only does that cause increased CPU use as it has to create a new command buffer every switch between the two, but some GPUs render the two as separate commands too, and that can cause significant performance bottlenecks.

    I have some ideas for this (IE treat solid rects/lines as "sprites" that just sample from a single pixel in the texture atlas that's the correct colour - we can have all 256 colours available in each palette stuck at the side of each texture atlas very easily). This may improve this area.



    14) We need to speed up rendering of primitives.

    Game uses a lot of lines and brackets and it seems to take a long time to render, can this be sped up? I mean it's just coloring some pixels, is that so resource heavy?

    See above - there's not much "optimization" we can do to simplify colouring of pixels on a GPU, the and in fast it will likely be better to treat them as sprites that just sample a single coloured pixel from the texture atlas. The "real" cost being effectively switching between the "sprite" and "colour" draw modes.
     
    makus likes this.
  4. makus

    makus Designer, forum admin Administrator Designer/Artist
    64/112

    Joined:
    Sep 24, 2014
    Messages:
    206
    Likes Received:
    31
    Best Answers:
    2
    Last edited: Sep 15, 2017
  5. Istrebitel

    Istrebitel Well-Known Member Official Developer Administrator
    39/56

    Joined:
    Aug 8, 2016
    Messages:
    189
    Likes Received:
    80
    Best Answers:
    1
    @JonnyH
    Okay, I'm studying parallel computing on C++ now. It seems that default way to do that is threads. Why are we using future? Or is it same as threads?
    Also, why do we have threadpool in framework and things like enqueue? Is it faster than just making and joining threads?

    On topic of graphics, so if I get it correctly primitives fuck everything up big time? So we need to avoid primitives? Does drawing a stretched image work better than a primitive? I mean we could just stretch a pixel instead of drawing a line or something? I assume you proposed the same thing when you talked about palletes?

    Squares and brackets for things like selected craft in city or selected units on strat map could be drawn as sprites - we could make simple sprites for that. Same with frames for things like viewport (what your camera sees) or buildings (the ones that show up which buildings belong to which org) - we could draw that by using 8 sprites (4 edges, 4 sides) stretched to make a proper sized frame. I assume drawing 8 sprites will be faster than drawing four lines?

    The only place where we really need lines is when we draw lines for units moving to locations (or vehicles). Could we change that to some better way (as said above)?
     
    Last edited: Sep 15, 2017
  6. JonnyH

    JonnyH Well-Known Member Official Developer Administrator
    64/112

    Joined:
    Jul 17, 2014
    Messages:
    181
    Likes Received:
    31
    Best Answers:
    0
    A 'future' is just a handle for an asynchronous task - it's not a thread itself. Our ThreadPool implementation has a fixed number of threads (based on the number of cores available on the CPU it's running on), and queue-ing a task on the threadpool causes it to be added to the threadpool scheduler.

    If there's a spare thread in the theadpool (IE not already running another task), it'll immediately start it, if not every time a thread finishes a task it'll take another off the queue and complete it.

    A "Future" is a relatively common primitive that allows you to manage the task, you can query if it's completed asynchronously, or wait for the task to be complete and block until it's done. I think it's easier and cleaner than creating a std::thread directly, especially with smaller tasks (e.g. on some platforms, like windows, the cost of creating a new thread is high, so it's advantageous to create a number of threads in a pool and get them to sleep until there's some work to do)

    For rendering: "coloured primitives" do kinda screw up the current rendering pipeline if they're interleaved with sprites, as it stops the render being able to package the multiple sprite draws together, so the fixed-time but high-cost action of putting all the command lists together in the GL driver has to be done every time the state switches from 'sprite' to 'primitive' - while we can package up /thousands/ of sprites in a single command list if rendering nothing else. The sprite code handles scaled sprites the same, so if you're already drawing a number of sprites, the cost of an extra sprite draw attached to the end would be very low, even if it's scaled and stretched. My suggestion of copying the palette into the texture atlas and treating coloured primitives as 'sprites' would mean we wouldn't have to pay this switching 'cost' as much, but is just one possible implementation of that speedup (drawing the common brackets etc. directly as sprites would be another - it'll probably be pretty much equivalent speed-wise). That would mean all the draws go through the sprite path, so wouldn't be paying the (expensive) mode switching cost as much.

    I can try to fix this up myself, and there's other values I may be able to tweak to improve performance (likely around spritesheet size and batching buffer size, that could help with this further)
     
    makus likes this.
  7. Istrebitel

    Istrebitel Well-Known Member Official Developer Administrator
    39/56

    Joined:
    Aug 8, 2016
    Messages:
    189
    Likes Received:
    80
    Best Answers:
    1
    On topic of parallel, I've studied the topic, and made an issue on github to ask for your opinion. It looks like it's not worth it but maybe you know better.

    On topic of rendering primitives, again, all we really need (and cannot do via sprites easily) is:
    1) Lines for unit movement destinations
    2) Rectangle for viewport (even this we could do by creating an image via setting pixels and then using it, changing only if screen size changes)
    3) Rectangles for buildings when buying or viewing org ownership

    I think that's it. Other places where using primitives may be useful is:
    1) Circles for target buildings (alien spotted or alien dimension mission)
    2) Brackets for vehicles in city (they're like those four angles around a vehicle)
    3) Rectangles around selected units in map view
    But these could also be done via sprites easilly (and probably should)

    So, could you implement top three that will work by stretching? If so, I'm implementing the bottom three and we're fine.
     
    makus likes this.
  8. JonnyH

    JonnyH Well-Known Member Official Developer Administrator
    64/112

    Joined:
    Jul 17, 2014
    Messages:
    181
    Likes Received:
    31
    Best Answers:
    0
    "sprites" are just rotated scaled rects sourcing their colour from a spritesheet texture - any single-colour rectangle could be implemented in the same path by having a single-pixel 'source' scaled and rotated to however big the rect needs to be. Lines just being 1 pixel wide rectangles. So I don't see why all 3 can't be done with the sprite code (with a bit of modification)

    The other primitives (circles around buildings, brackets around selected vehicles) are always drawn on top of the city - so need to be drawn last. This likley means they don't interrupt the sprite batching optimizations as much, as by then you've probably drawn all your sprites underneath it.
     
    makus likes this.
  9. Istrebitel

    Istrebitel Well-Known Member Official Developer Administrator
    39/56

    Joined:
    Aug 8, 2016
    Messages:
    189
    Likes Received:
    80
    Best Answers:
    1
    Well then please do it. I've already implemented bottom three as sprites.
     
  10. Istrebitel

    Istrebitel Well-Known Member Official Developer Administrator
    39/56

    Joined:
    Aug 8, 2016
    Messages:
    189
    Likes Received:
    80
    Best Answers:
    1
    Moved stuff from this topic to github issues
     
Thread Status:
Not open for further replies.

Share This Page