Devlog: Flecs City (Part 2)

It’s been a while since I last posted back in May. A lot has happened since then: Hytale was cancelled, I entered the job market for the first time in eight years, I spent a few months focusing on sharpening my skills and building my portfolio, and I was lucky enough to catch the interest of several studios. Ultimately I accepted an offer from one of those studios and I’m now working full time on an Unreal Engine project.

Since settling into that new job, I decided to revisit this project and wanted to focus my attention on two things—library compatibility and some architectural changes.

Getting Raylib and ENet to co-exist

In the previous post, I mentioned a bug with GameNetworkingSockets on Linux, which I reported here. I’ve not been able to figure this one out and don’t feel like investigating it further, so I decided to replace GNS with ENet, which is a popular UDP library that has been around for many years and should be more than enough for my purposes. Of course it lacks the Steam integration, but I don’t need that for this project.

Switching to it was not the smoothest experience because it uses some Windows headers that lead to collisions with Raylib function names, an issue that has been discussed at length in this GitHub issue. For reasons that aren’t very clear to me, the author of Raylib has no plans to prefix the library’s function names (despite that being a common practice in C where namespacing doesn’t exist), making them prone to this type of issue when you’re using Windows APIs (whether directly or indirectly).

To work around that issue, after some trial and error, I found it was enough to set a few definitions via CMake:

if(WIN32)
    target_link_libraries(CoreDependencies INTERFACE winmm ws2_32)
    target_compile_definitions(CoreDependencies INTERFACE
        WIN32_LEAN_AND_MEAN
        NOGDI
        NOUSER
    )
endif()

This seems to give ENet everything it needs while avoiding the declarations that upset Raylib.

Running mode

Previously, I made the application decide whether to open a socket or connect to one (or neither) based on whether the user passed a value to the --listen arg or the --connect arg (or neither). I realised this wasn’t working very well, so I added a separate --mode arg:

enum RunMode
{
    Monolith,
    Server,
    Client
};

std::unordered_map<std::string, RunMode> mModeMap{
    {"monolith", RunMode::Monolith},
    {"server", RunMode::Server},
    {"client", RunMode::Client}
};

args::ArgumentParser mParser{"Flecs City"};
args::MapFlag<std::string, RunMode> mMode{mParser, "mode", "Mode to run in (monolith|server|client). Defaults to monolith.", {'m', "mode"}, mModeMap};

The general idea with these modes is as follows:

• Monolith: Run the client and server code monolithically with no networking.
• Server: Run as a dedicated server (execute the server code while listening for connections on a given port).
• Client: Run as a client (connect to a server address and execute the client code).

My current thinking is that where a server and a client would each get their own ECS world and the network layer would take care of synchronising entity and component updates, in monolith mode, there’s just a single ECS that everything gets registered to, including both client-specific and server-specific systems. I’m sure in practice it won’t be that simple at all, but that’s the plan. The end result I have in mind is a single binary that can run totally offline, as a dedicated server, or as a client.

This does exclude the option of running a server and connecting to it in the same process, but I want to implement monolith mode instead since that achieves the same result with zero networking overhead.

Network thread

Up until now, there was no concurrency of any kind in this project. Networking seemed like a good first candidate for changing that, so I implemented a basic network thread for the modes that use networking. The network thread runs for the duration of the main thread and handles the complete ENet lifecycle from enet_initialize() to enet_deinitialize().

The main part of this is a simple abstract NetworkThread:

#pragma once

#include <atomic>
#include <condition_variable>
#include <mutex>
#include <queue>
#include <thread>

#include <enet/enet.h>

#include "Network/Types.h"

namespace fc::Network
{

class NetworkThread
{
public:
    enum State
    {
        Idle,
        Active,
        PendingExit
    };

    NetworkThread() = default;
    ~NetworkThread();

    bool Start();
    void Stop();

protected:
    std::thread mThread;
    std::atomic<State> mState{State::Idle};

    std::atomic<ENetHost*> mHost{nullptr};

    /// @brief Connection address (when this is a client) or listen address (when this is a server).
    ENetAddress mAddress;

    std::mutex mIncomingMutex;
    std::mutex mOutgoingMutex;
    std::condition_variable mOutgoingCV;

    std::queue<NetworkMessage> mIncomingMessages;
    std::queue<NetworkMessage> mOutgoingMessages;

    virtual bool Init() = 0;
    virtual void HandleEvent(const ENetEvent& event);
    virtual void Disconnect() {}

private:
    /// @brief Thread function
    void Main();
};

} // namespace fc::Network

#include "NetworkThread.h"

#include <chrono>

#include <enet/enet.h>
#include <spdlog/spdlog.h>

namespace fc::Network
{

NetworkThread::~NetworkThread()
{
    Stop();
}

bool NetworkThread::Start()
{
    if (mThread.joinable())
    {
        return false;
    }

    mThread = std::thread(&NetworkThread::Main, this);

    return true;
}

void NetworkThread::Stop()
{
    mOutgoingCV.notify_all();

    if (mThread.joinable())
    {
        mThread.join();
    }
}

void NetworkThread::Main()
{
    if (enet_initialize() != 0)
    {
        spdlog::error("Failed to initialize ENet");
        return;
    }

    if (!Init())
    {
        enet_deinitialize();
        return;
    }

    State state = mState.load();
    ENetEvent event;
    while (state != State::PendingExit)
    {
        if (state == State::Idle)
        {
            state = mState.load();
            continue;
        }

        int result = enet_host_service(mHost, &event, 1000);
        if (result > 0)
        {
            HandleEvent(event);
        }
        else if (result < 0)
        {
            spdlog::error("An ENet service error occurred");
            break;
        }

        // Small yield to prevent busy waiting
        // TODO: replace with a more suitable alternative (CV?)
        std::this_thread::sleep_for(std::chrono::milliseconds(1));

        state = mState.load();
    }

    Disconnect();

    if (mHost)
    {
        enet_host_destroy(mHost);
    }

    enet_deinitialize();
}

void NetworkThread::HandleEvent(const ENetEvent& event)
{
    switch (event.type)
    {
        case ENET_EVENT_TYPE_RECEIVE:
            {
                std::vector<uint8_t> data(event.packet->data, event.packet->data + event.packet->dataLength);

                {
                    std::lock_guard<std::mutex> lock(mIncomingMutex);
                    mIncomingMessages.push({std::move(data), event.channelID, 0});
                }

                enet_packet_destroy(event.packet);
            }

        default:
            break;
    }
}

} // namespace fc::Network

I wrote two implementations of this—one for the server and one for the client. They implement Init() and HandleEvent(const ENetEvent& event), and the client additionally implements methods for handling connections to peers during runtime.

A quick test with a server and two clients reveals successful connections in the server log:

[info] Registering components...
[info] Initialising ECS...
[info] Entering main loop...
[info] Server host created.
[info] Client connected from 127.0.0.1:60973 (peer ID: 0). Total clients: 1.
[info] Client connected from 127.0.0.1:60974 (peer ID: 1). Total clients: 2.

There’s a lot more to do here, most obviously data transfer, which is probably what I’ll focus on next. Once I have a reasonable foundation there, I’ll think about how to handle replication of ECS components.

If you want to take a closer look at the source code, you can find the repo here:

GitHub - Riari/flecs-city: C++ sandbox project with Flecs v4 and raylib.