Client throws a NotServerException when connecting to a Host that shut down too quickly

[A1win]

Description

This issue sometimes occurs as Client when connecting to a Host that shut down too quickly during the connection process. We can reproduce this quite reliably in our project when the connecting Client has a high latency (200+ ms) and there's at least two Clients connecting to the Host simultaneously. We haven't been able to reproduce this with only one connecting Client. We usually need to shut down and restart the Host multiple times before the error occurs, and we're not sure if it can happen on the first connection attempt.

The Client gets the following error:

NotServerException: Destroy a spawned NetworkObject on a non-host client is not valid. Call Destroy or Despawn on the server/host instead.
  at Unity.Netcode.NetworkObject.OnDestroy () [0x000b0] in ***\Library\PackageCache\com.unity.netcode.gameobjects@1.2.0\Runtime\Core\NetworkObject.cs:451 

With log level set to Developer in NetworkManager, the Client also gets the following warning about several NetworkObjects right after the exception:

[Netcode] NetworkBehaviour-SomeGameObject is being destroyed while NetworkObject-SomeGameObject is still spawned! (could break state synchronization)
UnityEngine.StackTraceUtility:ExtractStackTrace ()
UnityEngine.DebugLogHandler:LogFormat (UnityEngine.LogType,UnityEngine.Object,string,object[])
UnityEngine.Logger:Log (UnityEngine.LogType,object)
UnityEngine.Debug:LogWarning (object)
Unity.Netcode.NetworkLog:LogWarning (string) (at Library/PackageCache/com.unity.netcode.gameobjects@1.2.0/Runtime/Logging/NetworkLog.cs:28)
Unity.Netcode.NetworkObject:OnNetworkBehaviourDestroyed (Unity.Netcode.NetworkBehaviour) (at Library/PackageCache/com.unity.netcode.gameobjects@1.2.0/Runtime/Core/NetworkObject.cs:1476)
Unity.Netcode.NetworkBehaviour:OnDestroy () (at Library/PackageCache/com.unity.netcode.gameobjects@1.2.0/Runtime/Core/NetworkBehaviour.cs:1029)

The full sequence of events is logged as follows (we can provide additional details if necessary):

Notes:

• The exception is thrown before NetworkManager.OnClientConnectedCallback is handled on the Client.
• The Client also logs the following warning during the connection attempt, though I'm not sure if it's related to this issue: Trying to send ServerLogMessage to client 0 which is not in a connected state. It seems to log it even during a connection attempt that doesn't throw the exception.
• We tried to work around this issue by destroying and recreating the NetworkManager before each connection attempt, but it didn't have an effect on the exception.
• We've checked that there are no NetworkObjects in the scene when the Client calls NetworkManager.StartClient().
• Waiting for all Clients to finish loading the menu scene before starting a new server didn't have an effect on the issue.

Reproduce Steps

Difficult to reproduce. This is how we reproduce the error in our project:

1. Join a Steam Lobby with Host and two Clients in a menu scene.
2. Start a game as Host and load a level.
3. As soon as the level loads, set the server available for Clients in the Steam Lobby.
4. Automatically connect to Host as Clients as soon as the server becomes available and load the level.
5. As Host, shut down the server almost immediately when it loads a level.
6. As Clients, get disconnected from the Host and return to the menu scene.
7. Repeat the above steps.
8. As Client (with 200+ ms latency), notice that you sometimes get the NotServerException and the game breaks.

Actual Outcome

Client gets a NotServerException when connecting to a Host (trying to destroy a spawned NetworkObject).

Expected Outcome

Client doesn't get a NotServerException when connecting to a Host (by trying to destroy a spawned NetworkObject).

Environment

• OS: Windows 10
• Unity version: 2021.3.19f1
• Netcode version: 1.2.0

[NoelStephensUnity]

Are you using the community SteamNetworkingSockets?

If so, could you describe the shutdown sequence of the host in a bit more detail in regards to shutting down the host after it loads a level?

• Are you loading the level via the integrated scene management (i.e. NetworkManager.SceneManager) or are you handling scene loading and synchronization yourself?
  • On the host side, how are you determining the scene has been loaded?
• When shutting down the host do you just invoke NetworkManager.Shutdown immediately? (my assumption is yes but just confirming)

With 3rd party transports there can be alternate timing considerations that you might need to account for. The more common issue I have seen is that during the 3rd party transport shutdown pending outbound messages are not being flushed (sent) which can result in abnormal termination on the client side.

What you might consider doing (if you are using integrated scene management), as opposed to just directly invoking NetworkManager.Shutdown, is to account for clients in the middle of synchronization. The easiest way to detect this by subscribing to the NetworkSceneManager.OnSynchronize event and keep track of clients that are in the middle of synchronizing.

Here is one example of how to control the shutdown sequence in order to assure clients are disconnected prior to completely shutting down the transport on the host-server side. You would want to attach this to the same GameObject that the NetworkManager component is attached to:

using System.Collections;
using System.Collections.Generic;
using System.Linq;
using Unity.Netcode;
using UnityEngine;

public class SynchronizedShutdown : MonoBehaviour
{
    private NetworkManager m_NetworkManager;
    private List<ulong> m_ClientsSynchronizing = new List<ulong>();
    private bool m_CanShutdown;

    private void Start()
    {
        m_NetworkManager = GetComponent<NetworkManager>();
        m_NetworkManager.OnServerStarted += NetworkManager_OnServerStarted;
    }

    private void OnDestroy()
    {
        if (m_NetworkManager != null)
        {
            m_NetworkManager.OnServerStarted -= NetworkManager_OnServerStarted;
        }
    }

    private void NetworkManager_OnServerStarted()
    {
        m_NetworkManager.SceneManager.OnSynchronize -= SceneManager_OnSynchronize;
        m_NetworkManager.SceneManager.OnSynchronizeComplete -= SceneManager_OnSynchronizeComplete;
        m_NetworkManager.SceneManager.OnSynchronize += SceneManager_OnSynchronize;
        m_NetworkManager.SceneManager.OnSynchronizeComplete += SceneManager_OnSynchronizeComplete;
    }

    private void SceneManager_OnSynchronizeComplete(ulong clientId)
    {
        m_ClientsSynchronizing.Remove(clientId);
    }

    private void SceneManager_OnSynchronize(ulong clientId)
    {
        m_ClientsSynchronizing.Add(clientId);
    }

    public void ShutdownServer()
    {
        if (m_NetworkManager == null || !m_NetworkManager.IsServer)
        {
            return;
        }

        if (m_ClientsSynchronizing.Count > 0 || ConnectedClientCount() > 0)
        {
            StartCoroutine(OnShuttingDownServer());
        }
        else
        {
            InternalShutdown();
        }
    }

    private void InternalShutdown()
    {
        m_NetworkManager.SceneManager.OnSynchronize -= SceneManager_OnSynchronize;
        m_NetworkManager.SceneManager.OnSynchronizeComplete -= SceneManager_OnSynchronizeComplete;
        m_NetworkManager.Shutdown();
    }

    private int ConnectedClientCount()
    {
        return m_NetworkManager.ConnectedClientsIds.Where((c) => c != m_NetworkManager.LocalClientId).Count();
    }

    private IEnumerator OnShuttingDownServer()
    {
        m_CanShutdown = false;

        foreach (var clientId in m_ClientsSynchronizing)
        {
            if (clientId == m_NetworkManager.LocalClientId)
            {
                continue;
            }
            m_NetworkManager.DisconnectClient(clientId);
        }

        foreach (var clientId in m_NetworkManager.ConnectedClientsIds)
        {
            if (clientId == m_NetworkManager.LocalClientId)
            {
                continue;
            }
            m_NetworkManager.DisconnectClient(clientId);
        }

        var waitPeriod = new WaitForSeconds(0.1f);
        while (!m_CanShutdown)
        {
            yield return waitPeriod;
            m_CanShutdown = ConnectedClientCount() == 0 && m_ClientsSynchronizing.Count == 0;
        }

        InternalShutdown();
        yield break;
    }
}

The only other adjustment you would need to make is to invoke the SynchronizedShutdown.ShutdownServer as opposed to NetworkManager.Shutdown. (It is pseudo code...I haven't tested this under your specific scenario but it might provide you with insights as to how the issue you are experiencing could be avoided).

[A1win]

Yes, we're using SteamNetworkingSockets and integrated scene management. We do track when all connecting clients have finished loading the scene using Network SceneManager's OnLoadEventCompleted event, but the server is allowed to shut down even before that, which is when the issue sometimes arises.

So far, we've intentionally not disallowed the server from shutting down early, because that's always possible in cases where the server crashes, abruptly loses network connection, or when the server process is abruptly killed on the server machine. Ideally, the client should handle such cases gracefully since the client has no control over what happens on the server's end.

If there's currently no way to fix that on the client's end, we'll at least prevent manual shut down from within the game on the server like you suggested so that it's less likely to cause problems for clients. Thanks for the detailed pseudo code. 👍

[NoelStephensUnity]

@A1win
I am going to open a bug for the exception either case.
That section of code could check to see if the client is in the middle of shutting down and not throw exceptions under that condition. For the time being, the manual shut down prevention would be the best way to prevent that from happening when manually shutting down... and once the bug is fixed on the client side it should prevent that disruption of the call-stack so it doesn't break the client-side shutdown process. 👍