Stale NetworkList event sent to client when its containing object is re-shown.

[ezoray]

Description

I have a network object observed by a client containing a network list of structs. On direction by the client a timer is started on the server to remove the entries from the list from the bottom at half a second intervals. At any point the client can choose to deselect the object and the server hides the object from the client and continues to remove entries from the list until all are removed, all is good at this point.

The problem is, the next time the object is shown to the client a pending network list event message is also sent to the client to remove an entry from the network list and because the list is empty an exception is thrown.

Reproduce Steps

1. Spawn object on client and populate network list
2. Start timer to remove list entries one at a time
3. Server hides object from client
4. All entries in list removed, timer stops
5. Server shows object to client
6. NetworkList exception message logged on client

Actual Outcome

"Exception: Shouldn't be here, index is higher than list length"

Expected Outcome

No network list event message is sent to client.

Environment

• OS: macOS Big Sur
• Unity Version: [e.g. 2020.3]
• Netcode Version: 1.1.0

Additional Context

This happens consistently in my main project, I wasn't able to reproduce it in a test project but it was only a simple test.

Both list.RemoveAt and list.Remove are producing the same error:

Exception: Shouldn't be here, index is higher than list length Unity.Netcode.NetworkList1[T].ReadDelta (Unity.Netcode.FastBufferReader reader, System.Boolean keepDirtyDelta) (at Library/PackageCache/com.unity.netcode.gameobjects@1.1.0/Runtime/NetworkVariable/Collections/NetworkList.cs:365) Unity.Netcode.NetworkVariableDeltaMessage.Handle (Unity.Netcode.NetworkContext& context) (at Library/PackageCache/com.unity.netcode.gameobjects@1.1.0/Runtime/Messaging/Messages/NetworkVariableDeltaMessage.cs:200) Unity.Netcode.MessagingSystem.ReceiveMessage[T] (Unity.Netcode.FastBufferReader reader, Unity.Netcode.NetworkContext& context, Unity.Netcode.MessagingSystem system) (at Library/PackageCache/com.unity.netcode.gameobjects@1.1.0/Runtime/Messaging/MessagingSystem.cs:457) Unity.Netcode.MessagingSystem.HandleMessage (Unity.Netcode.MessageHeader& header, Unity.Netcode.FastBufferReader reader, System.UInt64 senderId, System.Single timestamp, System.Int32 serializedHeaderSize) (at Library/PackageCache/com.unity.netcode.gameobjects@1.1.0/Runtime/Messaging/MessagingSystem.cs:387) UnityEngine.Debug:LogException(Exception) Unity.Netcode.MessagingSystem:HandleMessage(MessageHeader&, FastBufferReader, UInt64, Single, Int32) (at Library/PackageCache/com.unity.netcode.gameobjects@1.1.0/Runtime/Messaging/MessagingSystem.cs:391) Unity.Netcode.MessagingSystem:ProcessIncomingMessageQueue() (at Library/PackageCache/com.unity.netcode.gameobjects@1.1.0/Runtime/Messaging/MessagingSystem.cs:407) Unity.Netcode.NetworkManager:OnNetworkEarlyUpdate() (at Library/PackageCache/com.unity.netcode.gameobjects@1.1.0/Runtime/Core/NetworkManager.cs:1581) Unity.Netcode.NetworkManager:NetworkUpdate(NetworkUpdateStage) (at Library/PackageCache/com.unity.netcode.gameobjects@1.1.0/Runtime/Core/NetworkManager.cs:1513) Unity.Netcode.NetworkUpdateLoop:RunNetworkUpdateStage(NetworkUpdateStage) (at Library/PackageCache/com.unity.netcode.gameobjects@1.1.0/Runtime/Core/NetworkUpdateLoop.cs:185) Unity.Netcode.<>c:<CreateLoopSystem>b__0_0() (at Library/PackageCache/com.unity.netcode.gameobjects@1.1.0/Runtime/Core/NetworkUpdateLoop.cs:208)

[jeffreyrainy]

Thanks for submitting this issue!

I managed to find a case when updates might be missed. But this is only for dynamically spawned object, and it would not cause an exception, only prevent updates from showing for a while. So, I'm puzzled.

Start timer to remove list entries one at a time

At what point during the frame is this timer running? Is this a coroutine, a thread, a check during Update(), during FixedUpdate(), something else altogether? Most likely, this is happening at a time we didn't expect. So, the most useful would be a repro sample, or some description of the timer script you are using.

Please let me know.

[ezoray]

Hi @jeffreyrainy, this is running in Update(), there's a MonoBehaviour that on each update calls a TimerManager routine which checks against deltaTime if enough time has passed to run any current timed events.

This is the MonoBehaviour, I think I grabbed it from the forums:

    public class UpdateCallManager : MonoBehaviour
    {
        private static UpdateCallManager instance;
        private Action updateCallback;

        private void Start()
        {
            Debug.Log("UpdateCallManager Start");
        }

        private void Update()
        {
            updateCallback();
        }

        public static void RemoveUpdateCallback(Action updateFunction)
        {
            instance.updateCallback -= updateFunction;
        }

        public static void AddUpdateCallback(Action updateFunction)
        {
            if (instance == null)
            {
                instance = new GameObject("[Update Caller]").AddComponent<UpdateCallManager>();
            }
            instance.updateCallback += updateFunction;
        }
    }

and this is the TimerManager:

public class TimerManager : IDisposable
    {
        uint timerUid = 0;
        List<TimerDetail> timerDetails = new List<TimerDetail>();

        public TimerManager()
        {
            UpdateCallManager.AddUpdateCallback(OnUpdate);
        }

        public void KillAllTimedFunctions()
        {
            Debug.Log("TimerManager KillAllTimedFunctions count: " + timerDetails.Count);

            for (int i = timerDetails.Count -1; i >= 0; i--)
            {
                timerDetails.RemoveAt(i);
            }
        }

        public void StopFunctionTimer(uint timerId)
        {
 //           Debug.Log("TimerManager StopFunctionTimer timerId: " + timerId);

            int index = timerDetails.FindIndex(timerDetail => timerDetail.TimerId == timerId);

            if (index != -1)
            {
                TimerDetail timerDetail = timerDetails[index];

                EndFunctionTimer(index, timerDetail);
            }
            else
            {
                Debug.LogWarning("TimerManager StopFunctionTimer timerId not found: " + timerId);
            }
        }

        private void EndFunctionTimer(int index, TimerDetail timerDetail)
        {
            Debug.Log("TimerManager EndFunctionTimer index: " + index + " timerDetail: " + timerDetail);

            timerDetails.RemoveAt(index);

            if (timerDetail.TimerFlags.HasFlag(FunctionTimerFlags.EndFunction))
            {
                timerDetail.EndFunction();
            }

            Debug.Log("TimerManager EndFunctionTimer timer count: " + timerDetails.Count);
        }

        private void OnUpdate()
        {
            for (int index = timerDetails.Count - 1; index >= 0; index--)
            {
                TimerDetail timerDetail = timerDetails[index];

                if (TryExecuteTimedFunction(timerDetail, Time.deltaTime))
                {
                    EndFunctionTimer(index, timerDetail);
                }
            }
        }

        private bool TryExecuteTimedFunction(TimerDetail timerDetail, float deltaTime)
        {
            bool removeFunctionTimer = false;

            timerDetail.TimePassed += deltaTime;

      //      Debug.Log("TimerManager TryExecuteTimedFunction deltaTime: " + deltaTime + " timePassed: " + timerDetail.TimePassed);

            if (timerDetail.TimePassed > timerDetail.Interval)
            {
                timerDetail.TimePassed %= timerDetail.Interval;

                if (timerDetail.TimerFlags.HasFlag(FunctionTimerFlags.TimedFunction))
                {
     //               Debug.Log("TimerManager TryExecuteTimedFunction run TimedFunction: " + timerDetail.TimerId);
                    removeFunctionTimer = timerDetail.TimedFunction();
                }

                if (--timerDetail.Count == 0)
                {
                    removeFunctionTimer = true;
                }
            }

            return removeFunctionTimer;
        }

        public uint AddFunctionTimer(FunctionTimerDetail functionTimerDetail)
        {
            TimerDetail timerDetail = new TimerDetail(timerUid, functionTimerDetail);

            timerDetails.Add(timerDetail);

             Debug.Log("TimerManager AddFunctionTimer: " + timerDetail);

            return timerUid++;
        }

        public void Dispose()
        {
            UpdateCallManager.RemoveUpdateCallback(OnUpdate);
        }
    }

[jeffreyrainy]

Hi,

I'm failing to repro this on develop. I tried:

• Adding and removing from NetworkList
• Showing and hiding objects
• Using dynamically spawned and in-scene placed objects
• Triggering the update to occur from a different behaviour, in Update and LateUpdate

The best way for us to debug this with you would be if you could provide a simple project that repros the issue.

Below, I'm adding two scripts. Maybe they can be of help to find a repro case? They are just meant as debug tool that might be of use.

SomeNewScript, a new script to put on some number of NetworkObject:

using System.Collections.Generic;
using Unity.Netcode;
using UnityEngine;

public class SomeNewScript : NetworkBehaviour
{
    private static List<SomeNewScript> s_AllInstances;
    private NetworkList<int> m_NetworkList;

    void Awake()
    {
        if (s_AllInstances == null)
        {
            s_AllInstances = new List<SomeNewScript>();
        }
        s_AllInstances.Add(this);

        m_NetworkList = new NetworkList<int>();
        m_NetworkList.OnListChanged += Changed;
    }

    void Changed(NetworkListEvent<int> le)
    {
        Display();
    }

    void Display()
    {
        var list = "";
        foreach (var i in m_NetworkList)
        {
            list += i.ToString() + ", ";
        }
        Debug.Log($"List contains {list}");
    }

    public override void OnNetworkSpawn()
    {
        Display();
        base.OnNetworkSpawn();
    }

    public static void AsyncUpdateAll()
    {
        foreach (var instance in s_AllInstances)
        {
            instance.AsyncUpdate();
        }
    }

    void AsyncUpdate()
    {
        if (Input.GetKeyDown(KeyCode.P))
        {
            if (IsServer)
            {
                m_NetworkList.Add(1);
                m_NetworkList.Add(2);
                m_NetworkList.Add(3);
                m_NetworkList.Add(4);
                m_NetworkList.Add(5);
                m_NetworkList.Add(6);
                m_NetworkList.Add(7);
                m_NetworkList.Add(8);
                Display();
            }
        }

        if (Input.GetKeyDown(KeyCode.S))
        {
            if (IsServer && !GetComponent<NetworkObject>().IsNetworkVisibleTo(1))
            {
                GetComponent<NetworkObject>().NetworkShow(1);
                Debug.Log("Shown");
                Display();
            }
        }

        if (Input.GetKeyDown(KeyCode.H))
        {
            if (IsServer && GetComponent<NetworkObject>().IsNetworkVisibleTo(1))
            {
                GetComponent<NetworkObject>().NetworkHide(1);
                Debug.Log("Hidden");
                Display();
            }
        }

        if (Input.GetKeyDown(KeyCode.R))
        {
            if (IsServer)
            {
                m_NetworkList.Remove(m_NetworkList[m_NetworkList.Count - 1]);
                Display();
            }
        }

        if (Input.GetKeyDown(KeyCode.A))
        {
            if (IsServer)
            {
                Debug.Log("Remove then hide");
                m_NetworkList.Remove(m_NetworkList[m_NetworkList.Count - 1]);

                GetComponent<NetworkObject>().NetworkHide(1);
                Debug.Log("Hidden");

                Display();
            }
        }

        if (Input.GetKeyDown(KeyCode.B))
        {
            if (IsServer)
            {
                Debug.Log("Hide then remove");
                GetComponent<NetworkObject>().NetworkHide(1);
                Debug.Log("Hidden");

                m_NetworkList.Remove(m_NetworkList[m_NetworkList.Count - 1]);
                Display();
            }
        }
    }
}

SomeNewTrigger, a script to put on any object in your scene

using UnityEngine;

public class SomeNewTrigger : MonoBehaviour
{
    // Update is called once per frame
    void Update()
    {
        SomeNewScript.AsyncUpdateAll();
    }
}

Then, on the host, you can press:
P to fill a NL with numbers
R to remove the last number
H to hide the object to client 1
S to show the object to client 1
A to hide and remove in the same update
B to remove and hide in the same update
(I also tried locally the other combos of add/remove hide/show in both orders)

So far, this turned up no repro. If you get a way to repro the issue, please provide the project, or the set of steps with the script above.

[ezoray]

Okay thanks Jeffrey, I'll see what I can do about reproducing it in a test environment.

[ezoray]

@jeffreyrainy, I was able to get what appear to be the same errors in testing.

I lifted the classes out of the main project and wired up a few buttons. To produce the error I started the timer removing entries from the list and have another button for toggling the visibility of the object to the client. Hammering that button whilst the timer removes the entries is producing a lot of errors relating to the network list.

I've attached the test files should you need to try it.
NetworkListTest.unitypackage.zip

[jeffreyrainy]

Hi @ezoray ,

I was not able to open your .zip. Well, it opened, but I had no idea what to do with those files without code.

I redid my test classes, included below. I'm now using your UpdateCallManager (even though it's unnecessary complexity, IMO), so that the update in a static instance triggers the component updates via an action. You can attach any number of SomeNewScript to various NetworkObjects and you can put a SomeNewTrigger anywhere in the scene.

I left it running for a good while and no exception occurred. It randomly adds and removes from NetworkList, while showing and hiding objects.

What I can suggest:

• Try from the develop branch, maybe it's another fix that got in
• Try to repro in a smaller scale project that you can share fully

SomeNewScript.cs

using System.Collections.Generic;
using Unity.Netcode;
using UnityEngine;

public class SomeNewScript : NetworkBehaviour
{
    private static List<SomeNewScript> s_AllInstances;
    private NetworkList<int> m_NetworkList;

    void Awake()
    {
        if (s_AllInstances == null)
        {
            s_AllInstances = new List<SomeNewScript>();
        }
        s_AllInstances.Add(this);

        m_NetworkList = new NetworkList<int>();
        m_NetworkList.OnListChanged += Changed;
    }

    void Changed(NetworkListEvent<int> le)
    {
        Display();
    }

    void Display()
    {
        var list = "";
        foreach (var i in m_NetworkList)
        {
            list += i.ToString() + ", ";
        }
        Debug.Log($"List contains {list}");
    }

    public override void OnNetworkSpawn()
    {
        Display();
        base.OnNetworkSpawn();
    }

    public override void OnNetworkDespawn()
    {
        if (!IsServer)
        {
            Debug.Log("I got despawned");
        }
    }

    public static void AsyncUpdateAll()
    {
        foreach (var instance in s_AllInstances)
        {
            instance.AsyncUpdate();
        }
    }

    void AsyncUpdate()
    {
        if (IsServer)
        {
            for (int i = 0; i < 3; i++)
            {
                if (Random.Range(0, 5) > 3 && m_NetworkList.Count < 4)
                {
                    m_NetworkList.Add(Random.Range(0, 10));
                }

                if (Random.Range(0, 5) > 3)
                {
                    m_NetworkList.Remove(Random.Range(0, 10));
                }

                if (Random.Range(0, 5) > 3 &&
                    !GetComponent<NetworkObject>().IsNetworkVisibleTo(1))
                {
                    GetComponent<NetworkObject>().NetworkShow(1);
                }

                if (Random.Range(0, 5) > 3 &&
                    GetComponent<NetworkObject>().IsNetworkVisibleTo(1))
                {
                    GetComponent<NetworkObject>().NetworkHide(1);
                }
            }
        }
    }
}

SomeNewTrigger.cs

using System;
using UnityEngine;

public class UpdateCallManager : MonoBehaviour
{
    private static UpdateCallManager instance;
    private Action updateCallback;

    private void Start()
    {
        Debug.Log("UpdateCallManager Start");
    }

    private void Update()
    {
        updateCallback();
    }

    public static void RemoveUpdateCallback(Action updateFunction)
    {
        instance.updateCallback -= updateFunction;
    }

    public static void AddUpdateCallback(Action updateFunction)
    {
        if (instance == null)
        {
            instance = new GameObject("[Update Caller]").AddComponent<UpdateCallManager>();
        }
        instance.updateCallback += updateFunction;
    }
}

public class SomeNewTrigger : MonoBehaviour
{
    private void Awake()
    {
        UpdateCallManager.AddUpdateCallback(OtherUpdate);
    }

    // Update is called once per frame
    void OtherUpdate()
    {
        SomeNewScript.AsyncUpdateAll();
    }
}

[ezoray]

Hi @jeffreyrainy, UpdateCallManager is meant for adding multiple timers where multiple game instances are running and each attaches its own timer. I did change the invocation, it should be updateCallback?.Invoke().

The files I attached are added to a project via Assets -> Import Package. I did test and run the import in a new project before uploading so hopefully it should be straight forward.

I did try the develop branch but no joy. Hopefully if you get the attached files running you'll be able to reproduce the issue.

[jeffreyrainy]

Best I can get from trying to import the provided package is:

and nothing gets added. 🤷

[ezoray]

You said you already opened it and saw the files, are they already imported, as that's the message I get when the files already exist in the project.

I'll see if I can zip up the whole project.

[ezoray]

This is the project minus the Library folder (I think it's missing ParrelSync).

AssetImportTest.zip