Some tests fail with "Could not validate timestamp: expired. Check system clock."

[mcepl]

When building python3-saml 1.10.1 package for openSUSE I get couple of tests failing with 'Could not validate timestamp: expired. Check system clock.'. We build in completely isolated environment so it is possible that we are missing some resource of the operating system, which is expected, but do you have any idea, what’s missing?

[    6s] ======================================================================
[    6s] FAIL: testIsInValidAudience (tests.src.OneLogin.saml2_tests.response_test.OneLogin_Saml2_Response_Test)
[    6s] ----------------------------------------------------------------------
[    6s] Traceback (most recent call last):
[    6s]   File "/home/abuild/rpmbuild/BUILD/python3-saml-1.10.1/tests/src/OneLogin/saml2_tests/response_test.py", line 1119, in testIsInValidAudience
[    6s]     self.assertIn('is not a valid audience for this Response', response_2.get_error())
[    6s] AssertionError: 'is not a valid audience for this Response' not found in 'Could not validate timestamp: expired. Check system clock.'
[    6s]
[    6s] ======================================================================
[    6s] FAIL: testIsInValidEncAttrs (tests.src.OneLogin.saml2_tests.response_test.OneLogin_Saml2_Response_Test)
[    6s] ----------------------------------------------------------------------
[    6s] onelogin.saml2.errors.OneLogin_Saml2_ValidationError: Could not validate timestamp: expired. Check system clock.
[    6s]
[    6s] During handling of the above exception, another exception occurred:
[    6s]
[    6s] Traceback (most recent call last):
[    6s]   File "/home/abuild/rpmbuild/BUILD/python3-saml-1.10.1/tests/src/OneLogin/saml2_tests/response_test.py", line 1010, in testIsInValidEncAttrs
[    6s]     response_2.is_valid(self.get_request_data(), raise_exceptions=True)
[    6s] AssertionError: "There is an EncryptedAttribute in the Response and this SP not support them" does not match "Could not validate timestamp: expired. Check system clock."
[    6s]
[    6s] ======================================================================
[    6s] FAIL: testIsInValidIssuer (tests.src.OneLogin.saml2_tests.response_test.OneLogin_Saml2_Response_Test)
[    6s] ----------------------------------------------------------------------
[    6s] onelogin.saml2.errors.OneLogin_Saml2_ValidationError: Could not validate timestamp: expired. Check system clock.
[    6s]
[    6s] During handling of the above exception, another exception occurred:
[    6s]
[    6s] Traceback (most recent call last):
[    6s]   File "/home/abuild/rpmbuild/BUILD/python3-saml-1.10.1/tests/src/OneLogin/saml2_tests/response_test.py", line 1191, in testIsInValidIssuer
[    6s]     response_3.is_valid(request_data, raise_exceptions=True)
[    6s] AssertionError: "Invalid issuer in the Assertion/Response" does not match "Could not validate timestamp: expired. Check system clock."
[    6s]
[    6s] ======================================================================
[    6s] FAIL: testIsInValidSessionIndex (tests.src.OneLogin.saml2_tests.response_test.OneLogin_Saml2_Response_Test)
[    6s] ----------------------------------------------------------------------
[    6s] onelogin.saml2.errors.OneLogin_Saml2_ValidationError: Could not validate timestamp: expired. Check system clock.
[    6s]
[    6s] During handling of the above exception, another exception occurred:
[    6s]
[    6s] Traceback (most recent call last):
[    6s]   File "/home/abuild/rpmbuild/BUILD/python3-saml-1.10.1/tests/src/OneLogin/saml2_tests/response_test.py", line 1220, in testIsInValidSessionIndex
[    6s]     response_2.is_valid(request_data, raise_exceptions=True)
[    6s] AssertionError: "The attributes have expired, based on the SessionNotOnOrAfter of the AttributeStatement of this Response" does not match "Could not validate timestamp: expired. Check system clock."
[    6s]
[    6s] ======================================================================
[    6s] FAIL: testIsInValidSubjectConfirmation (tests.src.OneLogin.saml2_tests.response_test.OneLogin_Saml2_Response_Test)
[    6s] ----------------------------------------------------------------------
[    6s] onelogin.saml2.errors.OneLogin_Saml2_ValidationError: Could not validate timestamp: expired. Check system clock.
[    6s]
[    6s] During handling of the above exception, another exception occurred:
[    6s]
[    6s] Traceback (most recent call last):
[    6s]   File "/home/abuild/rpmbuild/BUILD/python3-saml-1.10.1/tests/src/OneLogin/saml2_tests/response_test.py", line 1311, in testIsInValidSubjectConfirmation
[    6s]     response.is_valid(request_data, raise_exceptions=True)
[    6s] AssertionError: "A valid SubjectConfirmation was not found on this Response" does not match "Could not validate timestamp: expired. Check system clock."
[    6s]
[    6s] ----------------------------------------------------------------------
[    6s] Ran 255 tests in 0.727s
[    6s]
[    6s] FAILED (failures=5)

Complete log with all details of steps executed and packages used.

[cybojanek]

It looks like the test data may need to be updated.

It was updated before to work in 2020.

[pitbulk]

The test were already updated at #269

[vcunat]

A similar looking failure surfaced again in the past few days. (latest version = 1.15.0)

FAIL: testIsInvalidIssuer (tests.src.OneLogin.saml2_tests.logout_request_test.OneLogin_Saml2_Logout_Request_Test)
Tests the is_valid method of the OneLogin_Saml2_LogoutRequest
----------------------------------------------------------------------
onelogin.saml2.errors.OneLogin_Saml2_ValidationError: Could not validate timestamp: expired. Check system clock.)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/build/source/tests/src/OneLogin/saml2_tests/logout_request_test.py", line 329, in testIsInvalidIssuer
    with self.assertRaisesRegex(Exception, 'Invalid issuer in the Logout Request'):
AssertionError: "Invalid issuer in the Logout Request" does not match "Could not validate timestamp: expired. Check system clock.)"

I see some success on May 3 and failure on May 11 (and today), so I expect expiration happened in that range.

[pitbulk]

@vcunat It seems some test fails know because some SAML Messages I was using as payloads for the test failed.

Fixed them here:
bd65578

[BobocIonut]

It looks like the test data may need to be updated again