SAML Assertion NotBefore,NotOnOrAfter problem due to unsynced clocks : Explained
A SAML Response is sent by the Identity Provider(IDP) to the Service Provider(SP) if the user succeeds in the authentication process. A sample SAML response is given below.
NotBefore & NotOnOrAfter in Subject
In a SAML response, the Subject element has the SubjectConfirmation complex type. It specifies additional data that allows the subject to be confirmed or constrains the circumstances under which the act of subject confirmation can take place . In the SubjectConfirmation, Subject confirmation takes place when a relying party seeks to verify the relationship between an entity presenting the assertion and the subject of the assertion’s claims.SubjectConfirmation MAY includes attributes NotBefore and NotOnOrAfter 
NotBefore is a time instant before which the subject cannot be confirmed and NotOnOrAfter is a time instant at which the subject can no longer be confirmed.
NotBefore & NotOnOrAfter in Condition
The Condition element also MAY contain the attributes NotBefore and NotOnOrAfter. NotBefore Specifies the earliest time instant at which the assertion is valid and NotOnOrAfter Specifies the time instant at which the assertion has expired 
NotBefore , NotOnOrAfter problem
As shown in the above image, here the both system clocks are not synced and IDP clock is 60s faster than the SP clock, During this situation, if a SAML response is generated in the IDP, response will contain a NotBefore time stamp which is 60s greater than the SP time stamp, so when the assertion occurs in the SP with the SAML response, assertion will fail since the SP time is less than NotBefore time.
But for the same scenario, if the network delay between IDP and the SP is more than 60s, the response will reach the SP after the NotBefore time stamp, so assertion will not fail.
In the second scenario, IDP clock is 60s slower than the SP clock, so the SAML response NotBefore time stamp is always less than the SP time stamp and assertion will not fail due to the not before condition.
But lets say SP has a response validity period as 100 s and network delay between IDP and the SP is more than 40s
response validity period < IDP clock difference + network delay
So assertion will fail because of the NotOnOrAfter condition.
Solving the problem
Even though getting the clocks in sync is a solution to this problem, for the inevitable small drifts in clocks this is not always practical. As a solution to this, consumer of the time stamp should allow for a little clock skew to account for small clock drifts that naturally occur. In this case, SP is the consumer, so the SP system should be implemented with a clock skew.