How to fix high page load time for single page app in google analytics ?

Google analytic only take 1% of all the page views as sample to calculate page load time.

For single page applications. First time page load is way more then successive page loads. Thus if we increment the sample size for page load to 100% or less it should show some improvement.
You can do that by adding siteSpeedSampleRate: 100 to google analytics code.
Here is how it should look like.

ga('create', 'UA-XXXXXXXX', {
cookieDomain: 'example.com',
siteSpeedSampleRate: 100
});

Note that page load time includes DOM and rendering time. You can use newrelic to validate actual page load time and optimize your application accordingly.