Uranium is not rare. See for example: http://www.wise-uranium.org/umaps.html?set=ures. There is lots of it available in South America, Africa, the Middle East and Asia. Iran alone has an estimated 1100t that they can mine at reasonable price.
Restricting access to it can only be an important factor for countries that don’t have natural deposits, but there are many that do.
Moreover, plutonium doesn’t really occur naturally. It is produced through burning Uranium in a nuclear reactor, which means you need expertise and technology for it. Restricting access to it must mean restricting access to technology, then, which contradicts your statement.
I think the two properties I identify are more important and general than just restricting access to it through export and sales. Making it harder to access uranium for countries that don’t have deposits, and restricting access to technology for enriching uranium and creating plutonium both contribute to the long ramp-up time.
Thanks for your comment. It was good to consider resource access in slightly more detail.