How did ships replace a broken mast while at sea in the age of sailing?
The first thing to realize is that, in the Age of Sail, sailors were incredibly skilled at working with rope and wood. It was an art that was brought to near perfection, at a level of complexity and power that would be flatly unbelievable to a landsman (and to most sailors) today. Where people marvel and try to invent crazy magic explanations for things like the Easter Island mo‘ai or the multi-ton blocks of the Egyptian pyramids, a sailor from those days would simply shrug and wonder what the fuss was about… it would just be a day’s work, to him.
As a case in point, please consider this illustration:
Just as a bit of detail for the kind of work that was involved: that angled line leading forward from the loop around the mast (the fore topmast stay) is most likely wire cable… but since they didn’t have stainless steel in those days, it was necessary to serve and parcel it. Meaning that someone — lots of someones — would carefully lay thin tarred line into the spaces between the strands, tar it all, wrap it in canvas, paint that, and then wrap the entire thing, every inch of it, with serving line. That is, literally wrap several hundred feet of cable with string along its entire length, in such a way that nothing could be seen through that wrapping.
Oh, and — that cable would again be tarred before being put into service, and re-tarred on a regular basis. Some of them were cut open in our times, a couple of hundred years after they were made, and the metal looked brand new. And: there were dozens, if not hundreds, of served-and-parcelled cables about a large ship. Thousands of man-hours just in that detail alone.
So, having said all that: it was a lot of work, but hoisting and securing the topmasts was a standard procedure aboard ships. Every able sailor knew exactly how it was done, and had usually participated in doing it again and again. “Fishing” a broken mast was not all that different.
Square rigs, unlike modern masts, didn’t require any precise machining; as long as you could hoist the yards and bend the sails onto them, you were fine. Most ships also had multiple masts — your question shows that you’ve just realized one good reason for that. Another reason was that it kept the maximum effort necessary for hoisting each sail to a manageable level, in an age where mast winches didn’t exist.
If the mast broke well above the deck — as it usually would, due to shroud failure — the stump would normally be cleaned up with a bit of axe work and have a mortise cut into it, and the broken piece (also cleaned up and mortised) would be hoisted to the top, butt-end first. Then, the two would be lashed together — right at the top of the stump and somewhat below the butt of the other piece — and the new topmast would be pivoted right-side up, by leading a line from its lower end to a winch on deck through a purchase (a set of blocks that provided leverage) and using the lashing as a pivot point. Then, the two would be pinned together through the mortised holes — usually a pair of them — and woolded (lashed together with rope or chain), usually with a fish (one or more extra pieces of wood running parallel to the masts to strengthen the join.) In fact, battleships would often sacrifice one of their smaller masts to make a fish for their mainmast, because this would allow them to quickly fix the problem and return to battle.
This image demonstrates woolding a large yard (the spar to which the sail is bent, or attached, and which runs perpendicular to the mast) with chain and a single fish plus battens:
I will note that the whole thing became much easier if a mast other than the main was broken; the main could then be used for the hoisting. By the same token, when the main was broken (fortunately a very rare occurrence), life was pure hell and danger every moment — because having the main fall could destroy the entire ship. As, incidentally, could any mast that went overboard — a true emergency in any but the mildest weather, because it could put a hole in the hull at any moment! Lashing the broken mast and hoisting it back inboard — or cutting it loose, if it was too dangerous — was one of those awful decisions to which there was rarely a perfectly correct answer.
Pretty exciting stuff. And something that every sailor should be aware of, to a reasonable degree — because much of it still applies to sailing today.
This new ship here is fitted according to the reported increase of knowledge among mankind. Namely, she is cumbered end to end, with bells and trumpets and clock and wires… she can call voices out of the air of the waters to con the ship while her crew sleep. But sleep Thou lightly. It has not yet been told to me that the Sea has ceased to be the Sea.
— Rudyard Kipling