You have the timeline wrong. The fighting in north Africa started in 1939 (or maybe 1935, depending how much is ‘background’ and how much is the subject). Even some German forces were committed prior to Russia being invaded. The threat to the world, and to the Middle East and Caucasian oil fields, was there well before Russia was forced to fight Germany, so it was not a matter of Germany diverting troops from Russia to North Africa so much as a matter of Germany putting the troops in Russia because the approach through either Turkey or Egypt was not open.
The route from Libya to the Caucasus is less than the route from Poland, and the resistance far less. From many military planner’s views, it was an obvious choice with many side benefits. Thankfully, Corporal Hitler was so focused on his hatred of Bolshevism that he didn’t take grand strategy on the global scale seriously. For Britain and even for Mussolini, it was natural to look at a globe of the world and plot accordingly. Some in the German high command saw it, too, but their opinion was unwelcome.
The North African fight was not the push-over some imagine. The wide-spread British Empire and allies in the region, covered Africa, the Middle-East and through to Burma (British forces were providing support to China well before open war was declared on them by Japan), as well as into southern Europe through Greece, fighting both Austrian, German and Greek communist forces. The Axis troops pushed within a day’s drive of the Nile before being stopped at that most famous of non-places, El-Alamein. Surrendering the Suez was never an option for the British but holding it took some difficult and complex operations, with many of the casualties not even making it from Britain to land in Egypt.