The world in the early 19th century was a much different and brutal place than present America. Slavery was rampant all over and not just with Africans. Africans enslaved other Africans and sold them to Portugese traders. Muslims in what we know as the Middle East, Asians in the Orient and the Irish were all victims as well during this period. I do not hold those men accountable under the standard of present day America.
Yes, those words were written By F. S. Key as he sat on a British ship bombing Baltimore during the war of 1812 and were not meant for blacks of that era. But it certainly does today because you live in the most free multiethnic society in the history of mankind with opportunities afforded you abundant in every sector of society from educational opportunities to housing opportunites to career opportunities.
And the three fifths concept was a compromise between northern and southern elected officials where the south wanted slaves counted as whole as a white citizen in order to gain more power in elected seats and electoral votes. The northern politicians wanted less power for the southern slave holding states and did not want the blacks, who had zero power, to be included when they had no voice so they compromised on 3/5.