Zero is a number, infinity is not a number.
Zero is an integer number, therefore it is also a rational number, a real number. According to some if not most definitions of natural numbers, zero is not a natural number, because natural numbers start with 1.
In the set of natural numbers, subtraction does not always result in a natural number, because in the natural numbers there are no negative numbers. This doesn’t make negative numbers “not numbers”. Division is also defined differently. In the set of natural numbers, division is done with remainders.
In the set of integers, subtraction is defined as the addition of a negative number, e.g. 3–5 is actually 3+(-5). Division is still done with remainders.
In the set of rational numbers, fractions are added, so division is defined as multiplication by the inverse, e.g 3:5 is actually 3*(1/5). Still, not all numbers are representable using fractions, for example the square root of 2 or pi.
In the real numbers, are numbers on the real line are representable, and between any two real numbers there are infinitely many other real numbers.
(Complex numbers are then another thing, because they are two dimensional and represented on a plane.)
About division by zero: division by zero is not defined. This doesn’t make zero “not a number”, in the same way as subtraction not being defined in the natural numbers doesn’t make -5 “not a number”.
Example: “12/0” is not a number, it is undefined. But “0” is a number.
Infinity instead is not a number because it is not defined like you defined it. Infinity is defined as something without bounds or larger than any number. But it is not a number. If you assume that infinity is a number, you can prove things that are definitely not true, thus reaching a contradiction and proving that your assumption was wrong (this is called “proof by contradiction”). Here is an example of such a proof:
> assume infinity is a number
> consider that infinity + 1 = infinity
> but also infinity + 2 = infinity
> infinity + n = infinity
> therefore, you can create any equation such as:
> infinity + 2 = infinity + 1
> 2 = 1
> this is clearly wrong.
You can obtain such equalities for any number.
Many worse things can happen when you assume that infinity is a number.
When you are computing limits you might think that you are treating infinity as a number, but this is not true. What you are doing is using a simplified syntax which is derived by computations with functions, successions, infinite sums and so on.