its a long explanation, but basically there were many many other children having children since and after Cain. the way the math works out, there were at a very minimum 5000 other grown people and the maximum thousands more than that.

remember, people lived to be hundreds of years old.
population growth was rapid, very rapid.

here is a link that might help:
Observations about population growth

The fact that Cain, the son of Adam, built a city tells us much about the increase in population in the early history of mankind. If a city was built in the lifetime of a son of the very first man, it follows that population grew rapidly and was quickly civilized (in contrast to the views of evolutionists).

God had commanded Adam and Eve to reproduce and fill the earth (Gen. 1:28). Men in that time, before the flood, lived to great ages (see chapter 5). Adam, for example, lived to be 930 years old before he died (5:5), and most of his descendants before the flood lived about 900 years. If there are no gaps in the genealogies, this means Adam would have been alive following the birth of Lamech, father of Noah, 8 generations later! Hence, although they eventually died, people lived many years before they died. This greatly multiplied the number of people living on earth because, at any one time, there were many generations still living.

Further, men were capable of having many children, and large families were common. Noah was having children at age 500 (5:32). All men in chap. 5 are recorded as having "sons and daughters." In such long lifetimes with long periods of fertility, many children could be born.

Using conservative estimates, Morris estimates (p. 143) that, by the time Cain died, there could easily have been 120,000 people on the earth (certainly enough for there to be cities). By the time of Noah there could easily have been seven billion - more than on earth today! Do not think of Cain, Adam, and other such people as walking around on a bare, lonesome, uninhabited earth. The obeyed God's command to reproduce and fill the earth.

[link to www.gospelway.com]