There were about 4 million American citizens in 1777, around 700,000 lived in Philadelphia. Constantinople in 1500 had around 70,000 in comparison.
The population of Rome dropped from 2 million at it's peak to around 35,000 in the 13th century, Rome didn't reach a population of a million again until the 1930s.
I don't think you have to worry about a lack of buildings.