We document that isolated cities have lower skill wage premia in American census data. To explain this correlation and other correlations between population and wages, we build an equilibrium empirical model that incorporates high and low-skill labor, costly trade, and both agglomeration and congestion forces. Our paper bridges the gap between the economic geography literature which abstracts from inequality, and the spatial inequality literature which abstracts from geography. We find that geographical location explains 16.5% of observed variation in the skill wage premium across American cities. We use our model to simulate counterfactual trade and technology shocks. Reductions in domestic trade costs benefit both skill groups but low-skill workers benefit more.