Truly open borders might prove unworkable, especially in countries with welfare states, and kill the goose laying the proverbial golden eggs; in this regard Mr. Clemens's analysis may require some modification. Still, we should be obsessing over how many of those trillions can actually be realized.
Preferring your fellow countrymen's interests over those of foreigners is not equivalent to saying that other people do not count. It is saying that, when not fully compatible, their interests do not count equally.