Yes, if all that matters in your life is where you go to school then yes they were held back from achieving that.

For a lot of people higher education is seen as the mantle to climb the socioeconomic ladder and on average the Asian or white kid who was competitive but didn’t get to go to harvard will achieve similar outcomes regardless.