The government only goes after piracy groups; they don't bother with individuals, preferring to let the civil courts handle individual pirates through lawsuits.
But the ESA isn't going to sue you for downloading old games because they'd have a tough time proving the profitability of games which they're no longer manufacturing or selling.
Which is another reason they don't bother suing, since it would actually cost them money (legal fees) rather then save them money.
What the ESA does, is if they catch you hosting or sharing roms or ISO's from their "protected" list, they send you cease and desist (C & D) letter ordering you to remove those "protected" games.
Which is why the big rom sites often don't have Mario or Zelda games because the ESA sent them a C & D demanding that they remove those games.
Ironically though, you'll find Mario and Zelda games on crappy little GeoCities sites, because the ESA can't search for every site on the "Net that has rom downloads.
And then there's the curious fact that there are well-known GBA rom sites that have lasted for years without any interference from the ESA or Nintendo, even though they have Mario and Zelda GBA games. Same with NDS rom sites.
To be honest, I can't figure out the ESA's strategy, but then, who can ever understand how lawyers think anyway?
In any case, you shouldn't worry about it.
So, keep your collection, and just ignore your brother, who's just being an ass.