Take the second definition of elite - " a group or class of people seen as having the most power and influence in a society, especially on account of their wealth or privilege."
To me, "elite" collectors are collectors who mingle with the elitist people of blue chip galleries and auction houses - the ones having the most influence in the market - and they do solely for interest.
A guy who buy from blue chip galleries/auction houses becase he can afford it and does it without mingling with the all world of blue chip galleries and not caring much about the trickeries, hocus pocus, return on investment, etc. is not an elite collector.