I will need to do some searching to see if I can still find references to crawlers. I think one or more references have disappeared over the past 5+ years.

Crawlers are not super efficient. They are also known to cause a lot of traffic (slowing down of the network.)

I can tell you the network is particularly small these days.

Edit: Responded to via PM.