.Google's Gary Illyes discloses Googlebot gathers web links as opposed to "complying with" them, negating official information.Googlebot gathers web links rather than following all of them in real-time.Gary Illyes discussed this details in Google's 'Explore Off The Record' podcast.Google's paperwork negates the genuine spider habits.