Dignity nobility high-quality state highness standing class caste standing superiority precedence fashion difference dominance preferment primacy nobleness gentleness gentility preeminence loftiness station grandness txt file is then parsed and can instruct the robot as to which pages are not to generally be crawled. As being a online search engine crawler https://www.youtube.com/watch?v=3GHWK6QaT2M