tekeous@usenet.lol to Lemmy Shitpost@lemmy.world · 3 months agorobots.txt is a suggestionusenet.lolimagemessage-square22fedilinkarrow-up1570arrow-down14
arrow-up1566arrow-down1imagerobots.txt is a suggestionusenet.loltekeous@usenet.lol to Lemmy Shitpost@lemmy.world · 3 months agomessage-square22fedilink
minus-squareekZepp@lemmy.worldlinkfedilinkEnglisharrow-up15·3 months agoAll this robot.txt stuff “perplex” me.
All this robot.txt stuff “perplex” me.