|
|
The Webmaster Dictionary
Standard For Robot Exclusion |
World Wide Web Robot (which are also called spiders or wanderers) are programs that traverse many pages in the World Wide Web searching for information.There have been occasions where robot have visited web server where they were not wanted for one reason or another.Some robot overwhelmed server with rapid-fire requests, or requested the same file again and again.
|
| Semi Related Terms: 225 CLICK HERE TO VIEW |
:-'|, 8-|, AL, AT, AV, AI, .al, .am, .ar, .at, .AI, AYT?, .ARC, AND, API, .ag, .an, .as, AM, A, .ALL, .API, AS, B, BOT, BR, .be, .br, .bs, Bot, .bo, .ca, .ch, .co, .C, .COM, Call, .com, .CAL, C, CA, Cc, CHI, .cc, Ch, (c), CO, .de, Display, .DIC, |-D, De, .DER, EE, .ee, .es, EQ, ED, .eh, .er, .fi, .fo, .FON, .FOR, FO, F, File, Font, Format, G, .ge, .gr, GE, GA, .ga, .gh, .GRA, .H, HTML, HTTP, .hr, H, Hit, .html, .htm, .ht, Hat, .HEL, .HT, ID, .INF, IDE, IN, .id, .il, .in, .is, .it, ISP, IC, .IDE, IT, .io, .ir, Info, ITS, .info, |I, L, .li, LE, LED, Ls, .la, .ls, LD, MAN, M, Ms, .ma, .ml, .ms, .MAT, .nl, .no, .NG, NE, Ns, N, NBS, NOR, NOT, .na, .ne, .nf, .nt, ?, : |, (), ((())), *, :-?, :( ), :*, %-|, :*), :-*, :-|, :-| :-|, :-~|, [:-], [:], [:|], |-{, |-|, OT, OR, O, OBO, .om, O+, |-O, |^o, .pl, Page, PR, .pa, .pr, PAGE, P*, Prog, Q, Ques, .ro, .RA, .RAM, RAM, .rc, Re, RS, RL, Rm, .re, .rw, Robot, .RM, S, .se, .si, .SIT, .SAM, .STY, .S, Server, .SEA, SAM, SEA, Spider, .sa, .so, .st, SO, .th, .tr, .tt, TR, TM, T, .tm, .tp, TI, .ve, .vi, WAN, WWW, World Wide Web, W, Web, WO, .web, WE, .WID, .X, X, .Z
|
|
|
