[{"data":1,"prerenderedAt":43},["ShallowReactive",2],{"$fv3natykwko96":3},{"href":4,"title":5,"description":6,"kind":7,"mark":8,"planned":9,"contributors":10,"provenance":8,"html":11,"headings":12},"\u002Fdocs\u002Ftools\u002Fweb","Web","The public web, read directly: a page as text, a sitemap as a list, a URL's text or JSON. Nothing to connect.","operates",null,false,[],"\u003Ch2 id=\"what-a-member-can-do\">What a member can do\u003C\u002Fh2>\n\u003Cp>A member can read the public web with three built-in tools. A task names the ones it may use:\u003C\u002Fp>\n\u003Cdiv class=\"prose__table\">\n\u003Ctable>\n\u003Cthead>\n\u003Ctr>\n\u003Cth>Tool\u003C\u002Fth>\n\u003Cth>What it does\u003C\u002Fth>\n\u003C\u002Ftr>\n\u003C\u002Fthead>\n\u003Ctbody>\n\u003Ctr>\n\u003Ctd>\u003Ccode>web.read\u003C\u002Fcode>\u003C\u002Ftd>\n\u003Ctd>Reads one page and returns the words a reader would see, its title, and a hash of exactly those words.\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>\u003Ccode>web.sitemap\u003C\u002Fcode>\u003C\u002Ftd>\n\u003Ctd>Reads a sitemap and returns its addresses as a list, each with the date the site says it last changed.\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>\u003Ccode>web.request\u003C\u002Fcode>\u003C\u002Ftd>\n\u003Ctd>Sends a GET to an address and returns the text or JSON that comes back, with its status.\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003C\u002Ftbody>\n\u003C\u002Ftable>\n\u003C\u002Fdiv>\n\u003Cp>What each takes and returns is in the \u003Ca href=\"\u002Fdocs\u002Ftools\u002Fbuilt-in\u002Fweb\">Web tools reference\u003C\u002Fa>.\u003C\u002Fp>\n\u003Cp>These are plain requests, not a browser. No JavaScript runs, and no cookies or credentials are sent. For a product a\nmember signs in to and clicks through, bind a \u003Ca href=\"\u002Fdocs\u002Ftools\u002Fbrowser\">browser\u003C\u002Fa>. For a system with an MCP server, connect\nit from the \u003Ca href=\"\u002Fdocs\u002Ftools\">Tools\u003C\u002Fa> directory.\u003C\u002Fp>\n\u003Ch3 id=\"a-hash-that-changes-when-the-words-do\">A hash that changes when the words do\u003C\u002Fh3>\n\u003Cp>\u003Ccode>web.read\u003C\u002Fcode> returns a \u003Ccode>sha256\u003C\u002Fcode> of the page's visible text. Scripts, styles and tracking snippets are not part of it, so\na site that is redeployed with nothing reworded returns the same hash, and a page with one sentence changed returns a\nnew one. A task can keep the hash to tell whether a page needs reading again, and can pass it to \u003Ccode>memory.train\u003C\u002Fcode> as the\nrevision of what it learned.\u003C\u002Fp>\n\u003Cp>The hash follows the text, not the meaning. A page that prints the time or rotates a banner changes on every read.\u003C\u002Fp>\n\u003Ch3 id=\"public-addresses-only\">Public addresses only\u003C\u002Fh3>\n\u003Cp>A request reaches public addresses, and nothing else. A private, local or internal address is refused\n(\u003Ccode>web_address_refused\u003C\u002Fcode>), including a public name that points at one. Every redirect is checked again before it is\nfollowed, and a redirect to such an address is refused without being requested (\u003Ccode>web_redirect_refused\u003C\u002Fcode>).\u003C\u002Fp>\n\u003Ch3 id=\"whole-or-not-at-all\">Whole, or not at all\u003C\u002Fh3>\n\u003Cp>A page or a response too long for one result is refused, with its size and its hash and none of its content\n(\u003Ccode>web_page_too_large\u003C\u002Fcode>, \u003Ccode>web_body_too_large\u003C\u002Fcode>). A sitemap too long for one result returns how many addresses it has,\n\u003Ccode>complete: false\u003C\u002Fcode>, and no list. A member never receives part of a page as though it were the page.\u003C\u002Fp>\n\u003Cp>A page that answers with an error is refused by \u003Ccode>web.read\u003C\u002Fcode> and \u003Ccode>web.sitemap\u003C\u002Fcode> (\u003Ccode>web_status_not_ok\u003C\u002Fcode>), so an error page\nis not read as content. \u003Ccode>web.request\u003C\u002Fcode> returns the status for the task to judge.\u003C\u002Fp>\n\u003Cdiv class=\"prose__planned\">\n\u003Cp class=\"prose__flag\">Planned\u003C\u002Fp>\n\u003Ch3 id=\"reading-in-parts-writing-and-credentials\">Reading in parts, writing, and credentials\u003C\u002Fh3>\n\u003Cp>A long page or sitemap will be readable in parts, each carrying the hash of the whole. \u003Ccode>web.request\u003C\u002Fcode> will send\n\u003Ccode>POST\u003C\u002Fcode>, \u003Ccode>PUT\u003C\u002Fcode>, \u003Ccode>PATCH\u003C\u002Fcode> and \u003Ccode>DELETE\u003C\u002Fcode>, waiting for your approval unless the task is kept to reading. An enterprise will\nbe able to attach a credential to an address on \u003Cstrong>Tools\u003C\u002Fstrong>, which the OS adds to requests for that address and the\nmember never sees.\u003C\u002Fp>\n\u003C\u002Fdiv>\n\u003Ch2 id=\"who-connects-it\">Who connects it\u003C\u002Fh2>\n\u003Cp>Nobody. There is nothing to bind and no credential to store: the tools are built in, the same for every enterprise.\nA task has them when it names them.\u003C\u002Fp>\n\u003Ch2 id=\"what-waits-for-you\">What waits for you\u003C\u002Fh2>\n\u003Cp>Nothing a \u003Ccode>web.*\u003C\u002Fcode> tool does waits for you unless the task declares a gate on it. Gate \u003Ccode>web.request\u003C\u002Fcode> and every request\nwaits, with the address it was about to ask for, until you approve it on \u003Cstrong>Gates\u003C\u002Fstrong>. See\n\u003Ca href=\"\u002Fdocs\u002Fcontrol\u002Fgates\">Gates and tool scopes\u003C\u002Fa>.\u003C\u002Fp>\n\u003Ch2 id=\"setup\">Setup\u003C\u002Fh2>\n\u003Col>\n\u003Cli>On the task, name each tool it needs: \u003Ccode>web.read\u003C\u002Fcode>, \u003Ccode>web.sitemap\u003C\u002Fcode>, \u003Ccode>web.request\u003C\u002Fcode>.\u003C\u002Fli>\n\u003Cli>In the task's skill, say which addresses it should read, and what to do with a refusal: skip a page that is too\nlong, and never treat a refusal as the page.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Check it.\u003C\u002Fstrong> Run the task. The run page shows every request, where its redirects ended, and what came back.\u003C\u002Fli>\n\u003C\u002Fol>\n\u003Ch2 id=\"troubleshooting\">Troubleshooting\u003C\u002Fh2>\n\u003Cdiv class=\"prose__table\">\n\u003Ctable>\n\u003Cthead>\n\u003Ctr>\n\u003Cth>What you see\u003C\u002Fth>\n\u003Cth>Why\u003C\u002Fth>\n\u003Cth>What to do\u003C\u002Fth>\n\u003C\u002Ftr>\n\u003C\u002Fthead>\n\u003Ctbody>\n\u003Ctr>\n\u003Ctd>\u003Ccode>web_address_refused\u003C\u002Fcode>\u003C\u002Ftd>\n\u003Ctd>The address is private, local or internal, or its name points at one\u003C\u002Ftd>\n\u003Ctd>Use the public address. A system inside your network is not reachable this way\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>\u003Ccode>web_redirect_refused\u003C\u002Fcode>\u003C\u002Ftd>\n\u003Ctd>A redirect loops, or leads somewhere the tools do not go\u003C\u002Ftd>\n\u003Ctd>Read the address the site redirects to, if it is public\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>\u003Ccode>web_url_invalid\u003C\u002Fcode>\u003C\u002Ftd>\n\u003Ctd>The address is not \u003Ccode>http\u003C\u002Fcode> or \u003Ccode>https\u003C\u002Fcode>, names a port, or carries a username\u003C\u002Ftd>\n\u003Ctd>Pass a plain \u003Ccode>https:\u002F\u002F\u003C\u002Fcode> address\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>\u003Ccode>web_status_not_ok\u003C\u002Fcode>\u003C\u002Ftd>\n\u003Ctd>The page answered with an error status\u003C\u002Ftd>\n\u003Ctd>Check the address. \u003Ccode>web.request\u003C\u002Fcode> shows the status and body\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>\u003Ccode>web_not_html\u003C\u002Fcode>\u003C\u002Ftd>\n\u003Ctd>\u003Ccode>web.read\u003C\u002Fcode> was given something that is not a page\u003C\u002Ftd>\n\u003Ctd>Use \u003Ccode>web.sitemap\u003C\u002Fcode> for a sitemap, \u003Ccode>web.request\u003C\u002Fcode> for JSON or text\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>\u003Ccode>web_page_too_large\u003C\u002Fcode>, \u003Ccode>web_body_too_large\u003C\u002Fcode>\u003C\u002Ftd>\n\u003Ctd>The content does not fit in one result\u003C\u002Ftd>\n\u003Ctd>Have the task skip it. Nothing was returned, so there is nothing to summarise\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>A sitemap returns \u003Ccode>complete: false\u003C\u002Fcode>\u003C\u002Ftd>\n\u003Ctd>The list does not fit in one result\u003C\u002Ftd>\n\u003Ctd>Do not treat missing addresses as removed pages\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>\u003Ccode>web_header_refused\u003C\u002Fcode>\u003C\u002Ftd>\n\u003Ctd>The task set a header other than \u003Ccode>accept\u003C\u002Fcode> or \u003Ccode>accept-language\u003C\u002Fcode>\u003C\u002Ftd>\n\u003Ctd>Remove it. Credentials are never passed by a member\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>A page reads as nearly empty\u003C\u002Ftd>\n\u003Ctd>The site builds its content with JavaScript\u003C\u002Ftd>\n\u003Ctd>Bind a \u003Ca href=\"\u002Fdocs\u002Ftools\u002Fbrowser\">browser\u003C\u002Fa> for that site\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>The hash changes on every read\u003C\u002Ftd>\n\u003Ctd>The page prints something that changes: a time, a counter, a rotating banner\u003C\u002Ftd>\n\u003Ctd>Compare the text, or read a steadier address\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003C\u002Ftbody>\n\u003C\u002Ftable>\n\u003C\u002Fdiv>\n",[13,17,21,24,27,31,34,37,40],{"id":14,"text":15,"level":16,"planned":9},"what-a-member-can-do","What a member can do",2,{"id":18,"text":19,"level":20,"planned":9},"a-hash-that-changes-when-the-words-do","A hash that changes when the words do",3,{"id":22,"text":23,"level":20,"planned":9},"public-addresses-only","Public addresses only",{"id":25,"text":26,"level":20,"planned":9},"whole-or-not-at-all","Whole, or not at all",{"id":28,"text":29,"level":20,"planned":30},"reading-in-parts-writing-and-credentials","Reading in parts, writing, and credentials",true,{"id":32,"text":33,"level":16,"planned":9},"who-connects-it","Who connects it",{"id":35,"text":36,"level":16,"planned":9},"what-waits-for-you","What waits for you",{"id":38,"text":39,"level":16,"planned":9},"setup","Setup",{"id":41,"text":42,"level":16,"planned":9},"troubleshooting","Troubleshooting",1791124519831]