EGP Crawl Activity Connector 1.0.0 — owner setup ================================================ This is not an anonymous domain checker, a Google Crawl Stats API, authenticated Googlebot verification, a waste score or a measurement of allocated crawl budget. It automates collection of bounded evidence from ONE log source YOU control. No paid data API or executable third-party dependency is required. Requirements ------------ Node.js 22 or later on your log-owning server; plain UTF-8 Apache/Nginx Common or Combined log format; read-only access to ONE REGULAR NON-SYMLINK file; a private writable state directory; HTTPS access to eduguestpost.com:443. Custom formats, compressed archives, IIS/JSON and arbitrary columns are unsupported. This is not a WordPress upload of your raw logs. Windows owners must configure private NTFS ACLs themselves; POSIX config/state permissions are checked by the collector. Do not run as root just to grant log access. Use an appropriately restricted service account/log-read group. Do not change a production log format casually. This tool does not install a server service or change WordPress registration. Setup ----- 1. Sign in to your existing EduGuestPost account. Create one private project and select the real capture point: origin, or an edge access-log file you control. Issue its ingest credential. It is displayed once for two minutes. No one should ask you to paste it, raw logs or account passwords into chat. 2. Download and inspect egp-crawl-connector.zip from the private workspace. Keep collector.mjs, log-parser.cjs and the notices together on that server. No npm install, shell downloads, paid API, proxy or remote account connection needed. 3. Copy config.example.json to a PRIVATE directory OUTSIDE ALL WEB ROOTS. Fill in the real project number and its token, an absolute logFile and stateFile, and matching capturePoint. Config and state files must be mode 0600 on POSIX and owned by the account running Node. Protect their parent directory (0700). 4. Keep start="end" for a new forward-only source. The initial skipped history is explicitly flagged. Optional start="beginning" processes only this file, in bounded batches; records older than 30 days or >5 minutes in the future are rejected. Never change a project's file, grouping or capture point by erasing the state: create a separate source project instead. 5. Set local prefix-to-group mappings. Only these fixed codes leave the server: home, content, commerce, taxonomy, assets, search, other. Prefixes, paths and exact URLs are NOT sent. Longest boundary-matching prefix wins; '/' matches only the homepage. Percent-encoded and absolute targets are not normalized into different paths; unmatched targets remain other. No regular expressions. 6. Run once yourself (example paths; replace them with your private paths): node /opt/egp-crawl/collector.mjs --config /etc/egp-crawl/private.json Review the local safe status and your signed-in private report. Verify that expected traffic appears and that coverage/flags match your logging setup. 7. Only after that test, schedule the same command on YOUR server, for example every 15 minutes using an existing scheduler. Do not overlap runs. Nothing here silently installs a job, enables telemetry or creates an external account. Reliability and limits ---------------------- Each run reads at most 2 MiB / 5,000 records / 256 aggregate cells, with strict 16 KiB lines and 2 KiB agent strings. One run makes at most one HTTPS submission and never follows redirects. The server accepts at most one NEW batch per minute. If a send fails, a durable pending aggregate is retried EXACTLY on the next run; it is acknowledged idempotently. The source checkpoint advances only after that acknowledgment. Preserve it. A 409 sequence conflict needs owner investigation; the connector does not reset sequences or retry against a different project. Queued batches more than 30 days old need a new source, not timestamp rewriting. Rotation/truncation and changed checkpoint content are flagged as possible gaps. The collector never searches other paths for rotated files. A partial trailing line is retained for the next run; overlong records are skipped with a flag. Copy-truncate/regrowth between runs cannot always be detected. This is therefore not an exactly-once or complete-history guarantee. A cap means more work remains. Run frequency must suit your volume and rotation retention. No full-history extrapolation, merging of overlapping origin/edge sources or unique visitors. An exclusive .lock prevents concurrent collectors. It is not automatically broken. After a crashed process, first confirm no collector is running, then remove only its exact stale .lock file. Never delete the checkpoint to clear a lock. Keep the original source log as your system of record; server reports are not backups. Identity, privacy and interpretation ----------------------------------- Only declared user-agent categories are shown. No IP, DNS or Google range check is made; a spoofed Googlebot agent stays an UNVERIFIED declaration. Other Google crawlers are kept separate. Missing/unmatched agents do not mean human visitors. Common format has no user-agent evidence. No IP/forwarded-IP field is trusted. Raw IPs, usernames, referrers, log lines, agents, query VALUES and exact paths are discarded before transport. Query PRESENCE, safe groups, UTC dates, response groups and logged body-byte totals remain. Those aggregates are still PRIVATE. Origin logs omit CDN-cache-only requests. Edge logs represent only that source. Counts do not establish indexing, rankings, crawling demand, unique users or wasted budget. Redirects/errors/queries can be intentional. 3xx includes 304; the response group alone cannot classify redirects. '-' bytes means unknown, not zero. Numeric log time-zone offsets are applied before UTC bucketing. The public setup page is indexable. Private workspace/API responses are no-store and noindex, require your account, and exclude theme/analytics scripts. CDN rules must honor these headers and never cache the workspace query or REST namespace. Database administrators/backups can still access stored aggregates: this is account isolation, not end-to-end encryption. No raw-log browsing admin panel. Stopping, revocation and recovery -------------------------------- Stop your source scheduler; revoke its credential in the private workspace. Replacing a credential keeps sequence/history but invalidates the prior token. Update only the token in your existing private config; keep its checkpoint. If the checkpoint is truly lost, create a new project starting at current EOF. Do not try to reconstruct offsets and resend an entire historical file. Delete a project only when you intend to permanently remove its private history. Server limits: 64 KiB per submission, 2,048 stored batches/project and latest 200 batches per report/export, within the configured retention (default 14 days). Caps may drop older records sooner. Reads exclude expired records; hourly traffic-dependent WordPress cleanup removes stored expired batches. Lowering retention intentionally expires old aggregates. Failures are local; the website shows a stale source after its freshness threshold, not a invented error reason. Review JSON/CSV before sharing. No keys or raw logs belong in those exports.