Swiss Knife SEO: The Crawler We Built Because Nothing Else Answered Our Questions

Most agencies run on the same handful of rented tools, which means they inherit the same blind spots as everybody else. That is fine until a client asks a question the tool cannot answer, and then you are guessing in a meeting.

Three questions kept coming up that we could not answer properly. Why is this page invisible when it already has hundreds of internal links pointing at it? Which of those links actually pass anything worth having? And when an answer engine recommends a competitor instead of us, what exactly is it reading that we are not giving it?

So we wrote a crawler to answer them. It started as a script for one client and turned into Swiss Knife SEO, the platform that now runs under every audit we deliver.

Counting internal links is not the same as scoring them

Almost every crawler on the market counts internal links. You get a number next to each URL, and the number goes up when you add a link. The trouble is that the number treats every link as equal, and search engines do not.

A link sitting inside a paragraph of body copy, surrounded by relevant text, is not the same thing as the eighteenth item in a footer that appears on all 4,000 pages of the site. Both add one to the count. Only one of them means anything.

Our scoring weighs position. A link inside your content counts for more than one buried in boilerplate, and the report shows you which pages look well linked on paper while being effectively orphaned in practice. That gap is where a lot of quiet traffic loss lives, and a plain link count will never show it to you. If you want the background on how internal structure feeds rankings at all, our guide to the six types of SEO that increase traffic covers where this sits in the wider picture.

The crawl and the AI citation check belong in the same report

The second thing that fell out of building our own tool is more recent, and it is the part we would not have been able to buy.

We track whether four answer engines cite you: ChatGPT, Perplexity, Gemini and Google AI Overviews. On its own that is a scoreboard, and a scoreboard is not very useful. What makes it worth reading is putting it next to the crawl of the same site.

Doing that surfaces problems that neither view shows alone. A page an answer engine cites, while your own robots rules quietly block the crawler that would have read it. A page that only renders its content through JavaScript, which those crawlers never execute, so the useful part is invisible to them. A page carrying most of your internal authority that has never been cited once.

Those are three different jobs with three different fixes, and you cannot tell them apart from a rankings report. Being absent from an AI answer is invisible in a way a dropped position is not, because nothing moves and no chart dips.

What is actually in it

The platform runs a full technical crawl first: status codes, redirect chains, canonicals, indexability, duplicate content and Core Web Vitals across every URL it can reach. On the top tier it handles up to 250,000 URLs in a single crawl, which covers most retail catalogs we get asked about.

On top of that sit 36 single job tools, the kind of thing you reach for mid task rather than scheduling: schema and hreflang generators, redirect and header checkers, an instant page audit, a live SERP preview. There is also a studio of 100 AI workflows we built for research, drafting and campaign work, which is where a lot of our own content production starts.

Connect Search Console and Analytics read only and the crawl stops being a list of imperfections. Every finding gets ranked by the clicks it is costing you rather than by how many rows it fills, which is the difference between a report somebody reads and a queue somebody works through.

We run it on ourselves first

Every property we own goes through the same platform before a client account does. That is not a slogan, it is how we find the awkward bugs. When we recommend a fix, we have usually already made that mistake on one of our own sites and measured what the recovery looked like.

It also changes what happens when the tool cannot do something. When a client needs a report that does not exist yet, we add it, rather than explaining that the vendor has it on a roadmap. That is the practical benefit of owning the instrument, and it is most of the reason we still maintain it.

Where to start

The first scan is free, needs no account and no card, and the findings are yours whether or not you ever speak to us. If you would rather read before you run anything, the help center is public, and the crawl engine reference documents how the crawler behaves.

If the crawl turns up work you would rather hand over, that is what our SEO service is for, and it is worth reading what search engine optimization actually covers before deciding how much of it you want to own in house. For the pages that need writing rather than fixing, our guide to creating link worthy content is a better starting point than another audit.

Two related reads if you are working through a technical list: schema markup and what it does for you, and local SEO if your customers search with a place name attached. If you are earlier than that and still choosing infrastructure, how to choose a hosting provider will save you a migration later.

You can also just tell us what you are trying to grow, and we will tell you honestly whether any of this helps. More of our writing sits in the SEO archive and on the blog, and there is more on how we work if you want the background first.