• Dan Shure

    Nice job! Anytime something basically replaces me having to teach/show ppl something cause it’s thorough and useful – THAT’S quality content.

  • TheRobbie22

    Can’t say this emphatically enough — WELL DONE! This is both thorough and very well put together. This page has officially been bookmarked in my browser.

  • James Stewardson

    Awesome guide, so many little tricks! A timely reminder to renew my subscription.

    Question – Do you know of a way to crawl and check the follow status of links (same domain) on a list of external pages. The result would be something like: On page x, there are links a1, a2,… and their follow status is follow, nofollow, on page y, there are links a1, a2 etc. The inputs would be a domain and list of external page url’s.

    This would be a handy tool, and I guess other elements of the links could be found too. Thanks

  • Aichlee Bushnell

    Glad you found it useful!

  • Ryan McLaughlin

    This was my exact thought. There’s no doubt I’m going to reference this for training material.

  • Mike Gracen

    Thought it said “55+ Ways of Looking Like a Tool”. I thought to myself, ‘damn I know way more than that’ :-)

  • Jesse McFarlane

    Great article, I too love technical audits. The treasure hunt metaphor is perfect and there’s something really satisfying about hunting down on-page/structure issues.

  • Dan Sharp

    Upload the URLs in list mode and crawl.

    Then use the ‘advanced export’ option in the top level navigation and ‘All Out Links’. This will provide you with every ‘out link’ from those URLs and whether they are follow or nofollow. In Excel, whack a filter on the destination column for ‘’.

    Awesome post Aichlee, I’ll be referencing this to people as well :-)

  • Aichlee Bushnell

    Totally! I love the thrill of the hunt :)

  • Aichlee Bushnell

    Heh, I didn’t think of that ;)

  • Aichlee Bushnell

    Thank you! Glad you found it useful!

  • Aichlee Bushnell

    @google-516641ccdb8012d81bc22131ae490bc9:disqus Glad you found it useful!

    @twitter-16516264:disqus Thanks for jumping in! So glad that you approve :)

  • James Stewardson

    Brilliant, thanks for helping Dan!

  • EJ Garcia

    Mike, how are things going at Seaworld? Great to see you checking out the SEER blog.

  • Jane Freeman

    Can a tool get bigger than am agency? :)

  • Jane Freeman

    want to share this post more, but NO date. dont want to share an old post :(

  • Ian Macfarlane

    Fantastic post, I keep finding new uses for Screaming Frog on a regular basis.

    One of my biggest bugbears in crawling was jsessionids, commonly set on Java-based platforms when cookies are disabled. To remove those, simply regex replace ;jsessionid=[^?]+ with an empty string.

    Also, minor typo, the iframe example should omit the trailing “>” as otherwise it won’t match the iframe with attributes.

  • Brian Jensen

    Thanks for posting this incredibly comprehensive resource Aichlee. I can already tell this will be one I review multiple times over. I have noticed that when I use Screaming Frog, the total number of pages crawled will vary when I crawl the site multiple times using the tool. Is this something you have run into?

  • Aichlee Bushnell

    Hi, Brian: Thanks for reading! I do recall seeing that once or twice, where the number of pages crawled was off by one. Did you, by chance, start your crawl with a URL that was redirected?

    For instance, if you enter ‘’ as the URL to spider and that is redirected to, then you crawl again but enter ‘’ as the URL to spider, you would have one less URL in your crawl (as long as there are no internal links to the redirected URL).

    Sounds obvious, but I’ve made that mistake when not paying attention. Other than that, I’m not sure what the cause could be. Unless a server error is suddenly causing one page (and the links on it) to be unavailable during one crawl, and not the other…

  • Aichlee Bushnell

    Hi, Ian: Good catch on the iframe example! Should be fixed above. Thanks for sharing the jessionid tip, too :)

  • Aichlee Bushnell

    Thanks for your enthusiasm, Jane :) The post was published on 3/1/2013. The date is listed at the very bottom of the post.

  • Aidan Rogers

    How long did this take you to produce Aichlee? Including just coming up with the idea/planning?

  • Aichlee Bushnell

    I’ve been piecing it together over the course of several months. I would estimate 60 or so hours from start to finish. I am a bit of a perfectionist though, so it saw a lot of revision.

  • Aidan Rogers

    Well congrats on seeing it through as the end result is impressive. Perfectionism can be a crippling trait where nothing is ever good enough, as I’m sure you can relate!

  • Dan R

    An amazing post on an amazing tool. Glad to see SF is becoming better known (even if a little bit of me would like it kept quiet!)
    Is there a PDF of this at all? Would be a great ref guide. Dan @SF – this beats the manual :)

  • Aichlee Bushnell

    Thanks! At the moment, no, there’s no PDF. Good idea though!

  • Gyi Tsakalakis

    Shoot you guys are good…

  • Matt Dolge

    Great article!

  • Anna Lewis

    This has got to be the best blog post I’ve ever read! Soo many helpful tips for such a useful tool. I’ve been using Screaming Frog for ages and thought I knew most of it but still learnt a lot from this post. Have bookmarked and will revisit regularly I’m sure. Thank you!

  • Aichlee Bushnell

    Thanks so much, Anna! Glad you found it useful :)

  • Aichlee Bushnell

    Hi Carl,

    There are a couple additional articles that focus on different ways to use SF linked in the body of the guide above.

    In terms of general audit posts, I personally love Annie Cushing’s crazy audit checklist, but I see that you’ve got it in there.

    Aleyda Solis also has a great piece on mobile SEO audits, which includes a lot of technical info:

    Give me a couple days and I can make an analytics snippets resource for you ;)

  • Jay Peyton

    Kinda late chiming in on this post, but I was just about to write a training manual on using Screaming Frog, found this, and realized you did my job for me. You rule. The bonus round = mind blown. How did I miss that stuff?!

    One question: Somebody asked me the other day if they could differentiate between rel=”nofollow” links and meta robots nofollow links in a crawl. It seems a shade unnecessary, but I came up with a clunky workaround for it. Any silver bullets for that one?

  • Aichlee Bushnell

    If you export ‘All Out Links’ from the ‘Advanced Export’ menu, you will have a report of every link that contains rel=nofollow and the pages on which they are located. Rel=”nofollow” links will be indicated by the word ‘FALSE’ in the ‘Follow’ column of that report.

    In order to find pages that contain meta robots nofollow, you can crawl the site as usual, then use the ‘Nofollow’ filter in the ‘Directives’ tab to export a list of all of the pages that contain that meta robots tag.

    You can then use a vlookup to cross-reference the two. Hope that helps!

  • Jay Peyton

    Thanks, Aichlee! That’s exactly how I went about it originally. I thought maybe I had missed a magical button that would allow me to skip any cross-referencing, but alas… Glad to know the experts agree! ;-)

  • Denise

    Thanks for this (Probably the most in-depth explanation ) of anything I have come across so far. I downloaded Screaming frog and once I started it I had to go looking for info on how to use it…. And came here, great stuff

  • Den Seymour

    Better late than never. I saw this epic post the moment it came out but I didn’t have the time to properly thank you for it. So here it is, THANK YOU Aichlee :)
    A reference material for years to come for both blog posts and training.

  • Aichlee Bushnell

    glad you found it helpful!

  • Aichlee Bushnell

    My pleasure! Glad you came back to see it again :)

  • Ned Poulter

    Hands down one of the best ‘how to’ posts that I’ve ever read on using a specific tool. Aichlee, part of me wants to give you a hug and thank you for showing even more amazing features that you can use SF for, but part of me is angry at you as I’m going to be spending the next few days/weeks playing with everything you’ve included above…! :-p

    Seriously amazing post though, more of the same in the future please!

  • Benjamin Brady

    Just want to reflect what a lot of people have already said – this is fantastic and I’ll certainly be using it to help train our marketing team to use the tool. Thanks muchly!

  • Sjoerd Jongmans

    For a quick overview of possible empty category pages within your AdWords campaign URL’s you can run a custom filter containing the string pointing to the number of products (0) or empty products page. For example in Magento empty category pages contain this string: “There are no products matching the selection”.

  • boyiboyi

    Nice list! Helped me big time.

  • Andrew Pontius

    Found this incredible resource after a recommendation from Mike Ramsey of NiftyMarketing. Wow. Just wow. Great job.

  • Marcus

    Hey Aichlee

    Amazing article. We do a lot of site audits, technical SEO audits at Bowler Hat and this article has become part of our training process.

    Very cool. :)

  • Rodrigo Mantillan

    Great Job! This post goes to my Chrome’s bookmarks bar right now.

  • Mobisoft Infotech

    Great Info. Thanks for the your effort. Appreciate it.

  • Shumicps

    You have defined every sections so clearly and just easy to understand. Very good work.

  • gemusli

    wicked article on a wicked tool, my compliments to the author

  • Aichlee Bushnell


  • Aichlee Bushnell

    Hi Simone,

    The only other indicator of duplicate content is the Hash ID for each URL. If you go to the ‘Internal’ tab of your crawl, filter by ‘HTML’ and scroll the to ‘Hash’ column on the far right, you’ll see a unique series of letters and numbers for every page. If you click ‘Export’, you can use conditional formatting in Excel to highlight the duplicated values in this column, ultimately showing you pages that are identical and need to be addressed.

    If there are only certain sections of the pages that are duplicated, then this won’t work.

    I would recommend that you check out for more insight.

  • William Gomes

    Wow @aichleebushnell:disqus what a post! More like a book great work! Maybe you’d like to checkout the new free crawler around and let me know your views on that it’s not on SF’s level yet but it works in a different way and is free :)

  • Dávid Ursiny

    Great Article! This had to be written by creators of Screaming Frog, and they should be very grateful to You, Aichlee.

  • Aichlee Bushnell

    Thanks for reading! Your crawler looks like a good free alternative. I like the sidebar where you show warnings and items to note – very smart. Wondering if you’re planning to have an import functionality, so users can crawl a specific list of URLs?

  • William Gomes

    Hi Aichlee, Glad you like it. Not sure when the last time you checked was but we’ve now removed all limits in the crawler :-) Your not the first t request crawl by list so it’s something well be working on to implement in the next version. If you have any other ideas or find bugs feel free to contact me directly via the contact form on our website. Thanks.

Get our Newsletter

Keep up-to-date search trends, latest blog posts and more!