Scrape Top 100 SERP Results for a Keyword -- Titles and URLs

Тема в разделе "Поисковые системы", создана пользователем parabel, 12 фев 2018.

  1. parabel

    parabel New Member Registered

    The Chrome scraper plugin was giving me an aneyurism, so I wrote a quick script to do some basic SERP scraping. Use it, if you want; or don't, I'm not your dad.

    This script will scrape the URLs and Titles displaced for the top 100 SERP results formatted as a table.

    Two notes:

    • Google ads are not scraped by this script
    • Featured snippets ("people also ask") and similar are scraped by this script
    Anyway, here's a brief guide.

    Step #1: Open Search Settings

    [​IMG]
    Click "Settings" (Second from the Right) --> Search Settings


    Step #2: Set "Results per Page" to 100
    [​IMG]
    Or if you want to scrape fewer results, should work fine with a lower "results per page" setting.
    Step #3: Open the Console
    In Chrome (Right Click --> Inspect)

    [​IMG]

    In Firefox (Right Click --> Inspect Element)

    [​IMG]


    Other Browsers
    Ha ha ha ha ha ha ha ha ha ha ha ha ha ha... Okay, moving on...

    Step #4: Execute Code
    The following two pieces of code now must be executed. Copy-paste 4.1 into the console and press enter, then copy-paste 4.2 into the console and press enter. The images in step #3 also show what this code looks like after it has been pasted and before the enter key is pressed.

    Note that 4.1 and 4.2 must be run separately, since it takes a second for the code in 4.1 to execute and the code in section 4.2 must execute after 4.1 has had a second to complete.

    Step #4.1: Import jQuery


    Код:
    // Add jQuery from the console (if necessary)
    javascript: (function(e, s) {
        e.src = s;
        e.onload = function() {
            jQuery.noConflict();
            console.log('jQuery injected');
        };
        document.head.appendChild(e);
    })(document.createElement('script'), '//code.jquery.com/jquery-latest.min.js')
    // https://stackoverflow.com/questions/7474354/include-jquery-in-the-javascript-console
    Step #4.2: Run the Scrape

    Код:
    var t = jQuery("#search .rc .r a"); // For Google queries
    function list_scraped_urls_google(t)
    {
      var tt = "<table>";
      tt += "<thead><tr><td>SERP Pos.</td><td>URL</td><td>Title</td></tr></thead>";
      tt += "<tbody>";
      for (iii = 0; iii < t.length; ++iii)
        tt += "<tr><td>"+(iii+1)+"</td><td>" + t[iii] + "</td><td>" + t[iii].innerHTML + "</td></tr>"; // For Google with Title Tags
      tt += "</tbody>";
      display_output_onpage(tt);
    }
    
    function display_output_onpage(output)
    {
      // nl2br equivalent
      output = output.replace(/\n/g,"<br>");
    
      // Create a new tab and display the output
      var winPrint = window.open('', '_blank', '');
      winPrint.document.write(ghetto_styles()+'<title>SERP Result Data</title><body>'+output+"</body>");
      winPrint.document.close();
    }
    
    function ghetto_styles()
    {
      var t = '<style>'
      +'table                    { border: 2px solid black; width: 100%;    background-color: #eeeeee;}'
      +'table                    { margin:2%; font-size: 90%; font-family:monospace; width: 96%; box-shadow: 2px 2px rgba(70,70,70,0.3);}'
      +'table thead tr           { background-color: #6e6eed; text-align: center; color: white; font-weight: bold; }'
      +'table thead tr           { font-size: 2em; text-shadow: 1px 1px rgba(30,30,40,0.7); }'
      +'table tr                 { background-color: #e0e0ff; }'
      +'table tr:nth-child(even) { background-color: #d0d0ff; }'
      +'table tr td:first-child  { width:4%; font-weight:bold; }'
      +'table tr td              { padding: 4px 12px; width:32%;}'
      +'</style>';
      return t;
    }
    
    // EXECUTE CODE
    list_scraped_urls_google(t);
    Step #5: Do Things, Such as Copy-Paste to a Spreadsheet
    You now have the SERP result data displayed in a table. It can be transferred to a spreadsheet easily with Ctrl+A --> Ctrl+C, then open a spreadsheet and press Ctrl+V.

    Do things with this data at your discretion.

    [​IMG]


     
    Последнее редактирование: 12 фев 2018
  2. Поисковые системы

    Вопросы связанные с добычей трафика в поисковых системах и их работой
    Темы:
    16
    Сообщения:
    43
    (Нет сообщений)
    RSS

Поделиться этой страницей

Анализ сайта

Оптимизация вашего сайта и исправление ошибок.