Scrapod

A driver of remote headless scraping cluster for
Capybara
(aka remote Capybara Webkit).

Introduction

There are many browser automation tools, mostly built on top of
PhantomJS.
In my opinion, Capybara is still the best. Unfortunately most of Capybara
drivers are not enough suitable for web scraping purposes.
There are the reasons:

They run on the same server as your worker what can be ineffective
They do not care about headless browser process termination what can cause memory leaks
They open headless browser process on-demand what can be slow
They can take the total amount of available RAM and freeze the server

This happens because Capybara is intended firstly for testing purposes
but not for web scraping. Authors
do not want
to support such use cases. So you as a final product developer have to solve
these problems by yourself. This spawns
primitive and makeshift solutions
which are good until you have to run more than a few tens of tasks per hour.

The Scrapod
tries to solve all or most of the problems listed above.

Architecture

The Scrapod consists of two parts: client and server.

Client

Client is a driver for Capybara. It connects to server when you create session,
sends calls to Capybara API over the connection and converts responses to Ruby
data structures. This is what you want to use in a final product application.
This document describes the client completely.

Server

Server is a process which can run on the same or on another machine
than the client. Server configuration can be complex but still not difficult.
It is described in the server repository.
For testing purposes it is enough to install the gem and run
scrapod-server --debug. It will start listening on local port 20885.

Installation

Add the gem to your Gemfile (with git source because I do not push new
experimental gems to RubyGems):

gem 'scrapod', git: 'https://github.com/krowpu/scrapod.git'

This will register a Capybara driver with name :scrapod which connects
to local port 20885. To connect to the remote host register a driver
by yourself. Assuming you use Sidekiq with Ruby on Rails, create the file
config/initializers/scrapod.rb with the following content:

Capybara.register_driver :scrapod do |app|
  Scrapod::Driver.new app, Scrapod::Configuration::DEFAULT.merge(
    host: ENV['SCRAPOD_HOST']       || '127.0.0.1',
    port: ENV['SCRAPOD_PORT']&.to_i || 20885,
  )
end

Usage

Just create Capybara session with :scrapod driver and use it as usually:

session = Capybara::Session.new :scrapod
session.visit 'https://google.com'
session.title #=> "Google"
session.fill_in 'q', with: 'Capybara'
session.all('input')[1].trigger 'click'
session.title #=> "Capybara - Google Search"