module Vault

Persistent connections for Net::HTTP

PersistentHTTP maintains persistent connections across all the servers you wish to talk to. For each host:port you communicate with a single persistent connection is created.

Multiple PersistentHTTP objects will share the same set of connections.

For each thread you start a new connection will be created. A PersistentHTTP connection will not be shared across threads.

You can shut down the HTTP connections when done by calling shutdown. You should name your PersistentHTTP object if you intend to call this method.

Example:

require 'net/http/persistent'

uri = URI 'http://example.com/awesome/web/service'

http = PersistentHTTP.new 'my_app_name'

# perform a GET
response = http.request uri

# or

get = Net::HTTP::Get.new uri.request_uri
response = http.request get

# create a POST
post_uri = uri + 'create'
post = Net::HTTP::Post.new post_uri.path
post.set_form_data 'some' => 'cool data'

# perform the POST, the URI is always required
response http.request post_uri, post

Note that for GET, HEAD and other requests that do not have a body you want to use URI#request_uri not URI#path. The request_uri contains the query params which are sent in the body for other requests.

SSL¶ ↑

SSL connections are automatically created depending upon the scheme of the URI. SSL connections are automatically verified against the default certificate store for your computer. You can override this by changing verify_mode or by specifying an alternate cert_store.

Here are the SSL settings, see the individual methods for documentation:

certificate

This client's certificate

ca_file

The certificate-authorities

ca_path

Directory with certificate-authorities

cert_store

An SSL certificate store

ciphers

List of SSl ciphers allowed

private_key

The client's SSL private key

reuse_ssl_sessions

Reuse a previously opened SSL session for a new connection

ssl_timeout

SSL session lifetime

ssl_version

Which specific SSL version to use

verify_callback

For server certificate verification

verify_depth

Depth of certificate verification

verify_mode

How connections should be verified

Proxies¶ ↑

A proxy can be set through proxy= or at initialization time by providing a second argument to ::new. The proxy may be the URI of the proxy server or :ENV which will consult environment variables.

See proxy= and proxy_from_env for details.

Headers¶ ↑

Headers may be specified for use in every request. headers are appended to any headers on the request. override_headers replace existing headers on the request.

The difference between the two can be seen in setting the User-Agent. Using http.headers['User-Agent'] = 'MyUserAgent' will send “Ruby, MyUserAgent” while http.override_headers['User-Agent'] = 'MyUserAgent' will send “MyUserAgent”.

Tuning¶ ↑

Segregation¶ ↑

By providing an application name to ::new you can separate your connections from the connections of other applications.

Idle Timeout¶ ↑

If a connection hasn't been used for this number of seconds it will automatically be reset upon the next use to avoid attempting to send to a closed connection. The default value is 5 seconds. nil means no timeout. Set through idle_timeout.

Reducing this value may help avoid the “too many connection resets” error when sending non-idempotent requests while increasing this value will cause fewer round-trips.

Read Timeout¶ ↑

The amount of time allowed between reading two chunks from the socket. Set through read_timeout

Max Requests¶ ↑

The number of requests that should be made before opening a new connection. Typically many keep-alive capable servers tune this to 100 or less, so the 101st request will fail with ECONNRESET. If unset (default), this value has no effect, if set, connections will be reset on the request after max_requests.

Open Timeout¶ ↑

The amount of time to wait for a connection to be opened. Set through open_timeout.

Socket Options¶ ↑

Socket options may be set on newly-created connections. See socket_options for details.

Non-Idempotent Requests¶ ↑

By default non-idempotent requests will not be retried per RFC 2616. By setting retry_change_requests to true requests will automatically be retried once.

Only do this when you know that retrying a POST or other non-idempotent request is safe for your application and will not create duplicate resources.

The recommended way to handle non-idempotent requests is the following:

require 'net/http/persistent'

uri = URI 'http://example.com/awesome/web/service'
post_uri = uri + 'create'

http = PersistentHTTP.new 'my_app_name'

post = Net::HTTP::Post.new post_uri.path
# ... fill in POST request

begin
  response = http.request post_uri, post
rescue PersistentHTTP::Error

  # POST failed, make a new request to verify the server did not process
  # the request
  exists_uri = uri + '...'
  response = http.get exists_uri

  # Retry if it failed
  retry if response.code == '404'
end

The method of determining if the resource was created or not is unique to the particular service you are using. Of course, you will want to add protection from infinite looping.

Connection Termination¶ ↑

If you are done using the PersistentHTTP instance you may shut down all the connections in the current thread with shutdown. This is not recommended for normal use, it should only be used when it will be several minutes before you make another HTTP request.

If you are using multiple threads, call shutdown in each thread when the thread is done making requests. If you don't call shutdown, that's OK. Ruby will automatically garbage collect and shutdown your HTTP connections when the thread terminates.

A Net::HTTP connection wrapper that holds extra information for managing the connection's lifetime.

Generic connection pool class for e.g. sharing a limited number of network connections among many threads. Note: Connections are lazily created.

Example usage with block (faster):

@pool = ConnectionPool.new { Redis.new }

@pool.with do |redis|
  redis.lpop('my-list') if redis.llen('my-list') > 0
end

Using optional timeout override (for that single invocation)

@pool.with(:timeout => 2.0) do |redis|
  redis.lpop('my-list') if redis.llen('my-list') > 0
end

Example usage replacing an existing connection (slower):

$redis = ConnectionPool.wrap { Redis.new }

def do_work
  $redis.lpop('my-list') if $redis.llen('my-list') > 0
end

Accepts the following options:

• :size - number of connections to pool, defaults to 5

• :timeout - amount of time to wait for a connection if none currently available, defaults to 5 seconds

Examples:

ts = TimedStack.new(1) { MyConnection.new }

# fetch a connection
conn = ts.pop

# return a connection
ts.push conn

conn = ts.pop
ts.pop timeout: 5
#=> raises Timeout::Error after 5 seconds