crossroads

tips.yo (48430B)

---

 The following sections elaborate on the directives as described in
 section ref(config) to illustrate how crossroads works and to help you
 achieve the "optimal" balancing configuration.
 
 subsect(How back ends are selected in load balancing)label(howselected)
 
 In order to tune your load balancing, you'll need to understand how
 crossroads computes usage, how weighing works, and so on. In this
 section we'll focus on the dispatching modes tt(bysize), tt(byduration)
 and tt(byconnections) only. The other dispatching types are
 self-explanatory. 
 
 
 subsubsect(Bysize, byduration or byconnections?)
 
 As stated before, crossroads doesn't know 'what a service does' and
 how to judge whether a given back end is very busy or not. You
 must therefore give the right hints:
 
 itemization(
         it() In general, a service which is CPU bound, will be more
         busy when it takes longer to process a request. The dispatch
         mode tt(byduration) is appropriate here.
 
         it() In contrast, a service which is filesystem bound, will be
         more busy when more data are transferred. The dispatch mode
         tt(bysize) is apppropriate.
 
         it() The dispatch mode tt(byduration) can also be used when
         network latency is an issue. E.g., if your balancer has back
         ends that are geograpically distributed, then tt(byduration)
         would be a good way to select best available back ends.
 
         it() Furthermore it is noteworthy that tt(dispatchmode
         byduration) is not usable for interactive processes such as
         SSH logins. Idle time of a
         login adds to the duration, while causing (almost) no
         load. Mode tt(byduration) should only be used for automated
         processes that don't wait for user interaction (e.g., SOAP
         calls and other HTTP requests).
 
         it() As a last remark, the dispatching mode tt(byconnections) can
         be used if you don't have other clues for load
         estimations.
 
         E.g., consider a database connection. What's
         heavier on the back end, time-consuming connections, or connections
         where loads of bytes are transferred? Well, that depends. A
         tough tt(select) query that joins multiple tables can be very
         heavy on the back end, though the response set can be quite
         small - and hence the number of
         transferred bytes. That would suggest
         dispatching by duration. However, tt(byduration)
         balancing doesn't respresent the true world, when interactive
         connections can occur where users have an idle TCP connection to
         the database:
         this consumes time, but no bytes (see the SSH login example
         above). In this case, the dispatch mode tt(byconnections) may be
         your best bet.
 
         ) 
 
 
 subsubsect(Averaging size and duration)
 
 The configuration statement tt(dispatchmode bysize) or tt(byduration)
 allows an optional modifier tt(over) em(number), where the stated
 number represents a connection count. When this modifier is present, then
 crossroads will use a moving average over the last em(n) connections to
 compute duration and size figures.
 
 In the real world you'll always want this modifier. E.g., consider two
 back ends that are running for years now, and one of them is suddenly
 overloaded and very busy (it experiences a 'spike' in activity).
 When the tt(over) modifier is absent, then
 the sudden load will hardly show up in the usage figures -- it will
 flatten out due to the large usage figures already stored in the years
 of service.
 
 In contrast, when e.g. tt(over 3) is in effect, then a sudden load
 does show up -- because it highly contributes to the average of three
 connections.
 
 
 subsubsect(Specifying decays)
 
 Decays are also only relevant when crossroads computes the 'next best
 back end' by size (bytes) or duration (seconds). E.g., imagine two
 back ends A and B, both averaged over say 3 connections.
 
 Now when back end A is suddenly hit by a spike,
 its average would go up accordingly. But the back end would never
 again be used, unless B also received a similar spike, because A's
 'usage data' over its last three connections would forever be larger than
 B's data. 
 
 For that reason, you should in real situations probably always
 specify a decay, so that the backend selection algorithm recovers from
 spikes. Note that the usage data of the back end where a decay is
 specified, decay when bf(other) back ends are hit. The decay parameter
 is like specifying how fast your body regenerates when someone else
 does the work.
 
 The below configuration illustrates this:
 
 verb(\
 /* Definition of the service */
 service soap {
     /* Local TCP port */
     port 8080;
 
     /* We'll select back ends by the processing
      * duration
      */
     dispatchmode byduration over 3;
 
     /* First back end: */
     backend A {
         /* Back end IP address and port */
         server 10.1.1.1:8080;
 
         /* When this back end is NOT hit because
          * the other one was less busy, then the
          * usage parameters decay 10% per connection
          */
         decay 10;
     }
 
     /* Second back end: */
     backend B {
         server 10.1.1.2:8080;
         decay 10;
     }
 })
 
 subsubsect(Adjusting the weights)
 
 The back end modifier tt(weight) is useful in situations where your
 back ends differ in respect to performance. E.g,. your back ends may
 be geographically distributed, and you know that a given back end is
 difficult to reach and often experiences network lag.
 
 Or you may have
 one primary back end, a system with a fast CPU and enough memory, and a
 small fall-back back end, with a slow CPU and short on memory. In that
 case you know in advance that the second back end should be used only
 rarely. Most requests should go to the big server, up to a certain load.
 
 In such cases you will know in advance that the best performing back ends
 should be selected the most often. Here's where the tt(weight)
 statement comes in: you can simply increase the weight of the back
 ends with the least performance, so that they are selected less
 frequently.
 
 E.g., consider the following configuration:
 
 verb(\
 service soap {
     port 8080;
     dispatchmode byduration over 3;
     backend A {
         server 10.1.1.1:8080;
         decay 20;
     }
     backend B {
         server 10.1.1.2:8080;
         weight 2;
         decay 10;
     }
     backend C {
         server 10.1.1.3:8080;
         weight 4;
         decay 5;
     }
 })
 
 This will cause crossroads to select back ends by the processing time,
 averaging over the last three connections. However, backend B will kick
 in only when its usage is half of the usage of A (back end B is
 probably only half as fast as A). Backend C will kick in only when its
 usage is a quarter of the usage of A, which is half of the usage of B
 (back end C is probably very weak, and just a fall-back system incase
 both A and B crash). Note also that A's usage data decay much faster
 than B's and C's: we're assuming that this big server recovers quicker
 than its smaller siblings.
 
 
 subsubsect(Throttling the number of concurrent connections)
 
 If you suspect that your service may occasionally receive 'spikes' of
 activity+footnote(which you should always assume), then it might be a
 good idea to protect your service by specifying a maximum number of
 concurrent connections. This protection can be specified on two levels:
 
 description(
         dit(On the service level) a statement like tt(maxconnections
             100;) states that the service as a whole will never
             service more than 100 concurrent connections. This means that
             all your back ends and the crossroads balancer itself
             will be protected from being overloaded.
         dit(On the back end level) a statement like tt(maxconnections 10;)
             states that this particular back end will never have more
             than 10 concurrent connections; regardless of the overall
             setting on the service level. This means that this
             particular back end will be protected from being
             overloaded (regardless of what other back ends may
             experience).)
 
 The tt(maxconnections) statement, combined with a back end selection
 algorithm, allows very fine granularity. The tt(maxconnections) statement
 on the back end level is like a hand brake: even when you specify a
 back end algorithm that would protect a given back end from being used
 too much, a situation may occur where that back end is about to be
 hit. A tt(maxconnections) statement on the level of that back may then
 protect it.
 
 
 subsect(Using an external program to dispatch)
 label(externalhandler)
 
 As mentioned before, Crossroads supports several built-in dispatch
 modes. However, you are always free to hook-in your own dispatch mode
 that determines the next back end using your own specific
 algorithm. This section explains how to do it.
 
 subsubsect(Configuring the external handler)
 
 First, the tt(dispatchmode) statement needs to inform Crossroads that
 an external program will do the job. The syntax is: tt(dispatchmode
 externalhandler) em(program arguments). The em(program) must point to
 an executable program that will be started by Crossroads. The
 specifier em(arguments) can be anything you want; those will be the
 arguments to Crossroads. You can however use the following special
 format specifiers:
 
 INCLUDEFILE(formattable)
 
 Note that the format specifiers such as tt(%b) don't make sense in the
 phase in which an external handler is called, since there is no
 current back end yet (the job of the handler is to supply one).
 
 subsubsect(Writing the external handler)
 
 The external handler is activated using the arguments that are
 specified in tt(/etc/crossroads.conf). The external handler can do
 whatever it wants, but ultimately, it must write a back end name on
 its em(stdout). Crossroads reads this, and if the back end is
 available, uses that back end for the connection.
 
 subsubsect(Examples of external handlers)
 
 This section shows some examples of Crossroads configurations
 vs. external handlers. The sample handlers that are shown here, are
 also included in the Crossroads distribution, under the directory
 tt(etc/). Also note that the examples shown here are just
 quick-and-dirty Perl scripts, meant to illustrate only. Your
 applications may need other external handlers, but you can use the
 shown scripts as a starting point.
 
 subsubsubsect(Round-robin dispatching)
 
 This example is trivial in the sense that round-robin dispatching is
 already built into Crossroads, so
 that using an external handler for this purpose only slows down
 Crossroads. However, it's a good starting example.
 
 The Crossroads configuration is shown below:
 
 verb(\
 service test {
     port 8001;
     verbosity on;
     revivinginterval 5;
     
     dispatchmode externalhandler
         /usr/local/src/crossroads/etc/dispatcher-roundrobin
             %1b %1a %2b %2a;
 
     backend testone {
         server localhost:3128;
         verbosity on;
     }
     backend testtwo {
         server locallhost:3128;
         verbosity on;
     }
 })
 
 The relevant tt(dispatchmode) statement invokes the external program
 tt(dispatcher-roundrobin) with four arguments: the name of the first
 back end (tt(testone)), its availability (0 or 1), the name of the
 second back end (tt(testtwo)) and its availability (0 or 1).
 
 The external handler, which is also included in the Crossroads
 distribution, is shown below. It is a Perl script.
 
 verb(\
 #!/usr/bin/perl
 
 use strict;
 
 # Example of a round-robin external dispatcher. This is totally
 # superfluous, Crossroads has this on-board; if you use the external
 # program for determining round-robin dispatching, then you'll only
 # slow things down. This script is just meant as an example.
 
 # Globals / configuration
 # -----------------------
 my $log = '/tmp/exthandler.log';    # Debug log, set to /dev/null to suppress
 my $statefile = '/tmp/rr.last';	    # Where we keep the last used
 
 # Logging
 # -------
 sub msg {
     return if ($log eq '/dev/null' or $log eq '');
     open (my $of, ">>$log") or return;
     print $of (scalar(localtime()), ' ', @_);
 }
 
 # Read the last used back end
 # ---------------------------
 sub readlast() {
     my $ret;
     
     if (open (my $if, $statefile)) {
         $ret = <$if>;
         chomp ($ret);
         close ($if);
         msg ("Last used back end: $ret\n");
         return ($ret);
     }
     msg ("No last-used back end (yet)\n");
     return (undef);    
 }
 
 # Write back the last used back end, reply to Crossroads and stop
 # ---------------------------------------------------------------
 sub reply ($) {
     my $last = shift;
 
     if (open (my $of, ">$statefile")) {
         print $of ("$last\n");
     }
     print ("$last\n");
     exit (0);
 }
 
 # Main starts here
 # ----------------
 
 # Collect the cmdline arguments. We expect pairs of backend-name /
 # backend-availablility, and we'll store only the available ones.
 msg ("Dispatch request received\n");
 my @backend;
 for (my $i = 0; $i <= $#ARGV; $i += 2) {
     push (@backend,  $ARGV[$i]) if ($ARGV[$i + 1]);
 }
 msg ("Available back ends: @backend\n");
 
 # Let's see what the last one is. If none found, then we return the
 # first available back end. Otherwise we need to go thru the list of
 # back ends, and return the next one in line.
 my $last = readlast();
 if ($last eq '') {
     msg ("Returning first available back end $backend[0]\n");
     reply ($backend[0]);
 }
 
 # There **was** a last back end.  Try to match it in the list,
 # then return the next-in-line.
 for (my $i = 0; $i < $#backend; $i++) {
     if ($last eq $backend[$i]) {
         msg ("Returning next back end ", $backend[$i + 1], "\n");
         reply ($backend[$i + 1]);
     }
 }
 
 # No luck.. run back to the first one.
 msg ("Returning first back end $backend[0]\n");
 reply ($backend[0]);)
 
 The working of the script is basically as follows:
 
 itemization(
         it() The argument list is scanned. Back ends that are
         available are collected in an array tt(@backend).
 
         it() The script queries a state file tt(/tmp/rr.last). If a
         back end name occurs there, then the next back end is looked
         up in tt(@backend) and returned to Crossroads. If no last back
         is unknown or can't be matched, then the first available back
         end (first element of tt(@backend)) is returned to Crossroads.
 
         it() Informing Crossroads is done via the subroutine
         tt(reply()). This code writes the selected back end to file
         tt(/tmp/rr.last) (for future usage) and prints the back end
         name to em(stdout).
 
         it() The script logs its actions to a file
         tt(/tmp/exthandler.log). This log file can be inspected for
         the script's actions.)
 
         
 subsubsubsect(Dispatching by the client IP address)
 
 The following example shows a useful real-life situation. The
 situation is as follows:
 
 itemization(
         it() Crossroads is used as a single-address point to forward
         Remote Desktop requests to a farm of Windows systems, where
         users can work via remote access;
 
         it() However, users may stop their session, and when they
         re-connect, they expect to be sent to the Windows system that
         they had worked on previously;
 
         it() Client PC's have their distinct IP addresses, which
         distinguishes them.
 
         it() Of four windows systems, two are large servers, and two
         are small ones. We'll want to assign large servers to clients
         when we have a choice.)
 
 The requirements resemble session stickiness in HTTP, except that the remote
 desktop protocol doesn't support stickiness. This situation is a
 perfect example of how an external handler can help:
 
 itemization(
         it() A suitable dispatch mode isn't yet available in
         Crossroads, but can be easily coded in an external handler;
 
         it() The potential delay due to the calling of an external
         handler won't even be noticed. This is a network service where
         the connection time isn't critical; we'd expect only a few
         (albeit lengthy) TCP connections.)
 
 The approach to the solution of this problem uses several external
 program hooks:
 
 itemization(
         it() An external dispatcher handler will be responsible for
         suggesting a back end, given a client IP and given the current
         timestamp. This handler will consult an internal
         administration to see whether the stated IP address should
         re-use a back end, or to determine which back end is free for usage.
         it() An external hook tt(onstart) will be responsible for
         updating the internal administration; i.e., to flag a back end
         as 'occupied'.
         it() The external hooks tt(onfailure) and tt(onend) will be
         responsible for flagging a back end as 'free' again; i.e., for
         erasing any previous information that states that the back end
         was occupied.)
 
 The Crossroads configuration is shown below. Only four Windows back
 ends are shown. Each back end is configured on a
 given IP address, port 3389, and is limited to one concurrent connection
 (otherwise a new user might 'steal' a running desktop session).
 
 verb(\
 service rdp {
     port 3389;
     revivinginterval 5;
 
     /* rdp-helper dispatch IP STAMP ...  will suggest a back end to use,
      * arguments are for all back ends: name, availability, weight */
     dispatchmode externalhandler
         /usr/local/src/crossroads/etc/rdp-helper dispatch %r %e
             %1b %1a %1w
             %2b %2a %2w
             %3b %3a %3w
             %4b %4a %4w;
             
     backend win1 {
         server 10.1.1.1:3389;
         maxconnections 1;
         /* rdp-helper start IP STAMP BACKEND will log the actual start
          * of a connection;
          * rdp-helper end IP will log the ending of a connection */
         onstart /usr/local/src/crossroads/etc/rdp-helper start %r %e %b;
         onend   /usr/local/src/crossroads/etc/rdp-helper end %r;
         onfail  /usr/local/src/crossroads/etc/rdp-helper end %r;
     }
     backend win2 {
         server 10.1.1.2:3389;
         maxconnections 1;
         onstart /usr/local/src/crossroads/etc/rdp-helper start %r %e %b;
         onend   /usr/local/src/crossroads/etc/rdp-helper end %r;
         onfail  /usr/local/src/crossroads/etc/rdp-helper end %r;
     }
     backend win3 {
         server 10.1.1.3:3389;
         maxconnections 1;
         weight 2;
         onstart /usr/local/src/crossroads/etc/rdp-helper start %r %e %b;
         onend   /usr/local/src/crossroads/etc/rdp-helper end %r;
         onfail  /usr/local/src/crossroads/etc/rdp-helper end %r;
     }
     backend win4 {
         server 10.1.1.4:3389;
         maxconnections 1;
         weight 3;
         onstart /usr/local/src/crossroads/etc/rdp-helper start %r %e %b;
         onend   /usr/local/src/crossroads/etc/rdp-helper end %r;
         onfail  /usr/local/src/crossroads/etc/rdp-helper end %r;
     }
 })
 
 Depending on the dispatcher stage, the exernal handler tt(rdp-helper)
 is invoked in different ways:
 
 description(
         dit(During dispatching) the helper is called to suggest a back
         end. The arguments are an action indicator tt(dispatch), the
         client's IP address, the timestamp, and four triplets that
         represent back ends: per back end its name, its availability,
         and its weight. The purpose of the helper is to tell
         Crossroads which back end to use.
 
         dit(During connection start) the helper will be invoked to
         inform it of the start of a connection, given a client IP
         address.
 
         dit(When a connection terminates) the helper will be invoked
         to inform it that the connection has ended.)
 
 Here's the external handler as Perl script. It uses the module
 tt(GDBM_File) which most likely will not be part of standard Perl
 distributions, but can be added using CPAN. (Alternatively, any other
 database module can be used.)
 
 verb(\
 #!/usr/bin/perl
 
 use strict;
 use GDBM_File;
 
 # Global variables and configuration
 # ----------------------------------
 my $log = '/tmp/exthandler.log';    # Debug log, set to /dev/null to suppress
 my $cdb = '/tmp/client.db';         # GDBM database of clients
 my %db;                             # .. and memory representation of it
 my $timeout = 24*60*60;             # Timeout of a connection in secs
 
 # Logging
 # -------
 sub msg {
     return if ($log eq '/dev/null' or $log eq '');
     open (my $of, ">>$log") or return;
     print $of (scalar(localtime()), ' ', @_);
     close ($of);
 }
 
 # Reply a back end to the caller and stop processing.
 # ---------------------------------------------------
 sub reply ($) {
     my $b = shift;
     msg ("Suggesting $b to Crossroads.\n");
     print ("$b\n");
     exit (0);
 }
 
 # Is a value in an array
 # ----------------------
 sub inarray {
     my $val = shift;
     for my $other (@_) {
         return (1) if ($other eq $val);
     }
     return (0);
 }
 
 # A connection is starting
 # ------------------------
 sub start {
     my ($ip, $stamp, $backend) = @_;
     msg ("Logging START of connection for IP $ip on stamp $stamp, ",
          "back end $backend\n");
     $db{$ip} = "$backend:$stamp";
 }
 
 # A connection has ended
 # ----------------------
 sub end {
     my $ip = shift;
     msg ("Logging END of connection for IP $ip\n");
     $db{$ip} = undef;
 }
 
 # Request to determine a back end
 # -------------------------------
 sub dispatch {
     my $ip = shift;
     my $stamp = shift;
 
     msg ("Request to dispatch IP $ip on stamp $stamp\n");
     
     # Read the next arguments. They are triplets of
     # backend-name / availability / weight. Store if the back end is
     # available.
     my (@backends, @weights);
     for (my $i = 0; $i < $#_; $i += 3) {
         if ($_[$i + 1] != 0) {
             push (@backends, $_[$i]);
             push (@weights,  $_[$i + 2]);
             msg ("Candidate back end: $_[$i] with weight ", $_[$i + 2], "\n");
         }
     }
 
     # See if this is a reconnect by a previously seen client IP. We'll
     # treat this as a reconnect if the timeout wasn't yet exceeded.
     if ($db{$ip} ne '') {
         my ($last_backend, $last_stamp) = split (/:/, $db{$ip});
         msg ("IP $ip had last connected on $last_stamp to $last_backend\n");
         if ($stamp < $last_stamp + $timeout) {
             msg ("Timeout not yet exceeded, this may be a reconnect\n");
             # We'll allow a reconnect only if the stated last_backend is
             # free (sanity check).
             if (inarray ($last_backend, @backends)) {
                 msg ("Last back end $last_backend is available, ",
                      "letting through\n");
                 reply ($last_backend);
             } else {
                 msg ("Last used back end isn't free, suggesting a new one\n");
             }
         } else {
             msg ("Timeout exceeded, suggesting a new back end\n");
         }
     } else {
         msg ("Np preveious connection data, suggesting a new back end\n");
     }
 
     my $bestweight = -1;
     my $bestbackend;
     for (my $i = 0; $i <= $#weights; $i++) {
         if ($bestweight == -1 or $bestweight > $weights[$i]) {
             $bestweight  = $weights[$i];
             $bestbackend = $backends[$i];
         }
     }
 
     msg ("Best back end: $bestbackend (given weight $bestweight)\n");
     reply ($bestbackend);
 }
 
 # Main starts here
 # ----------------
 msg ("Start of run, attaching GDBM database '$cdb'\n");
 tie (%db, 'GDBM_File', $cdb, &GDBM_WRCREAT, 0600);
 
 # The first argument must be an action 'dispatch', 'start' or 'end'.
 # Depending on the action, we do stuff.
 my $action = shift (@ARGV);
 if ($action eq 'dispatch') {
     dispatch (@ARGV);
 } elsif ($action eq 'start') {
     start (@ARGV);
 } elsif ($action eq 'end') {
     end (@ARGV);
 } else {
     print STDERR ("Usage: rdp-helper {dispatch|start|end} args\n");
     exit (1);
 })
 
 
 subsect(HTTP Session Stickiness)
 
 This section focuses on HTTP session stickiness. This term refers to
 the ability of a balancer to route a conversation between browser and
 a backend farm always to the same back end. In other words: once a
 back end is selected by the balancer, it will remain the back end of
 choice, even for subsequent connections.
 
 subsubsect(Don't use stickiness!)
 
 The rule of thumb as far as the balancer is concerned, is: bf(Do not
 use HTTP session stickiness unless you really have to.) Enabling
 session stickiness hampers failover, balancing and performance:
 
 itemization(
     it() Failover is hampered because during the session,
          the balancer has to assign new connections to the same back
          end that was selected at the start of a session. If the back
          end suddenly goes 'down', then the session will most likely
          crash. (Actually, when a back end becomes unreachable in the
          middle of a session, Crossroads will assign a new back end to
          that session. This will most likely result in a malfunction
          of the underlying application.)
     it() Balancing is hampered because at the start of the session,
          the balancer has selected the next-best back end. But during
          the session, that back end may well become overloaded. The
          balancer however must continue to send the requests there.
     it() Performance is hampered because crossroads needs to 'unpack'
          messages as they are passed to and fro. That's because
          crossroads needs to check the HTTP headers in the messages
          for persistence cookies.)
 
 There is a number of measures that you can take to avoid using session
 stickiness. E.g., session data can be 'shared' between web back
 ends. PHP offers functionality to store session data in a database, so
 that all PHP applications have access to these data. Application
 servers such as Websphere can be configured to replicate session data
 between nodes.
 
 subsubsect(But if you must..)
 
 However, if you bf(must) use session stickiness, then proceed as
 follows:
 
 itemization(
     it() At the level of a tt(service) description, set the type to
          tt(http). 
     it() At the level of each back end description, configure the
          tt(stickycookie) and a tt(addclientheader) directives.)
 
 Once crossroads sees that, it will examine each HTTP message that it
 shuttles between client and back end:
 
 itemization(
     it() If there is no persistence cookie in the HTTP headers of a
          client's request, then the message must be the first one and
          a new session should be established. 
          Crossroads selects an appropriate back
          end, sends the message to that back end, catches the reply,
          and inserts a tt(Set-Cookie) directive.
     it() If there is a persistence cookie in the HTTP headers of a
          client's request, then the request is part of an already
          established session. Crossroads analyzes the cookie and
          forwards the request to the appropriate back end.)
 
 Below is a short example of a configuration.
 
 verb(\
 service www {
     port 80;
     type http;
     revivinginterval 15;
     dispatchmode byconnections;
 
     backend one {
         server 10.1.1.100:80;
         stickycookie XRID=100;
         addclientheader "Set-Cookie: XRID=100; Path=/";
     }
 
     backend two {
         server 10.1.1.101:80;
         stickycookie XRID=101;
         addclientheader "Set-Cookie: XRID=101; Path=/";
     }
 })
 
 Note how the cookie names and values in the directives
 tt(stickycookie) and tt(addclientheader) match. That is obviously a
 prerequisite for stickiness.
 
 
 subsect(Passing the client's IP address)
 
 Since Crossroads just shuttles bytes to and fro, meta-information of
 network connections is lost. As far as the back ends are concerned,
 their connections originate at the Crossroads junction.
 For example, standard Apache access logs will show the IP address of
 Crossroads. 
 
 In order to compensate for this, Crossroads can insert a special
 header in HTTP connections, to inform the back end of the original
 client's IP address. In order to enable this, the Crossroads
 configuration must state the following:
 
 itemization(
     it() The service type must be tt(http), and not tt(any);
     it() In the back end definition, the following statement must
          occur: nl()
          tt(addserverheader "X-Real-IP: %r";) nl()
          You are of course free to choose the header name; the here
          used tt(X-Real-IP) is a common name for this purpose.)
 
 After this, HTTP traffic that arrives at the back ends has a new
 header: tt(X-Real-IP), holding the client's IP address.
 bf(Note that) once the type is set to tt(http), Crossroads'
 performance will be hampered -- all passing messages will have to be
 unpacked and analyzed.
 
 subsubsect(Sample Crossroads configuration)
 
 The below sample configuration shows two HTTP back ends that receive
 the client's IP address:
 
 verb(
 service www {
     port 80;
     type http;
     revivinginterval 5;
     dispatchmode roundrobin;
 
     backend one {
         server 10.1.1.100:80;
         addserverheader "X-Real-IP: %r";
     }
 
     backend two {
         server 10.1.1.200:80;
         addserverheader "X-Real-IP: %r";
     }
 })
 
 
 subsubsect(Sample Apache configuration)
 
 The method by which each back end analyzes the header tt(X-Real-IP)
 will obviously be different per server implementations. However, a
 common method with the Apache webserver is to log the client's IP
 address into the access log.
 
 Often this is accomplished using the log format tt(custom), defined as
 follows:
 
 verb(\
 LogFormat "%h %l %u %t %D \"%r\" %>s %b" common
 CustomLog logs/access_log common)
 
 The first line defines the format tt(common), with the remote host
 specified by tt(%h). The second line sends access information to a log
 file tt(logs/access_log), using the previously defined format
 tt(common).
 
 Furtunately, Apache's tt(LogFormat) allows one to log contents of
 headers. By replacing the tt(%h) with tt(%{X-Real-IP}i), the desired
 information is sent to the log. Therefore, normally you can simply
 redefine the tt(common) format to 
 
 verb(\
 LogFormat "%{X-Real-IP}i %l %u %t %D \"%r\" %>s %b" common)
 
 
 subsect(Debugging network traffic)
 
     Incase the traffic between
     client and backend
     must be debugged, the statement tt(trafficlog) em(filename) can
     be issued. This causes the traffic to be dumped in hexadecimal
     format to the stated filename.
 
     Traffic sent by the client is prefixed by a bf(C), traffic sent by
     the back end is prefixed by a bf(B). Below is a sample traffic
     dump of a browser trying to get a HTML page. The server replies
     that the page was not modified.
 
     verb(\
 C 0000  47 45 54 20 68 74 74 70 3a 2f 2f 77 77 77 2e 63 GET http://www.c
 C 0010  73 2e 68 65 6c 73 69 6e 6b 69 2e 66 69 2f 6c 69 s.helsinki.fi/li
 C 0020  6e 75 78 2f 6c 69 6e 75 78 2d 6b 65 72 6e 65 6c nux/linux-kernel
 C 0030  2f 32 30 30 31 2d 34 37 2f 30 34 31 37 2e 68 74 /2001-47/0417.ht
 C 0040  6d 6c 20 48 54 54 50 2f 31 2e 31 0d 0a 43 6f 6e ml HTTP/1.1..Con
 C 0050  6e 65 63 74 69 6f 6e 3a 20 63 6c 6f 73 65 0d 0a nection: close..
 .
 . etcetera
 .
 B 0000  48 54 54 50 2f 31 2e 30 20 33 30 34 20 4e 6f 74 HTTP/1.0 304 Not
 B 0010  20 4d 6f 64 69 66 69 65 64 0d 0a 44 61 74 65 3a  Modified..Date:
 B 0020  20 54 75 65 2c 20 31 32 20 4a 75 6c 20 32 30 30  Tue, 12 Jul 200
 B 0030  35 20 30 39 3a 34 39 3a 34 37 20 47 4d 54 0d 0a 5 09:49:47 GMT..
 B 0040  43 6f 6e 74 65 6e 74 2d 54 79 70 65 3a 20 74 65 Content-Type: te
 B 0050  78 74 2f 68 74 6d 6c 3b 20 63 68 61 72 73 65 74 xt/html; charset
 .
 . etcetera
 .)
 
     Turning on traffic dumps will em(significantly)
     slow down crossroads.
 
     Besides tt(trafficlog), there is also a directive
     tt(throughputlog). This directive also takes one argument, a
     filename. The file is appended, and the following information is
     logged:
 
     itemization(
         it() The process ID of the crossroads image that serves the
         TCP connection;
         it() The time of the request, in seconds and microseconds
         since start of the run;
         it() A bf(C) when the request originated at the client, or
         bf(B) when the request originated at the back end;
         it() The first 100 bytes of the request.)
 
     As an example, consider the following (the lines are shortened for
     brevity and prefixed by line numbers for clarity):
 
     verb(
 1 0000594 0.000001 C GET http://public.e-tunity.com/index.html...
 2 0000594 0.173713 B HTTP/1.0 200 OK..Date: Fri, 18 Nov 2005 0...
 3 0000594 0.278125 B  width="100" bgcolor="#e0e0e0" valign="to...
 4 0000595 0.000001 C GET http://public.e-tunity.com/css/style/...
 5 0000594 0.944339 B /a></td>..  </tr>.</table>.</td><td class...
 6 0000594 0.946356 B smallboxdownl">Download</td>..  <td class...
 7 0000594 0.961102 B td><td class="smallboxodd" valign="top"><...
 8 0000595 0.698215 B HTTP/1.0 304 Not Modified..Date: Fri, 18 ...)
 
     This tells us that:
 
     itemization(
         it() Line 1:  PID 594 served a request that originated at
              the client. The corresponding time is (almost) 0 seconds,
              so this is really the start of the run.
         it() Line 2: A back end replied 0.17 seconds later, and
              0.28 seconds later, it was still replying (this is the
              third line, again a bf(B)-type transmission).
         it() Line 4: PID 595 served a request that originated
              at the client. Again, the corresponding time is (almost)
              0 seconds, since this is the first conversation part of
              this connection.
         it() Lines 5 to 7: This is the continuation of line 2. Line 7
              is the last line of the bf(B) series (not visible from
              the example, but trust me, it is), so that we may
              conclude that it took the back end 0.96 seconds to serve
              the file tt(index.html) requested in line 1.
         it() Line 8: This is the answer to the client's request of
              line 4 (you can tell by the process ID  number).
              So the back end took 0.68 seconds to confirm that
              the stylesheet requested in line 4 wasn't modified.)
 
     It is also worth while remembering that the start time of a bf(C)
     request is the time that crossroads sees the activity. Any latency
     between the true client and crossroads is obviously not
     included. This is illustrated by the below simple ASCII art:
 
     verb(
 client ---->---->---->--->*crossroads ====>====>====>
                                                      \    
                                                   back end
                                                      /
 client ----<----<----<---< crossroads ====<====<====<
 )
 
     This simple picture shows a typical HTTP request that originates
     at a client, travels to crossroads, and is relayed via the back
     end. The bf(C) entry in a throughput log is the time when
     crossroads sees the request, indicated by an asterisk. The bf(B)
     entries are the times that it takes the back end to answer,
     indicated by tt(===) style lines. Therefore, the true roundtrip
     time will be longer than the number of seconds that are logged in
     the throughput log: the latency between client and crossroads
     isn't included in that measurement.
     
     Summarizing, the throughput times of a client-back end connection
     can be analyzed using the directive tt(throughputlog). In a
     real-world analysis, you'd probably want to write up a script to
     analyze the output and to compute round trip times. Such scripts
     are not (yet) included in Crossroads.
 
 subsect(Limiting Access to Crossroads by Client IP Address)
 
 subsubsect(General Examples)
 
 The directives tt(allowfrom), tt(denyfrom), tt(allowfile) and
 tt(denyfile) can be used to instruct Crossroads to specifically allow
 access by using a "whitelist" of IP addresses, or to specifically deny
 access by using a "blacklist". E.g., the following configuration
 allows access to service tt(webproxy) only to em(localhost):
 
 verb(\
 service webproxy {
     port 8000;    
     allowfrom 127.0.0.1;
     backend one {
         .
         . Back end definitions occur here
         .
     }
     .
     . Other back ends or other service directives
     . may occur here
     .
 })
 
 In this example there is a "whitelist" having only one entry: IP
 address 127.0.0.1, or em(localhost). (Incidentally, the same behaviour
 could be accomplished by stating em(bindto 127.0.0.1), in which case
 Crossroads would only listen to the local network device.)
 
 In the same vein, the directive tt(allowfrom 127.0.0.1 192.168.1/24)
 would allow access to em(localhost) and to all IP addresses that start
 with 192.168.1. The specifier tt(192.168.1/24) states that there are
 three network bytes (192, 168 and 1), and 24 bits (or 3 bytes) are
 relevant; so that the fourth network byte doesn't matter.
 
 subsubsect(Using External Files)
 
 The directives tt(allowfile) and tt(denyfile) allow you to specify IP
 addresses in external files. The Crossroads configuration states
 e.g. tt(allowfile /tmp/allow.txt), and the IP addresses are then in
 tt(/tmp/allow.txt). The format of tt(/tmp/allow.txt) is as follows:
 
 itemization(
         it() The specifications follow again em(p.q.r.s/mask), where
         p, q, r and s are network bytes which can be left out on the
         right hand side when the mask allows it;
 
         it() The specifications must be separated by white space
         (spaces, tabs or newlines).)
 
 E.g., the following is a valid example of an external specification
 file:
 
 verb(\
 127.0.0.1
 192.168.1/24
 10/8)
 
 When external files are in effect, then the signal tt(SIGHUP) (1)
 causes Crossroads to reload the external file. E.g., while Crossroads
 is running, you may edit tt(/tmp/allow.txt), and then issue tt(killall
 -1 crossroads). The new contents of tt(/tmp/allow.txt) will be
 reloaded.
 
 subsubsect(Mixing Directives)
 
 Crossroads allows to mix all directives in one service
 description. However, some mixes are less meaningful than others. It's
 up to you to take this into account.
 
 The following rules apply:
 
 itemization(
         it() Blacklisting and whitelisting can be used together. When
         combined, the blacklist will always be interpreted
         first. E.g., consider the following directives:
 
         verb(\
 allowfrom 192.168.1/24
 denyfrom  192.168.1.100)
 
         Given the fact that the deny list is checked first, client
         192.168.1.100 won't be able to access Crossroads. Then the
         allow list will be checked, stating that all clients whose IP
         address starts with 192.168.1 may connect. The effect will be
         that e.g., client 192.168.1.1 may connect, 192.168.1.2 may
         connect too, 192.168.1.100 will be blocked, and 10.1.1.1 will
         be blocked as well.
 
         Now consider the following directives:
 
         verb(\
 allowfrom 192.168.1.100 127.0.0.1
 denyfrom  192.168.1/24)
 
         This will first of all deny access to all IP addresses that
         start with 192.168.1. So the rule that allows 192.168.1.100
         won't ever be effective. The net result will be that access
         will be granted to 127.0.0.1, and IP addresses that don't
         match 192.168.1/24.
 
         it() Blacklisting or whitelisting can be left out.
         A list is considered empty when no appropriate directives
         occur in tt(/etc/crossroads.conf), or when the directive
         points to an empty or non-existent external file.
         
         it() Using tt(*from) and tt(*file) statements is allowed, but
         doesn't make sense. E.g., the following configuration sample
         is such a case:
 
         verb(\
 allowfrom 127.0.0.1 192.168.1/24
 allowfile /tmp/allow.txt)
 
         There is a technical reason for this. Once Crossroads
         processes the tt(allowfile) directive, then the whole
         whitelist is cleared (thereby removing the entries 127.0.0.1
         and 192.168.1/24), and new entries are reloaded from the
         file. The net result is that the tt(allowfrom) specification
         is overruled.
 
         Crossroads doesn't check for such configurations, which are
         syntactially correct, but make no semantic sense.)
         	
 
 subsect(Configuration examples)
 
 As a general hint, use tt(crossroads sampleconf) to view the most
 up-to-date examples of configurations. The description below shows a
 few examples too.
 
 
 subsubsect(A load balancer for three webserver back ends)
 
 The following configuration example binds crossroads to port 80 of the
 current server, and distributes the load over three back ends. This
 configuration shows most of the possible settings.
 
 verb(\
 service www {
     /* We don't need session stickyness. */
     type any;
     
     /* Port on which we'll listen in this service: required. */
     port 8000;
 
     /* What IP address should this service listen? Default is 'any'.
      * Alternatively you can state an explicit IP address, such as
      * 127.0.0.1; that would bind the service only to 'localhost'. */
     bindto any;
     
     /* Verbose reporting or not. Default is off. */    
     verbosity on;
     
     /* Dispatching mode, or: How to select a back end for an incoming
      * request. Possible values:
      *   roundrobin: just the next back end in line
      *   random: like roundrobin, but at random to make things more
      *          confusing. Probably only good for testing.
      *   bysize: The backend that transferred the least nr of bytes
      *          is the next in line. As a modifier you can say e.g.
      *          bysize over 10, meaning that the 10 last connections will
      *          be used to compute the transfer size, instead of all
      *          transfers.
      *   byduration: The backend that was active for the shortest time
      *          is the next in line. As a modifier you can say e.g.
      *          byduration of 10 to compute over the last 10 connections.
      *   byconnections: The back end with the least active connections
      *          is the next ine line.
      *   byorder: The first available back end is always taken.
      */
     dispatchmode byduration over 5;
 
     /* Interval at which we'll check whether a temporarily unavailable
      * backend has woken up.
      */
     revivinginterval 5;
     
     /* TCP backlog of connections. Default is 0 (no backlog, one
      * connection may be active).
      */
     backlog 5;
     
     /* For status reporting: a shared memory key. Default is the same
      * as the port number, OR-ed by a magic number.
      */
     shmkey 8000;
 
     /* This controls when crossroads should consider a connection as
      * finished even when the TCP sockets weren't closed. This is to
      * avoid hanging connections that don't do anything. NOTE THAT when
      * crossroads cuts off a connection due to timeout exceed, this is
      * not marked as a failure, but as a success. Default is 0: no timeout.
      */
     connectiontimeout 300;
 
     /* The max number of allowed client connections. When present, connections
      * won't be accepted if the max is about to be exceeded. When
      * absent, all connections will be accepted, which might be misused
      * for a DOS attack.
      */
     maxconnections 300;
 
     /* Now let's define a couple of back ends. Number 1: */
     backend www_backend_1 {
         /* The server and its port, the minimum configuration. */
         server httpserver1;
         port 9010;
         /* The 'decay' of usage data of this back end. Only relevant
          * when the whole service has 'dispatchmode bysize' or
          * 'byduration'. The number is a percentage by which the usage
          * parameter is decreased upon each connection of an other back
          * end.
          */
         decay 10;
         
         /* To see what's happening in /var/log/messages: */
         verbosity on;
     }
 
     /* The second one: */
     backend www_backend_2 {
         /* Server and port */
         server httpserver2;
         port 9011;
 
         /* Verbosity of reporting when this back end is active */
         verbosity on;
 
         /* Decay */
         decay 10;
 
         /* This back end is twice as weak as the first one */
         weight 2;
 
         /* Event triggers for system commands upon succesful activation
          * and upon failure.
          */
         onsuccess echo 'success on backend 2' | mail root;
         onfailure echo 'failure on backend 2' | mail root;
     }
 
     /* And yet another one.. this time we will dump the traffic
      * to a trace file. Furthermore we don't want more than 10 concurrent
      * connections here. Note that there's also a total maxconnections for the
      * whole service.
      */
     backend www_backend_3 {
         server httpserver3;
         verbosity on;
         port 9000;
         verbosity on;
         decay 10;
         trafficlog /tmp/backend.3.log;
         maxconnections 10;
     }
 })
 
 subsubsect(An HTTP forwarder when travelling)
 
 As another example, here's my tt(crossroads.conf) that I use on my
 Unix laptop. The problem that I face is that I need many HTTP proxy
 configurations (at home, at customers' sites and so on) but I'm too
 lazy to reconfigure browsers all the time.
 
 Here's how it used to be before crossroads:
 
 itemization(
         it() At home, I would surf through a squid proxy on my local
         machine. The browser proxy setting is then
         tt(http://localhost:3128).
 
         it() Sometimes I start up an SSH tunnel to our offices. The
         tunnel has a local port 3129, and connects to a squid proxy on
         our e-tunity server. Hence, the browser proxy is then
         tt(http://localhost:3129).
 
         it() At a customer's location I need the proxy
         tt(http://10.120.34.113:8080), because they have configured it
         so.
 
         it() And in yet other instances, I use a HTTP diagnostic tool
         url(Charles)(http://www.xk72.com/charles)
         that sits between browser and website and shows me
         what's happening. I run charles on my own machine and it
         listens to port 8888, behaving like a proxy. The browser
         configuration for the proxy is then
         tt(http://localhost:8888).)
 
 Here's how it works with a crossroads configuration:    
 
 itemization(
         it() I have configured my browsers to use
         tt(http://localhost:8080) as the proxy. For all situations.
         
         it() I use the following crossroads configuration, and let
         crossroads figure out which proxy backend works, and which
         doesn't. Note two particularities:
 
         itemization(
                 it() The statement tt(dispatchmode byorder). This
                 makes sure that once crossroads determines which
                 backend works, it will stick to it. This usage of
                 crossroads doesn't need to balance over more than one
                 back end.
 
                 it() The statement tt(bindto 127.0.0.1) makes sure
                 that requests from other interfaces than loopback
                 won't get serviced.)
 
 verb(\
 service HttpProxy {
     port 8080;
     bindto 127.0.0.1;
     verbosity on;
     dispatchmode byorder;
     revivinginterval 15;
 
     backend Charles {
         server localhost:8888;
         verbosity on;
     }
 
     backend CustomerProxy {
         server 10.120.34.113:8080;
         verbosity on;
     }
 
     backend SshTunnel {
         server localhost:3129;
     }
 
     backend LocalSquid {
         server localhost:3128;
     }
 }))
 
 As a final note, the commandline argument tt(tell) can be used to
 influence crossroad's own detection mechanism of back end availability
 detection. E.g., if in the above example the back ends tt(SshTunnel)
 and tt(LocalSquid) are both active, then tt(crossroads tell httpproxy
 sshtunnel down) will 'take down' the back end tt(SshTunnel) -- and
 will automatically cause crossroads to switch to tt(LocalSquid).
 
 
 subsubsect(SSH login with enforced idle logout)
 
 The following example shows how crossroads 'throttles' SSH
 logins. Connections are accepted on port
 22 (the normal SSH port) and forwarded to the actual SSH daemon
 which is running on port 2222.
 
 Note the usage of the
 tt(connectiontimeout) directive. This makes sure that users are logged
 out after 10 minutes of inactivity. Note also the tt(maxconnections)
 setting, this makes sure that no more than 10 concurrent logins occur.
 
 verb(\
 service Ssh {
     port 22;
     backlog 5;
     maxconnections 10;
     connectiontimeout 600;
     backend TrueSshDaemon {
         server localhost:2222;
     }
 })