Basic CGI starts a new process to handle each request. This is simple, but resource hungry. To make things more efficient, a number of protocols have developed that keep a CGI server running in the background. The frontend webserver then passes requests back to the CGI server for processing without having to fire up a new program.

FastCGI is the original CGI server protocol, but here we'll focus on it's simpler cousin SCGI. I write most of my scripts in Python, so I use the handy scgi module to write the server (www-apps/scgi on Gentoo). There's a great article on Linux Journal to walk you through things, but in short, you need something like:

import scgi
import scgi.scgi_server
import urlparse

class YourHandler(scgi.scgi_server.SCGIHandler):
    def produce(self, env, bodysize, input, output):
        url = env.get('DOCUMENT_URI', None)
            if url is None:  # fall back to the request URI                         
            url = env.get('REQUEST_URI', None)
        data = urlparse.parse_qs(env.get('QUERY_STRING', ''))
        output.write('Status: 200 OK\r\n')
        output.write('Content-type: text/plain; charset=UTF-8\r\n')
        output.write('\r\n')
        output.write('URI: {}\n'.format(url))
        output.write('data: {}\n'.format(data))

if __name__ == '__main__':
    server = scgi.scgi_server.SCGIServer(
        handler_class=YourHandler, host='localhost', port=4000)
    server.serve()

Fire this script up, point your server at localhost:4000, and you're good to go!

I've written up a little script to test your SCGI server from the command line: scgi-test.py:

$ scgi-test.py /your/uri/
Status: 200 OK
Content-type: text/html; charset=UTF-8

<!DOCTYPE html>
…

Configuring your frontend webserver to point to the SCGI server is beyond the scope of this post. I give an Nginx example in my Nginx post.