The Joyent Community

Read the past post on this page to get around this issue arpan

http://discuss.joyent.com/viewtopic.php?id=17280&p=3

Although I did that, but not all my templates are being called into the admin end now, so I reverted to django0.96 and they are back. I'm trying to find a way to install and run django 1.0 as well, but I have no idea how we're going to do that if django0.96 is in the site-packages directory.

Any tips I'd be grateful.

Here's the current state of my notes on this. Happy new year. Let me know if anything in here is blatantly screwed up, or you've got a better idea, or I've left my passwords in there in plaintext or something.

###################################################
Deploying Django-1.0 on a Joyent Shared Accelerator

###################################################

Overview
====
When releasing Django projects into the wild on a Joyent Shared Accelerator, we're going to follow the tried-and-true path of proxying to lighttpd via FastCGI, since the shared hosts don't offer Apache's mod_python (the recommended deployment).

In this document, we'll refer to your username on Joyent's server as ${USER}, the hostname of your server as ${HOST}, and your DNS domain as ${DOMAIN}.

Set up Apache as a proxy server for lighttpd
========================================
Request a lighttpd port
-----------------------
You'll need to file a `support ticket`_ and ask an admin to assign a TCP port you can use for lighttpd. It can take a business day for your request to go through, so do this first.

We'll refer to the port number you receive as ${PORT}.

Configure lighttpd
------------------
Log into your host via SSH, and confirm that lighttpd is installed::

lighttpd -v

Set up a directory structure for lighttpd::

mkdir ~/tmp

mkdir -p ~/var/run

mkdir -p ~/etc/init.d

mkdir -p ~/etc/lighttpd/vhosts.d

touch ~/logs/lighttpd.error.log ~/logs/lighttpd.access.log

Using a text editor, create ~/etc/lighttpd/lighttpd.conf::

#-- Lighttpd modules

server.modules              = ( "mod_rewrite",

                                "mod_redirect",
                                "mod_access",
                                "mod_cgi",
                                "mod_fastcgi",
                                "mod_compress",
                                "mod_accesslog",
                                "mod_alias" )

    
#-- Fundamental process configs

server.port                 = ${PORT}

server.username             = "${USER}"

server.groupname            = server.username

var.base                    = "/users/home/" + server.username

server.pid-file             = base + "/var/run/lighttpd.pid"
    
#-- Logging

server.errorlog             = base + "/logs/lighttpd.error.log"

accesslog.filename          = base + "/logs/lighttpd.access.log"
    
#-- Default

server.document-root        = base + "/web/public"

server.indexfiles           = ( "index.php", "index.html",
                                "index.htm", "default.htm" )
    
#-- Security

url.access-deny             = ( "~", ".inc", ".ht" )

    
#-- Mimetypes

include_shell "cat " + base + "/etc/lighttpd_mimetypes.conf"

    
#-- VHOSTS

Create ~/etc/lighttpd/mimetypes.conf::

mimetype.assign             = (

".pdf"          =>      "application/pdf",

".sig"          =>      "application/pgp-signature",

".spl"          =>      "application/futuresplash",

".class"        =>      "application/octet-stream",

".ps"           =>      "application/postscript",

".torrent"      =>      "application/x-bittorrent",

".dvi"          =>      "application/x-dvi",

".gz"           =>      "application/x-gzip",

".pac"          =>      "application/x-ns-proxy-autoconfig",

".swf"          =>      "application/x-shockwave-flash",

".tar.gz"       =>      "application/x-tgz",

".tgz"          =>      "application/x-tgz",

".tar"          =>      "application/x-tar",

".zip"          =>      "application/zip",

".mp3"          =>      "audio/mpeg",

".m3u"          =>      "audio/x-mpegurl",

".wma"          =>      "audio/x-ms-wma",

".wax"          =>      "audio/x-ms-wax",

".ogg"          =>      "audio/x-wav",

".wav"          =>      "audio/x-wav",

".gif"          =>      "image/gif",

".jpg"          =>      "image/jpeg",

".jpeg"         =>      "image/jpeg",

".png"          =>      "image/png",

".xbm"          =>      "image/x-xbitmap",

".xpm"          =>      "image/x-xpixmap",

".xwd"          =>      "image/x-xwindowdump",

".css"          =>      "text/css",

".html"         =>      "text/html",

".htm"          =>      "text/html",

".js"           =>      "text/javascript",

".asc"          =>      "text/plain",

".c"            =>      "text/plain",

".conf"         =>      "text/plain",

".text"         =>      "text/plain",

".txt"          =>      "text/plain",

".dtd"          =>      "text/xml",

".xml"          =>      "text/xml",

".mpeg"         =>      "video/mpeg",

".mpg"          =>      "video/mpeg",

".mov"          =>      "video/quicktime",

".qt"           =>      "video/quicktime",

".avi"          =>      "video/x-msvideo",

".asf"          =>      "video/x-ms-asf",

".asx"          =>      "video/x-ms-asf",

".wmv"          =>      "video/x-ms-wmv",

".bz2"          =>      "application/x-bzip",

".tbz"          =>      "application/x-bzip-compressed-tar",

".tar.bz2"      =>      "application/x-bzip-compressed-tar"

)

Finally, create an init script at ~/etc/init.d/lighttpd (thanks to `Jared Kuolt`_ )::

#!/bin/sh


    
HOME=/users/home/${USERNAME}

LIGHTTPD_CONF=$HOME/etc/lighttpd/lighttpd.conf

PIDFILE=$HOME/var/run/lighttpd.pid

    
case "$1" in
    start)
    # Starts the lighttpd daemon
    echo "Starting lighttpd"
    PATH=$PATH:/usr/local/bin /usr/local/sbin/lighttpd -f $LIGHTTPD_CONF

;;
    stop)
    # stops the daemon bt cat'ing the pidfile
    echo "Stopping lighttpd"
    kill `/bin/cat $PIDFILE`

;;
    restart)
    ## Stop the service regardless of whether it was
    ## running or not, start it again.
    echo "Restarting lighttpd"
    $0 stop
    $0 start

;;
    reload)
    # reloads the config file by sending HUP
    echo "Reloading config"
    kill -HUP `/bin/cat $PIDFILE`

;;
    *)
    echo "Usage: lighttpd (start|stop|restart|reload)"
    exit 1

;;

esac

Don't forget to make the init script executable::

chmod 755 ~/etc/init.d/lighttpd

Proxy Apache to lighttpd
------------------------
Open up a web browser, and log into https://virtualmin.joyent.us/${HOST}/

Click Webmin; expand "Servers"; click "Apache Webserver".
For your ${DOMAIN} virtual server, click "Aliases and Redirects". Enter these::

Map local to remote URLs

If Apache is serving anything else directly (such as a PHP application), then add that URL to the list, mapped to "None". Make sure the default mapping of / to your lighttpd port appears last on the list.

Basic Django setup
==============
Configure environment
---------------------
.. note:: the Solaris implementation of the bourne shell doesn't support the

::

mkdir ~/bin

mkdir -p ~/src/django_projects

mkdir -p ~/lib/python2.5/site-packages

for dotfile in .bashrc .profile; do echo -e 'PATH=$HOME/bin:$PATH\nexport PATH\nPYTHONPATH=$HOME/src/django_projects:$HOME/lib/python2.5/site-packages\nexport PYTHONPATH' >> ./$dotfile; done

source ~/.bashrc

Verify dependencies
-------------------
The shared accelerators come installed with Python 2.5, Django 1.0, and flup, which is pretty much what you need. This is easy to check::

python -V

ls /usr/local/lib/python2.5/site-packages/

Or more to the point, start a python interpreter and load modules::

import django

django.get_version()

from django.core.handlers.wsgi import WSGIHandler

import flup

from flup.server.fcgi_fork import WSGIServer

[optional] Install Django bash-completion
-----------------------------------------
If you're the type of person who imprudently tinkers with production sites, you might like::

wget http://www.djangoproject.com/download/1.0.1/tarball/ # or whatever latest release

tar xvzf Django-1.0.1-final.tar.gz

mv Django-1.0.1-final/extras/django_bash_completion ~/bin/

for dotfile in .bashrc .bash_profile; do echo ". ~/bin/django_bash_completion" >> ./$dotfile; done

source ~/.bashrc

rm -rf Django-1.0.1-final*

Deploying a Django project
======================
Usually I would check out my project from a Subversion repository, but as an example I'll develop a dead simple project (nothing but flatpages) directly on the server. I'll call it "project" and its database "djangoproject"; obviously these names are arbitrary.

Create database
---------------
Since Joyent's running OpenSolaris, and `Sun has pissed away $797M to support MySQL`_, I figure I might as well stick with the house brand. Postgres setup is very similar, though.

Open up a web browser; log into https://virtualmin.joyent.us/${HOST}/

Select ${DOMAIN} from the dropdown list; click "Edit Databases".
Click "Create New Database".
Database name: "${USER}_djangoproject"; Database server: MySQL; character set: <Default>
Click "Create".

Create database user
--------------------

http://wiki.joyent.com/shared:kb:mysql-users

In Virtualmin, Select ${DOMAIN} from the dropdown list.
Click "Edit FTP Users" > "Add a user to this server".
Input "django" for the "Username" field.
Make a note of the autogenerated password (we'll need it for the Django project's settings.py file).
Expand "Quota and home directory settings". Limit the user's home directory quota to 1MB.
Expand "Other user permissions". Allow the user access to the "${USER}_djangoproject" database we just created. Click create.

The user should be created with a login username something like "django-${DOMAIN}"; make a note of the exact username to enter in settings.py

Upload Django project
---------------------
Ordinarily I'd get the project on the server by checking it out from (e.g.) a Subversion repository, but for the sake of example I'll develop a trivial project (just flatpages) right on the server. I'll call the project "project" and its database "djangoproject"; obviously these names are abitrary.

::

cd ~/src/django_projects

django-admin.py startproject project

cd project

mkdir -p media templates/flatpages

Although normally an uploaded project will come complete with templates and media, there are some design considerations affected by the technical details of how we serve media. We'll skip it for now, and come back to the media and templates belows in the section titled `Offload static media to lighttpd`_.

To turn on Django admin, uncomment the relevant bits in ~/src/django_projects/project/urls.py::

from django.contrib import admin

admin.autodiscover()

urlpatterns += patterns(

        (r'^admin/(.*)', admin.site.root),

)

Configure project settings
--------------------------

.. note:: Since we're using lighttpd + fastcgi, apparently we have to include a FORCE_SCRIPT_NAME directive in settings.py. This seems to be not well documented; q.v. http://tinyurl.com/5sf4vq. What works for me is to assign this as the empty string (rather than the default None).

As discussed in `Rob Hudson's blog`_ I prefer to use relative paths in my settings file. Use a text editor to create ~/src/django_projects/project/settings.py.::

FORCE_SCRIPT_NAME=''


    
import os.path

ROOT_DIR = os.path.abspath(os.path.dirname(file))

    
DATABASE_ENGINE = 'mysql'

DATABASE_NAME = '${USER}_djangoproject'

DATABASE_USER = '${USER}'

DATABASE_PASSWORD = 'password"

DATABASE_HOST = ''

DATABASE_PORT = ''

    
MEDIA_ROOT = os.path.join(ROOT_DIR, 'media')

MEDIA_URL = '/media/'

ADMIN_MEDIA_PREFIX = '/media/admin/'

    
TEMPLATE_DIRS = (
    os.path.join(ROOT_DIR, 'templates'),

)

    
INSTALLED_APPS = (
    'django.contrib.sites',
    'django.contrib.admin',
    'django.contrib.flatpages',

)

Since the settings file has our MySQL password inside, don't let others read it::

chmod 600 ~/src/django_projects/project/settings.py

Then create the database tables in the usual fashion::

python manage.py syncdb

Create project init script
-------------------------
Again thanks to `Jared Kuolt`_ . Create ~/src/djangoprojects/project/init.sh::

#!/bin/sh

HOME="/users/home/${USER}" # Edit to your own username

PYTHONPATH=$HOME/lib/python2.5/site-packages:$HOME/src/django_projects

export PYTHONPATH


    
PROJECT_NAME="project"

PROJECT_DIR="$HOME/src/django_projects/$PROJECT_NAME"

PID_FILE="$HOME/var/run/$PROJECT_NAME.pid"

SOCKET_FILE="$HOME/tmp/$PROJECT_NAME.socket"

MANAGE_FILE="$PROJECT_DIR/manage.py"

METHOD="prefork"

    
case "$1" in
    start)
    # Starts the Django process
    echo "Starting Django project $PROJECT_NAME"
    python $MANAGE_FILE runfcgi maxchildren=2 maxspare=2 minspare=1 method=$METHOD socket=$SOCKET_FILE pidfile=$PID_FILE

;;
    stop)
    # stops the daemon by cat'ing the pidfile
    echo "Stopping Django project $PROJECT_NAME"
    kill `/bin/cat $PID_FILE`

;;
    restart)
    ## Stop the service regardless of whether it was
    ## running or not, start it again.
    echo "Restarting Django project $PROJECT_NAME"
    $0 stop
    $0 start

;;
    *)
    echo "Usage: init.sh (start|stop|restart)"
    exit 1

;;

esac

And of course make the init script executable::

chmod 755 ~/src/djangoprojects/project/init.sh

Offload static media to lighttpd
--------------------------------
.. note:: In this example, we'll use the default domain of our Joyent account as the lighttpd document root; obviously we'd want to change the file paths if we were using a subdomain.

We want to have lighttpd directly serve static files (media) without going through FastCGI. These are served from below the lighttpd document root (e.g. ~/web/public/), rather than from the django project dir. However, we don't want our media to actually live in the lighttpd document root, because that makes for messy and/or duplicative source management (e.g. Subversion checkouts), so we use symlinks in the filesystem.

For example, take the Django admin media, which is installed in the host's global Python site-packages. A little bit later on, we'll configure lighttpd to serve directly any URLs beginning with settings.ADMIN_MEDIA_PREFIX instead of doing a proxy handover to Django. So all we need to do here is to symlink the admin media directory to a subdirectory named settings.ADMIN_MEDIA_PREFIX below the lighttpd document root::

mkdir ~/web/public/media

ln -s /usr/local/lib/python2.5/site-packages/django/contrib/admin/media/ ~/web/public/media/admin

Taking this a step further, we want to have our own media live inside the Django project folder, yet be served from the lighttpd document root, so we'll need another symlink::

mkdir -p ~/src/django_projects/project/media/public

ln -s /users/home/${USER}/src/django_projects/project/media/public ~/web/public/media/public

We use the media/public subdirectory because this lighttpd setup offers no access control; any media that requires authentication will still be served by Django from other subdirectories.

As an example of public media to be served by lighttpd, we'll add a CSS stylesheet::

mkdir -p ~/src/django_projects/project/media/public/css

svn checkout http://css-boilerplate.googlecode.com/svn/trunk/boilerplate ~/src/django_projects/project/media/public/css

We'll create a base template referencing the stylesheet at ~/src/django_projects/project/templates/base.html::

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN">

<html lang="en">

<head>

    <title>{% block title %}{% endblock %}</title>
    <link rel="stylesheet" type="text/css" href="/media/public/css/screen.css" media="all" />

</head>

<body>
    {% block content %}{% endblock %}

</body>

</html>

We'll extend the base template for our default flatpage template by creating ~/src/djangoprojects/project/templates/flatpages/default.html::

{% extends "base.html" %}

{% block title %}

{{ flatpage.title }}

{% endblock %}

    
{% block content %}

<div id=flatpage>
    {{ flatpage.content}}

</div>

{% endblock %}

Configure lighttpd
==============
.. note:: In this example, we'll use the default domain of our Joyent account as the lighttpd document root; obviously we'd want to change that if we were using a subdomain.

For the machinery of configuring the fastcgi socket, again please note that the socket name "project" is purely arbitrary.

For the media, we want to instruct lighttpd to serve Django's admin media (i.e. URLs beginning with ADMIN_MEDIA_PREFIX) directly from the host's global Python site-packages where we have it installed. Other offloaded static media (i.e. URLs beginning with /media/public) should be and to directly With your favorite text editor create ~/etc/lighttpd/vhosts.d/project.conf::

$HTTP["host"] =~ "(www\.)?${DOMAIN}" {

    var.domain = base
    server.document-root = domain + "/web/public"
    fastcgi.server = (
        "/project.fcgi" => (
            "main" => (
                "socket" => base + "/tmp/project.socket",
                "bin-environment" =>
                            ( "TZ" => "America/Chicago" ),
                "check-local" => "disable",
            )
        ),
    )

    url.rewrite-once = (
        "^(/media/admin.*)$" => "$1",
        "^(/media/public.*)$" => "$1",
        "^/favicon.ico$" => "/media/public/img/favicon.ico",
        "^(/.*)$" => "/project.fcgi$1",
    )

}

Then include these configs in your lighttpd.conf::

echo 'include "vhosts.d/project.conf"' >> ~/etc/lighttpd/lighttpd.conf

...whoops, looks like this is too long; continued next post...

.. _`Django deployment docs`: http://docs.djangoproject.com/en/dev/ho … ttpd-setup
.. _`support ticket`: http://help.joyent.com/index.php?pg=request
.. _`Jared Kuolt`_: http://superjared.com/static/code/textd … install.py
.. _`Rob Hudson's blog`: http://rob.cogit8.org/blog/2008/Jun/20/ … elativity/
.. _`Sun has pissed away $797M to support MySQL`: http://blogs.sun.com/jonathan/entry/win … re_blowing
.. _`Joyent's boot-up actions wiki page`: http://wiki.joyent.com/shared:kb:bootup-actions

Last edited by csingley (2009-01-02 15:58:50)

I just moved an app to a new server, and I'm running into the following (that I haven't seen before)

the app is login only, so when you try and visit any page while not logged it, it'll redirect you to /login/?next=<the page you came from> on my new server I get the following:

http://1466westwarner.com/login/?next=/1466westwarner.fcgi/

instead of my expected

http://1466westwarner.com/login/?next=/

any thoughts as to what might be causing this / how to fix it?

of note, the only thing I did other than move servers is update to django trunk from something pre-1.0..

Last edited by drackett (2009-06-17 23:18:41)