LAN-Spiegel-Server für GIT Repositories

Mirror an external GIT repository in the local network

In diesem Artikel geht es darum, mit GIT größere (externe) Repositories mithilfe eines LAN Servers zwischenzuspeichern, um Zeit und Bandbreite zu sparen. Die Installtion von GIT bringt dabei alle wichtigen Tools bereits mit sich. Der Server soll in einem gewissen Zeitintervall die Änderungen vom Original-Repository ziehen und durch das git://-Protokoll ohne Login oder SSH-Keys im LAN zur Verfügung stellen. Wichtig zu erwähnen ist, dass dies auch in gitolite bereits gelöst ist - mehr noch, auch "push-Spiegeln" ist möglich, bei dem der Server seine Änderungen auf Slave-Server weiterverteilt. Hier geht es mehr darum, z.B. kernel.org Repos schnell aus dem LAN laden zu können.

Im Kern ist die Sache bereits durch Automation von "git daemon", "git clone --mirror" und "git fetch" gelöst. Ein paar Details gelten der Sicherheit, Einfachheit und Übersicht:

• Die mirror Repositories sollten von den internen gitolite/gitosis Repositories getrennt sein - am Besten ein eigenes home-Verzeichnis.

• Der git-daemon (GIT-Server im LAN) sollte als eingeschränkter Nutzer laufen und nur Lesezugriff auf die gespiegelten Repos haben.

• Ein AppArmor-Profil für weitere Sicherheit

Der Ansatz: Struktur ein Wenig wie gitolite, d.h ein Daemon-Nutzer (ich starte git-daemon nicht via inetd) mit home, die Schreibrechte im home liegen jedoch bei root und einem Account, der die Repositories updaten ("clonen", "fetchen") darf. Darin ein Verzeichnis für Skripte und Binaries, sowie eines für die Repositories.

Der Daemon-Nutzer sei git-mirror, home /home/git-mirror, und git-mirror ist der Nutzer, der "clonen" und "fetchen" darf. Hier sind die Nutzer als "normale" Nutzer angelegt, --system tut auch. git-writer kann auch weggelassen werden und git-mirror oder der eigene Account angegeben werden - "your choice".

This article is about mirroring/caching bigger GIT remote read-only repositories and making them available via the git:// protocol in the local network, this saving bandwith and time. The GIT installation provides pretty much all required tools to accomplish this, only some administrative details have to be taken care of. Note that gitolite has also these capabilities, and also "push-mirroring" from a master server to slave server is possible. However, the purpose described here is to pull remote repositories, like from kernel.org.

Basically, the task is done by automating git-daemon, git clone --mirror and git fetch with the following aspects:

• The mirrored repositories are separated form the gitolite home,

• git-daemon runs as restricted user without write access to the mirrored repos.

• apparmor profiling for additional salt.

Basic approach: Make it a bit "gitolite-like", means creating a daemon user with its home in /home, restricting the write operations for this user - basically almost everywhere, event in its own home directory. Adding a "write-user" for cloning or fetching - or defining an existing one to to this (e.g. your own user account):

$ su
# apt-get install git
# adduser --quiet --disabled-login --shell /bin/false \
#         --home /home/git-mirror --no-create-home git-mirror
# mkdir /home/git-mirror
# mkdir /home/git-mirror/bin
# mkdir /home/git-mirror/repositories
# chmod 755 /home/git-mirror/repositories
# chown $WRITER:$WRITER /home/git-mirror/repositories
#

Dann brauchen wir noch ein Skript, um das Handling ein Wenig zu erleichtern ("das, meine Damen und Herren, habe ich schon mal für Sie vorbereit ..."): git-mirror. Dieses kommt in /home/git-mirror/bin und sollte root gehören. Um das Chaos komplett zu machen habe ich dieses Skript auch git-mirror genannt. Angenommen es liegt jetzt in /tmp/:

Then a script is needed to simplify the handling. I prepared one here: git-mirror. Copy it to /home/git-mirror/bin, and assign it to root. Assuming you downloaded it into /tmp/:

# mv /tmp/git-mirror /home/git-mirror/bin/git-mirror
# chmod 755 /home/git-mirror/bin/git-mirror
# chown root:root /home/git-mirror/bin/git-mirror
# vi/nano /home/git-mirror/bin/git-mirror
  --> "# MIRROR_OWNER=<user name>" ---> "MIRROR_OWNER=$WRITER" --> save,quit

Wie immer - bitte das Skript vor der Anwendung schütteln und LESEN.

Das war's eigentlich schon. Die Firewall muss noch ein Loch am Port 9418 (der Port für git-daemon) haben. Für ufw und 192.168.0.{1-253} IPv4-Standard-LAN wäre das:

And as always - READ IT before blindly using it. Almost done. Punch a hole in the firewall at port 9418 (git protocol), e.g. when using ufw and a simple ole IPv4 LAN:

# ufw allow in from 192.168.0.0/24 to $MY_IP proto tcp port 9418
# ufw allow out from $MY_IP to any proto tcp port 9418

Das Skript genügt den Konventionen für sysv/initd, d.h. Die Argumente start, stop, restart, status, reload werden beachtet. Weiterhin sind die Kommandos clone und fetch implementiert. Sie legen ein neues Repository von einer gegebenen Quelle im /home/git-mirror/repositories-Verzeichnis an bzw. updaten alle darin befindlichen Repositories. Falls diese Kommandos als root ausgeführt werden, so wird erst der Nutzer zum "Schreib-User" gewechselt (drop permissions).

The script follows the conventions of sysv, t.m. it respects the command arguments start, stop, restart, status, reload. Additionally, the commands clone and fetch are implemented. When executing the script as root, permissions are dropped before cloning or fetching, so that the configured write-user is active. As shown you can modify this in the script config header, the variable is $MIRROR_OWNER. When git-mirror start is executed NOT as root, the script will run as the current user and print a warning. So, in short the usage is:

$ su
# cd /home/git-mirror/bin
# ./git-mirror clone git://somehost/somerepo.git somerepo.git
# ./git-mirror fetch
# ./git-mirror start
#
# exit
$
$ cd /tmp/
$ git clone git://localhost/somerepo.git
$

Nun kann, wer will, git-mirror noch in den PATH legen (ich habe ein Verzeichnis ~/.bin im Suchpfad liegen, /usr/local/bin tut auch).

You can also link the script into the exec PATH:

sudo ln -s /home/git-mirror/bin/git-mirror /usr/local/bin/git-mirror

Dann müssen noch, z.B. einmal täglich, alle Repos aktualisiert werden:

Then we automate the (e.g. daily) repository update via crond:

$ su
# echo -e '#!/bin/sh\n/home/git-mirror/bin/git-mirror fetch >/dev/null 2>&1\n' \
           >/etc/cron.daily/git-mirror-update
# chmod 755 /etc/cron.daily/git-mirror-update

Shellscript-Datei

The shell script

git-mirror

Scriptquelltext

Script source code

#!/bin/bash
 
# ----------------------------------------------------------------------
# config
# ----------------------------------------------------------------------
#
# See help section below or --help for config details
#
# MIRROR_USER=<daemon user name>   optional, e.g. git-mirror
# MIRROR_GROUP=<daemon group name> optional, e.g. git-mirror
# PID_FILE=<file>                  default: /tmp/git-mirror.pid
# MIRROR_OWNER=<user name>         default: $MIRROR_USER
 
# ----------------------------------------------------------------------
# auxiliary functions / globals
# ----------------------------------------------------------------------
 
function eecho() {
  echo -e "$@" >&2
}
 
function exit_script() {
  cd "$PWD" >/dev/null 2>&1
  if test "$1" = ""; then
    exit 0
  else
    exit $1
  fi
}
 
function die() {
  eecho $@
  exit_script 1
}
 
function git_bin() {
  local GIT_BIN=$(/usr/bin/which git)
  if test "$GIT_BIN" = ""; then
    if test -x /usr/bin/git; then
      GIT_BIN=/usr/bin/git
    else
      eecho "git is not installed"
      exit_script 1
    fi
  fi
  echo $GIT_BIN
  return 0
}
 
# ----------------------------------------------------------------------
# daemon / service functions
# ----------------------------------------------------------------------
 
function daemon_pid() {
  echo $(/bin/ps -C git-daemon --User "$MIRROR_USER" --no-headers --format "%p")
}
 
function daemon_pid_from_file() {
  echo $(/bin/cat "$PID_FILE" 2>/dev/null)
}
 
function daemon_status() {
  if test "$(daemon_pid)" = ""; then
    echo "stopped"
    return 1
  else
    echo "running"
    return 0
  fi
}
 
function daemon_start() {
  if test "$(daemon_status)" = "running"; then
    echo "already running"
    return 0
  else
    local GITD_USER=""
    local GITD_GROUP=""
    if test "$(/usr/bin/whoami)" != "root"; then
      echo "notice: Not started as root, means git-daemon will not run "
      echo "        with the configured user/group, but with the current"
      echo "        user/group (user=$(/usr/bin/whoami))"
    else
      if test "$MIRROR_USER" = ""; then
        echo "error: No daemon user specified, and refusing to run the daemon as root"
        return 1
      fi
      GITD_USER="--user=$MIRROR_USER"
      if test "$GITD_GROUP" != ""; then
        GITD_GROUP="--group=$MIRROR_GROUP"
      fi
    fi
 
    $(git_bin) daemon \
      --informative-errors \
      --reuseaddr \
      --detach \
      --export-all \
      --init-timeout=3 \
      --port=9418 \
        $GITD_USER \
        $GITD_GROUP \
      --disable=upload-archive \
      --disable=receive-pack \
      --enable=upload-pack \
      --forbid-override=upload-archive \
      --forbid-override=receive-pack \
      --forbid-override=upload-pack \
      --pid-file="$PID_FILE" \
      --base-path="$REPOSITORIES/" \
      "$REPOSITORIES/" \
    2>&1
 
    if test $? -eq 0; then
      if test "$(daemon_pid)" = "$(daemon_pid_from_file)"; then
        if test "$1" != "-q"; then
          echo "started"
        fi
        return 0
      else
        echo "warning: Determined PID differs from the created PID file "
        echo "         content (started, but something is suspiceous)"
        return 0
      fi
    else
      echo "error: Failed to start git daemon, output is: \"$GIT_OUT\""
      return 1
    fi
  fi
}
 
function daemon_stop() {
  local DPID=$(daemon_pid)
  if test "$(daemon_status)" != "running" -o "$DPID" = ""; then
    echo "not running"
  elif test "$DPID" != "$(daemon_pid_from_file)"; then
    echo "error: Determined PID does not match the PID file contents of git-daemon"
  elif test "$(kill $DPID >/dev/null 2>&1; echo $?)" != "0"; then
    if test "$(/usr/bin/whoami)" != "root"; then
      echo "error: You are lacking the permissions to stop the daemon"
    else
      echo "error: Could not kill the git-daemon process (PID=$DPID)"
    fi
  else
    # stfwi: dblchk gnu corutils float sleep compatability
    sleep 0.25
    test "$(daemon_status)" != "running" || sleep 4
    if test "$(daemon_status)" = "running"; then
      echo "error: Git daemon still running after killing the process (PID=\"$DPID\")"
    else
      if test "$1" != "-q"; then
        echo "stopped"
      fi
      return 0
    fi
  fi
  return 1
}
 
function daemon_restart() {
  if test "$(daemon_pid)" = ""; then
    echo "not running"
    return 1
  else
    daemon_stop -q
    if test $? -ne 0; then
      echo "error: Failed to restart"
    else
      daemon_start -q
      if test $? -eq 0; then
        if test "$1" != "-q"; then
          echo "restarted"
        fi
        return 0
      else
        daemon_stop -q >/dev/null 2>&1
        echo "error: Failed to restart"
        return 1
      fi
    fi
  fi
}
 
function daemon_user_check() {
  if test "$(/usr/bin/whoami)" != "root"; then
    die "You are not root (git will not be able to switch the \
          user to \"$MIRROR_USER\")"
  fi
}
 
function repository_user_check() {
  if test "$(/usr/bin/whoami)" = "root"; then
    eecho "(dropping root permissions, user=\"$MIRROR_OWNER\")"
    if test -x /usr/bin/sudo; then
      /usr/bin/sudo -n -u $MIRROR_OWNER -- /bin/true
      if test $? -ne 0; then
        die "error: Could not switch user to \"$MIRROR_OWNER\""
      else
        /usr/bin/sudo -n -u $MIRROR_OWNER -- $0 $@
        exit_script $?
      fi
    else
      die "Need the 'sudo' command to drop permissions (switch to user \"$MIRROR_OWNER\")"
    fi
  elif test "$(/usr/bin/whoami)" != "$MIRROR_OWNER"; then
    eecho "error: The user operating on repositories does not match the configured repository owner, aborting."
    eecho "       \"$(/usr/bin/whoami)\" != \"$MIRROR_OWNER\""
  else
    return 0
  fi
  exit_script 1
}
 
# ----------------------------------------------------------------------
# repository functions
# ----------------------------------------------------------------------
 
function repository_clone() {
  if test "$(/usr/bin/whoami)" = "root"; then
    die "error: !!! Still root after user check"
  elif test "$1" = "" -o "$2" = "" -o "$3" != ""; then
    die "Need a remote to clone (arg1) and the local repository name (arg2)"
  elif test ! -d "$REPOSITORIES"; then
    die "Missing repository base directory (\"$REPOSITORIES\")"
  elif test "$(basename $2)" != "$2"; then
    die "Your local mirror repository name contains slashes, not ok"
  elif test -d "$REPOSITORIES/$2"; then
    die "The local mirror repository directory exists already (\"$REPOSITORIES/$2\")"
  else
    case "$2" in
      *.git)
      ;;
    *)
      die "You local mirror repository name must end with \".git\""
      ;;
    esac
    echo -en "cloning \"$(basename $1)\" ... "
    local CPWD=$(pwd)
    cd "$REPOSITORIES" || die "Failed to switch to repository base directory"
    $(git_bin) clone --mirror "$1" "$2"
    local RES=$?
    cd "$CPWD"
    if test $RES -eq 0; then
      return 0
    fi
  fi
  return 1
}
 
function repository_update() {
  if test "$(/usr/bin/whoami)" = "root"; then
    die "error: !!! Still root after user check"
  elif test ! -d "$1"; then
    echo "update error: \"$1\" is not a directory"
  else
    echo -en "fetching \"$(basename $1)\" ... "
    local CPWD=$(pwd)
    cd "$1"
    $(git_bin) fetch --all >/dev/null
    local RES=$?
    cd "$CPWD"
    if test $RES -eq 0; then
      echo "ok"
      return 0
    else
      echo "failed"
    fi
  fi
  return 1
}
 
function repositories_update() {
  for I in $REPOSITORIES/*.git; do
    if test -d "$I"; then
      repository_update "$I"
    fi
  done
  echo "update done"
  return 0
}
 
# ----------------------------------------------------------------------
# help
# ----------------------------------------------------------------------
 
function script_help() {
 
cat <<'EOT'
 
NAME
 
    git-mirror
 
SYNOPSIS
 
    git-mirror [ start | stop | restart | reload | status ]
 
    git-mirror [ clone | fetch ]
 
    git-mirror [ help | --help | -h ]
 
SERVICE OPTIONS / COMMANDS
 
    help, --help, -h  Show this help
 
    start             Start git-daemon in background
 
                        If executed as root, the script will force git
                        to drop the permissions by switching to the
                        user/group specified in the script configuration.
 
                        If executed as non-root the deamon will run as
                        the current user/group.
 
    stop              Stop git-daemon
 
                        If the current user lacks if the required
                        permissions, an error will be raised.
 
    restart, reload   Restart git-daemon
 
                        Identical to start && stop
 
    status            Retrieve the status of the background daemon
 
                        Prints "stopped" or "running" according to
                        the daemon process state.
 
REPOSITORY OPTIONS / COMMANDS
 
    clone <rem> <loc> Clones a remote repository using the --mirror
                      option. The owner of the repositories is set
                      in the script configuration, and should be
                      different to the daemon user for the sake of
                      saftey and security. If executed as root, the
                      script drops permissions by switching to the
                      configured owner. Otherwise, if the current user
                      is not the configured owner, an error will be
                      raised.
 
                      <rem> is the remote repository to mirror,
                      <loc> is the local repository name.
 
                      Note that the local repository must end with
                      ".git" and must not contain slashes. The script
                      throws an error if these requirements are not
                      satisfied.
 
    fetch,update,pull Update all bare mirrors from their remote. If
                      executed as root, the permissions are dropped
                      as with `clone`, otherwise the current user must
                      match the configured owner.
 
DEPENDENCIES
 
    git, sudo
 
EXAMPLE SETUP
 
    This setup sequence initialises git-mirror (simply). Note that
    you should modify `chown git-mirror:git-mirror` to another user,
    so that git-mirror has no write access when running the git-daemon.
    For that, you also need to change `MIRROR_OWNER=<other user>` in
    the config section of this script.
 
    Assuming this script is currently located somewhere in your home,
    and you are in the parent directiory (script is "./git-mirror").
 
    $ su
    # apt-get install git
    # adduser --quiet --disabled-login --shell /bin/false \
    #         --home /home/git-mirror --no-create-home git-mirror
    # mkdir /home/git-mirror
    # mkdir /home/git-mirror/bin
    # mkdir /home/git-mirror/repositories
    # cp ./git-mirror /home/git-mirror/bin/
    # chmod 755 /home/git-mirror/bin/git-mirror
    # chown root:root /home/git-mirror/bin/git-mirror
    # chmod 755 /home/git-mirror/repositories
    # chown git-mirror:git-mirror /home/git-mirror/repositories
    #
    # cd /home/git-mirror/bin
    # ./git-mirror clone git://somehost/somerepo.git somerepo.git
    # ./git-mirror fetch
    # ./git-mirror start
    #
    # exit
    $
    $ cd /tmp/
    $ git clone git://localhost/somerepo.git
    $
 
    The script is suitable to be symlinked in /etc/rc?.d (/etc/init.d,
    respectively) or to be wrapped as service with minor effort.
 
    `/home/git-mirror/bin/git-mirror fetch` can also be used in a
    cron job:
 
    $ su
    # echo -e '#!/bin/sh\n/home/git-mirror/bin/git-mirror fetch \
              >/dev/null 2>&1\n' >/etc/cron.daily/git-mirror-update
    # chmod 755 /etc/cron.daily/git-mirror-update
 
EOT
 
  return 1
}
 
 
# ----------------------------------------------------------------------
# main
# ----------------------------------------------------------------------
 
PWD=$(pwd)
THIS_SCRIPT=$(readlink -f "$0")
BASE_DIR=$(dirname $(dirname $THIS_SCRIPT))
 
if test "$REPOSITORIES" = ""; then
  REPOSITORIES="$BASE_DIR/repositories"
fi
 
if test "$PID_FILE" = ""; then
  PID_FILE=/tmp/git-mirror.pid
fi
 
if test "$MIRROR_USER" = ""; then
  if test "$(dirname $BASE_DIR)" = "/home"; then
    MIRROR_USER=$(basename $BASE_DIR)
  else
    die "No MIRROR_USER configured"
  fi
fi
 
if test "$MIRROR_OWNER" = ""; then
  MIRROR_OWNER=$MIRROR_USER
fi
 
if test ! -d "$REPOSITORIES"; then
  die "error: Repository directory does not exist or not configured."
fi
 
case $1 in
  start)
    daemon_start
    exit_script $?
    ;;
  stop)
    daemon_stop
    exit_script $?
    ;;
  restart|reload)
    daemon_restart
    exit_script $?
    ;;
  status)
    daemon_status
    exit_script $?
    ;;
  clone)
    repository_user_check $@
    repository_clone "$2" "$3" "$4"
    exit_script $?
    ;;
  update|pull|fetch)
    repository_user_check $@
    repositories_update "$2"
    exit_script $?
    ;;
  help|-h|--help)
    script_help
    exit_script $?
    ;;
  *)
    echo "Unknown command: \"$1\""
    echo "- Server control usage: git-mirror [ start | stop | restart | status ]"
    echo "- Repository updates  : git-mirror [ pull | fetch | update ]"
    echo "- Repository init     : git-mirror clone <remote git repository>"
    exit 1
    ;;
esac
die "Unexpected script run until EOF (bug)"