it-artikel:linux:how-to-make-an-ubuntu-or-debian-apt-repository-locally-available-as-an-emergency-offline-fallback

How to make an Ubuntu or Debian apt repository locally available as an emergency offline fallback

Maybe you never had that problem yourself, but there may be reasons why someone want to have one or multiple Ubuntu or Debian online repositories available on your local network. Maybe because you throw an Linux installation party and dozens of times the same repos and files are been pulled over a thin DSL connection. Or you want your own local mirror in case your internet connection goes down. Or you might just want to keep a certain version or snapshot of the online repo for a specific UBUNTU release or distro, because one day it may become unavailable, if UBUNTU/DEBIAN decides to take it down if its “too old” for them. This might be interesting for some “retro computing” guys as well.

So how to achieve that ?

As always, there is more than just one solution. However… this is how i did it.

Pre-Requirements:

  1. Define what distro releases and what “repositories” or “sections” you want to have available on your LAN. With “repositories” or “sections” i mean those “keywords” that follow one of your “deb” lines in your /etc/apt/sources.list. In my example i want to have Ubuntu 16.04 LTS, Ubuntu 18.04 LTS and Ubuntu 19.04 available at all time. So ill need the code names for its respositories/sections later. The format is as shown here:
    #deb url distroReleaseName repo/sections
    deb http://archive.ubuntu.com/ubuntu/ disco main restricted universe multiverse
  2. Take a note what official Ubuntu/Debian mirror server your want to use for your download. For me it will be http://de.archive.ubuntu.com/ubuntu/
    • :!: BEWARE: This manual assumes the use of only ONE official mirror server, for all distros or releases we want! This is somewhat important, since later we only want to make one single directory available to our network clients. The apt-mirror tool however creates one directory per official mirror, which makes it more complex and is not shown here. So avoid the use of several different mirror servers and only use a single one if possible.
  3. Make sure your have PLENTY of disk space available for each distro release and repositories you select. Depending on your choices it can easily exceed 500GB total as in my example here.

Installation and configuration:

  1. :!::!::!: BUG: It seems that the apt-mirror package that comes with UBUNTU repositories up to recently 1/2020 contain a major BUG, that does'nt do a proper mirroring job! This results in local mirrors that will cause 404 errors about not finding “cnf” and other files when doing a apt update on your client side. In addition, for some reason the original apt-mirror package/source at the moment seems to be orphaned and people are about to fork and reorganize this open source project. So we are NOT USING THE UBUNTU VERSION OF APT-MIRROR, Instead we use some fork on GitHub. So the risk of installing software from unknown or untrusted sources is on you!Axel Werner 2020-01-24 15:08
  2. Install the special apt-mirror fork from https://github.com/josbraden/apt-mirror by:
    cd /opt/
    git clone https://github.com/josbraden/apt-mirror
    ln -vs /opt/apt-mirror/apt-mirror /usr/bin/apt-mirror
  3. Prepare your download/mirror directory:
    mkdir -p /pentyOfSpaceHere/apt-mirror
    chown -c apt-mirror.users /pentyOfSpaceHere/apt-mirror
    chmod -c u+rwx,g+rwxs /pentyOfSpaceHere/apt-mirror
  4. Make your own version of /etc/apt/mirror.list and put the gathered information in:
    cp /etc/apt/mirror.list /etc/apt/mirror.list.$(date +%F)
    
    cat << 'EOF' > /etc/apt/mirror.list
    ############# config ##################
    #
    set base_path    /pentyOfSpaceHere/apt-mirror
    #
    # set mirror_path  $base_path/mirror
    # set skel_path    $base_path/skel/etc/apt/sources.list.d/localRepo-ubuntu-2004-focal.list
    # set var_path     $base_path/var
    set cleanscript $var_path/clean.sh
    # set defaultarch  <running host architecture>
    # set postmirror_script $var_path/postmirror.sh
    # set run_postmirror 0
    set nthreads     10
    set _tilde 0
    #
    ############# end config ##############
    
    ##################################
    ##
    ##  UBUNTU 16.04 LTS (xenial)
    ##
    ##################################
    
    deb http://archive.ubuntu.com/ubuntu xenial main restricted universe multiverse 
    deb http://archive.ubuntu.com/ubuntu xenial-updates main restricted universe multiverse 
    deb http://archive.ubuntu.com/ubuntu xenial-backports main restricted universe multiverse 
    deb http://archive.ubuntu.com/ubuntu xenial-security main restricted universe multiverse 
    deb http://archive.canonical.com/ubuntu xenial partner
    
    deb-i386 http://archive.ubuntu.com/ubuntu xenial main restricted universe multiverse 
    deb-i386 http://archive.ubuntu.com/ubuntu xenial-updates main restricted universe multiverse 
    deb-i386 http://archive.ubuntu.com/ubuntu xenial-backports main restricted universe multiverse 
    deb-i386 http://archive.ubuntu.com/ubuntu xenial-security main restricted universe multiverse 
    deb-i386 http://archive.canonical.com/ubuntu xenial partner
    
    ##################################
    ##
    ##  UBUNTU 18.04 LTS (bionic)
    ##
    ##################################
    
    deb http://archive.ubuntu.com/ubuntu bionic main restricted universe multiverse 
    deb http://archive.ubuntu.com/ubuntu bionic-updates main restricted universe multiverse 
    deb http://archive.ubuntu.com/ubuntu bionic-backports main restricted universe multiverse 
    deb http://archive.ubuntu.com/ubuntu bionic-security main restricted universe multiverse 
    deb http://archive.canonical.com/ubuntu bionic partner
    
    deb-i386 http://archive.ubuntu.com/ubuntu bionic main restricted universe multiverse 
    deb-i386 http://archive.ubuntu.com/ubuntu bionic-updates main restricted universe multiverse 
    deb-i386 http://archive.ubuntu.com/ubuntu bionic-backports main restricted universe multiverse 
    deb-i386 http://archive.ubuntu.com/ubuntu bionic-security main restricted universe multiverse 
    deb-i386 http://archive.canonical.com/ubuntu bionic partner
    
    
    ##################################
    ##
    ##  UBUNTU 20.04 LTS (focal)
    ##
    ##################################
    
    deb http://archive.ubuntu.com/ubuntu focal main restricted universe multiverse 
    deb http://archive.ubuntu.com/ubuntu focal-updates main restricted universe multiverse 
    deb http://archive.ubuntu.com/ubuntu focal-backports main restricted universe multiverse 
    deb http://archive.ubuntu.com/ubuntu focal-security main restricted universe multiverse 
    deb http://archive.canonical.com/ubuntu focal partner
    
    deb-i386 http://archive.ubuntu.com/ubuntu focal main restricted universe multiverse 
    deb-i386 http://archive.ubuntu.com/ubuntu focal-updates main restricted universe multiverse 
    deb-i386 http://archive.ubuntu.com/ubuntu focal-backports main restricted universe multiverse 
    deb-i386 http://archive.ubuntu.com/ubuntu focal-security main restricted universe multiverse 
    deb-i386 http://archive.canonical.com/ubuntu focal partner
    
    
    
    
    ##################################
    ##
    ##  GLOBAL / CLEANUP
    ##
    ##################################
    
    clean http://archive.ubuntu.com/ubuntu
    clean http://archive.canonical.com/ubuntu
    
    
    
    
    EOF
  5. Apt-mirror does not come with a postmirror.sh script, even though its called by default on every mirroring. So to prevent the error message, we create our own script:
    cat << EOF > /pentyOfSpaceHere/apt-mirror/var/postmirror.sh
    #!/bin/bash -e
    
    ## Anything in this file gets run AFTER the mirror has been run.
    ## Put your custom post mirror operations in here (like rsyncing the installer
    ## files and running clean.sh automatically)!
    
    EOF
  6. For automatic updating your local mirrors, use crontab. In my case, i want to update my local mirrors once a day at 1am:
    cat << EOF > /etc/cron.d/apt-mirror
    #
    # Regular cron jobs for the apt-mirror package
    #
    #m	h	dom	mon	dow	user		command
    0	1	*	*	*	apt-mirror	/usr/bin/apt-mirror 2>&1 | tee -a /var/log/apt-mirror-cron.log
    
    EOF
  7. :!: WARNING: The first run of apt-mirror will download several hundred GB of files and will load down your internet connection for hours or days! Make sure this is no problem for you.
  8. Start the first run/download from the official mirrors with:
    # this will run apt-mirror as service user 'apt-mirror'
    # use CTRL+C to abort download if needed. Else let it do its job and 
    # check back 24h later.
    #
    su - -c /usr/bin/apt-mirror apt-mirror
  9. If the download is done, your local repo has been created and should be ready at /pentyOfSpaceHere/apt-mirror/mirror/yourOfficialMirrorChosen/ubuntu/ on your servers filesystem.
  10. To access it from your local network by clients you'll need some sort of webserver or ftp server. Its your choice. Just make sure, it can access (read only) your mirrored files using its service account. This manual assumes we use Apache and use its Ubuntu default webserver directory:
    # make symbolic link for Apache to reach the mirror dirs.
    ln -s /pentyOfSpaceHere/apt-mirror/mirror/yourOfficialMirrorChosen/ubuntu /var/www/ubuntu
    ln -s /pentyOfSpaceHere/apt-mirror/mirror/archive.canonical.com/ubuntu /var/www/ubuntuPartner
  11. In my example my Ubuntu mirror should be available to my network clients under http://myServer/ubuntu/ and can now be used on Ubuntu (or debian) Hosts in their /etc/apt/sources.list like this:
    # Example for UBUNTU 19.04 DISCO
    deb http://myLocalServer/ubuntu/ disco main restricted universe multiverse
    deb http://myLocalServer/ubuntu/ disco-updates main restricted universe multiverse
    deb http://myLocalServer/ubuntu/ disco-security main restricted universe multiverse
    deb http://myLocalServer/ubuntu/ disco-backports main restricted universe multiverse
    
    deb http://myLocalServer/ubuntuPartner/ disco partner
    
    

    OR you can just drop an additional file into /etc/apt/sources.list.d/ with the additional apt repositories you want to use:

    cat << EOF > /etc/apt/sources.list.d/localRepo-ubuntu-2004-focal.list
    
    # Example for UBUNTU 20.04 FOCAL
    deb http://myLocalServer/ubuntu/ focal main restricted universe multiverse
    deb http://myLocalServer/ubuntu/ focal-updates main restricted universe multiverse
    deb http://myLocalServer/ubuntu/ focal-security main restricted universe multiverse
    deb http://myLocalServer/ubuntu/ focal-backports main restricted universe multiverse
    
    deb http://myLocalServer/ubuntuPartner/ focal partner
    
    EOF
  12. Now do a sudo apt update and it should not throw any errors at you, if you have your Apache webserver configured well and it can access the mirrored files via the URL defined in the sources.list file.

I hope this helps someone.

Axel Werner 2020-10-08 13:34

it-artikel/linux/how-to-make-an-ubuntu-or-debian-apt-repository-locally-available-as-an-emergency-offline-fallback.txt · Last modified: 2022-08-31 12:30 by 127.0.0.1