tech_documents:linux:repo_mirror

Repo Mirror Using Rocky Linux 9

Repo Mirror Using Rocky Linux 9

https://help.ubuntu.com/community/Rsyncmirror
https://computingforgeeks.com/create-local-centos-7-centos-6-mirrors/

Plan is to create a slimmed down mirror of CentOS 7, Rocky Linux 8, Rocky Linux 9 ELRepo, Remi PHP Repo, Zimbra Repo, ZFS on Linux Repo and others for internal systems (not as a public mirror or as an archive). Starting with a 500GB disk.

Base Install

Download and install Rocky Linux 9 minimal, setup FQDN and set static IP, enable NTP and set timezone, set root password, use LVM for partitioning, select minimal software.

Install Misc Tools, Create Sudo User and Reboot

dnf install unzip wget vim tar rsync
dnf update
useradd example_user && passwd example_user
usermod -aG wheel example_user
shutdown -r now

Disallow root login over SSH

sudo rm /etc/ssh/sshd_config.d/01-permitrootlogin.conf

Restart sshd

sudo systemctl restart sshd

Setup/Sync Repos CentOS/Rocky

Create Repo Dirs

sudo mkdir -p /var/mirror/pub/centos/7
sudo mkdir -p /var/mirror/pub/rocky/8
sudo mkdir -p /var/mirror/pub/rocky/9
sudo mkdir -p /var/mirror/pub/epel/7
sudo mkdir -p /var/mirror/pub/epel/8
sudo mkdir -p /var/mirror/pub/epel/9
sudo mkdir -p /var/mirror/pub/remi/enterprise/7/php74
sudo mkdir -p /var/mirror/pub/remi/enterprise/8
sudo mkdir -p /var/mirror/pub/remi/enterprise/9
sudo mkdir -p /var/mirror/pub/debian

Repo Mirror Script for CentOS

https://gist.github.com/didip/4506837

Create file centos.sh and chmod +x it

Note: there are excludes so that only the packages I want are synced, modify to your needs

#!/bin/bash
 
# Number of arguments should be at least 1
if [ $# -lt 1 ]; then
  echo "Usage: $0 centos-version-number"
  exit 1
fi
 
VERSION=$1
 
# Change these variables as appropriate
LOCK_FILE=/tmp/centos_mirror_rsync_updates
DOWNLOAD_LOCATION=rsync://mirrors.kernel.org/centos
TARGET_DIR=/var/mirror/pub/centos
RSYNC_RETVAL=1
 
if [ -f $LOCK_FILE ]; then
  echo "CentOS updates via rsync already running."
  exit 0
fi
 
echo "Starting rsync from $DOWNLOAD_LOCATION/$VERSION to $TARGET_DIR/$VERSION..."
mkdir -p $TARGET_DIR/$VERSION
 
touch $LOCK_FILE
 
# add --exclude as necessary
rsync  -avSHP --delete --delete-excluded --copy-links --exclude "atomic" --exclude "cloud" --exclude "configmanagement" --exclude "cr" --exclude "dotnet" --exclude "fasttrack" --exclude "infra" --exclude "messaging" --exclude "nfv" --exclude "opstools" --exclude "paas" --exclude "rt" --exclude "sclo"  --exclude "storage"  --exclude "virt" $DOWNLOAD_LOCATION/$VERSION $TARGET_DIR/
RSYNC_RETVAL=$?
 
rsync -avSHP $DOWNLOAD_LOCATION/RPM-GPG-KEY-CentOS-$VERSION  $TARGET_DIR/RPM-GPG-KEY-CentOS-$VERSION 
 
/bin/rm -f $LOCK_FILE
 
if [ $RSYNC_RETVAL -eq 0 ]; then
  echo "Finished rsync from $DOWNLOAD_LOCATION/$VERSION to $TARGET_DIR/$VERSION."
fi
 
exit 0

Run the script for CentOS 7:

sudo ./centos.sh 7

Repo Mirror Script for Rocky Linux

https://gist.github.com/didip/4506837

Create file rocky.sh and chmod +x it

Note: there are excludes so that only the packages I want are synced, modify to your needs

#!/bin/bash
 
# Number of arguments should be at least 1
if [ $# -lt 1 ]; then
  echo "Usage: $0 rocky-version-number"
  exit 1
fi
 
VERSION=$1
 
# Change these variables as appropriate
LOCK_FILE=/tmp/rocky_mirror_rsync_update
DOWNLOAD_LOCATION=rsync://mirrors.vcea.wsu.edu/rocky
TARGET_DIR=/var/mirror/pub/rocky
RSYNC_RETVAL=1
 
if [ -f $LOCK_FILE ]; then
  echo "Rocky updates via rsync already running."
  exit 0
fi
 
echo "Starting rsync from $DOWNLOAD_LOCATION/$VERSION to $TARGET_DIR/$VERSION..."
mkdir -p $TARGET_DIR/$VERSION
 
touch $LOCK_FILE
 
# add --exclude as necessary
rsync  -avSHP --delete --delete-excluded --copy-links --exclude "AppStream/aarch64" --exclude "AppStream/source" --exclude "AppStream/x86_64/debug" --exclude "AppStream/x86_64/kickstart" --exclude "BaseOS/aarch64" --exclude "BaseOS/source" --exclude "BaseOS/x86_64/iso" --exclude "BaseOS/x86_64/debug" --exclude "BaseOS/x86_64/kickstart" --exclude "Devel/*" --exclude "HighAvailability/*" --exclude "isos/aarch64" --exclude "isos/x86_64" --exclude "live/x86_64" --exclude "Live/*" --exclude "Minimal/*" --exclude "PowerTools/*" --exclude "RT/*" --exclude "ResilientStorage/*" --exclude "extras/aarch64" --exclude "extras/source" --exclude "extras/x86_64/debug" --exclude "aarch64/*" --exclude "plus/source" --exclude "plus/x86_64/debug" --exclude "images/*"  --exclude "nfv/*"  --exclude "rockypi/*" --exclude "debug/*" --exclude "devel/*" $DOWNLOAD_LOCATION/$VERSION $TARGET_DIR/
RSYNC_RETVAL=$?
 
rsync -avSHP $DOWNLOAD_LOCATION/RPM-GPG-KEY-rockyofficial  $TARGET_DIR/RPM-GPG-KEY-rockyofficial
 
/bin/rm -f $LOCK_FILE
 
if [ $RSYNC_RETVAL -eq 0 ]; then
  echo "Finished rsync from $DOWNLOAD_LOCATION/$VERSION to $TARGET_DIR/$VERSION."
fi
 
exit 0

Run the script for Rocky Linux 8:

sudo ./rocky.sh 8

This a slightly customized exclude list for Rocky Linux 9, either use a separate script for Rocky 9 or merge this with the original Rocky script:

# add --exclude as necessary
rsync  -avSHP --delete --delete-excluded --copy-links --exclude "AppStream/aarch64" --exclude "AppStream/source" --exclude "AppStream/x86_64/debug" --exclude "AppStream/x86_64/kickstart" --exclude "aarch64/*" --exclude "BaseOS/source" --exclude "BaseOS/x86_64/iso" --exclude "BaseOS/x86_64/debug" --exclude "BaseOS/x86_64/kickstart" --exclude "Devel/*" --exclude "HighAvailability/*" --exclude "isos/aarch64" --exclude "isos/x86_64" --exclude "live/x86_64" --exclude "Live/*" --exclude "Minimal/*" --exclude "PowerTools/*" --exclude "RT/*" --exclude "ResilientStorage/*" --exclude "extras/aarch64" --exclude "extras/source" --exclude "extras/x86_64/debug" --exclude "plus/aarch64" --exclude "plus/source" --exclude "plus/x86_64/debug" --exclude "images/*"  --exclude "nfv/*"  --exclude "rockypi/*" --exclude "BaseOS/ppc64le/*" --exclude "BaseOS/s390x" --exclude "AppStream/ppc64le" --exclude "AppStream/s390x" --exclude "devel/aarch64" --exclude "ppc64le/*" --exclude "s390x/*" --exclude "devel/source" --exclude "i686/debug" --exclude "x86_64/debug" --exclude "isos/*" --exclude "live/*" --exclude "CRB/*" --exclude "kickstart/*" --exclude "debug/*" $DOWNLOAD_LOCATION/$VERSION $TARGET_DIR/
RSYNC_RETVAL=$?

EPEL

Same as Rocky but modified for EPEL. Pick an appropriate mirror closer to you and modify the exclusions as necessary

#!/bin/bash
 
# Number of arguments should be at least 1
if [ $# -lt 1 ]; then
  echo "Usage: $0 epel-version-number"
  exit 1
fi
 
VERSION=$1
 
# Change these variables as appropriate
LOCK_FILE=/tmp/epel_mirror_rsync_update
DOWNLOAD_LOCATION=rsync://dfw.mirror.rackspace.com/epel
TARGET_DIR=/var/mirror/pub/epel
RSYNC_RETVAL=1
 
if [ -f $LOCK_FILE ]; then
  echo "epel updates via rsync already running."
  exit 0
fi
 
echo "Starting rsync from $DOWNLOAD_LOCATION/$VERSION to $TARGET_DIR/$VERSION..."
mkdir -p $TARGET_DIR/$VERSION
 
touch $LOCK_FILE
 
# add --exclude as necessary
rsync  -avSHP --delete --delete-excluded --copy-links --exclude "Everything/SRPMS/" --exclude "Everything/aarch64/" --exclude "Everything/ppc64le" --exclude "Everything/s390x/" --exclude "Everything/source/" --exclude "Everything/x86_64/debug/" --exclude "Modular/SRPMS/" --exclude "Modular/aarch64/" --exclude "Modular/ppc64le" --exclude "Modular/s390x/" --exclude "Modular/source/" --exclude "Modular/x86_64/debug/" $DOWNLOAD_LOCATION/$VERSION $TARGET_DIR/
RSYNC_RETVAL=$?
 
rsync -avSHP $DOWNLOAD_LOCATION/RPM-GPG-KEY-EPEL-$VERSION  $TARGET_DIR/RPM-GPG-KEY-EPEL-$VERSION
 
/bin/rm -f $LOCK_FILE
 
if [ $RSYNC_RETVAL -eq 0 ]; then
  echo "Finished rsync from $DOWNLOAD_LOCATION/$VERSION to $TARGET_DIR/$VERSION."
fi
 
exit 0

Setup/Sync Repos Debian

We're going to use debmirror to create a private mirror: https://linux.die.net/man/1/debmirror

Install EPEL release and debmirror:

sudo dnf install epel-release
sudo dnf install debmirror

Edit the config file:

sudo vim /etc/debmirror.conf

Set the variables as such:

# Location of the local mirror (use with care)
$mirrordir="/var/mirror/debian";
 
# Output options
$verbose=1;
$progress=1;
$debug=0;
 
# Download options
$host="ftp.us.debian.org";
$user="anonymous";
$passwd="anonymous@";
$remoteroot="debian";
$download_method="rsync";
@dists="bullseye";
@sections="main,main/debian-installer,contrib,non-free";
@arches="amd64";

debian-archive-keyring.gpg

I couldn't find a simple way to get this file on a non-debian system so I ended up grabbing it from another Debian Bullseye/11 install.

Create dir for gpg file:

sudo mkdir -p /usr/share/keyring

Copy the debian-archive-keyring.gpg file to /usr/share/keyring

Import the key:

gpg --keyring /usr/share/keyrings/debian-archive-keyring.gpg --export | gpg --no-default-keyring --keyring trustedkeys.gpg --import

Run the mirror sync

sudo ./debmirror

ZFS on Linux

wget -r -np -nH -N http://download.zfsonlinux.org/epel/8/ -P /var/mirror/pub/zfs
wget -r -np -nH -N http://download.zfsonlinux.org/epel/9/ -P /var/mirror/pub/zfs
wget -r -l1 -np -nH -N "http://download.zfsonlinux.org/epel" -P /var/mirror/pub/zfs -A "*.rpm"

Note: when using the -N flag, it seems to check only the index.html file at the root of each folder and use that to determine if there are newer files. If you delete an rpm file in a subdirectory and run wget again it won't download the new file. If you delete the index.html in the subdirectory of the deleted rpm file it still won't re-download. You have to delete the index.html in parent folders as well as the final folder for the rpm file to be re-downloaded using the -N flag.

Install Apache

Configure hosts

sudo vim /etc/hosts

Add a line for your FQDN

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.10.30  mirror.domainname.com mirror

Install Apache and Configure Firewall

sudo dnf install httpd httpd-tools
sudo systemctl enable httpd
sudo systemctl start httpd
sudo firewall-cmd --permanent --zone=public --add-service=http
sudo firewall-cmd --permanent --zone=public --add-service=https
sudo firewall-cmd --reload

Dell Server Updates (not working, not sure how to use DSU with a local mirror)

https://linux.dell.com/repo/hardware/omsa.html#Creating_a_local_mirror
Note: The excludes listed below we configured to support a Dell Poweredge T350 on November 4th 2024 for RHEL 8/9. To keep things trim or to include older versions adjust as needed

rsync -avHz linux.dell.com::repo/hardware /var/mirror/pub/dell/repo --exclude "OMSA*/*" --exclude "DCIA*/*" --exclude "DSU_1*/*"  --exclude "DSU_20*/*" --exclude "DSU_21*/*" --exclude "DSU_22*/*" --exclude "DSU_23*/*" --exclude "DSU_24.0*/*" --exclude "ISM*/*" --exclude "customISO*/*" --exclude "Linux_Repository_14*/*" --exclude "Linux_Repository*/*" --exclude "RHEL7*/*" --exclude "SLES*/*" --exclude "omnia*/*" --exclude "vcenter-plugin*/*"

Grap the pgp pubkeys:

sudo rm -rf /var/mirror/pub/dell/repo/pgp_pubkeys
sudo rm -f /var/mirror/pub/dell/repo/index.html
sudo rm -f /var/mirror/pub/dell/repo/pgp_pubkeys.tmp
sudo rm -f /var/mirror/pub/dell/index.html
sudo rm -f /var/mirror/pub/dell/robots.txt
wget -r -l1 -np -nH -N "https://linux.dell.com/repo/pgp_pubkeys" -P /var/mirror/pub/dell -A "*.*"

Get the latest bootstrap and put it in the dsu folder (otherwise you'll have to update your path to the dus_24.x.x folder to get the latest version)

wget -np -nH -N https://linux.dell.com/repo/hardware/dsu/bootstrap.cgi -P /var/mirror/pub/dell/repo/hardware/dsu

On the client Dell PowerEdge server that needs dsu installed (replace with your own variables ofc):

curl -O http://repo01.int.yourinternalrepo.com/pub/dell/repo/hardware/dsu/bootstrap.cgi

Om the client Dell PowerEdge Server edit bootstrap.cgi and set the domain and path variables:

# The purpose of this script is to set up the Dell yum repositories on your 
# system. This script will also install the Dell GPG keys used to sign 
# Dell RPMS.

# these two variables are replaced by the perl script 
# with the actual server name and directory.
SERVER="http://repo01.int.courtesynetwork.com"
# mind the trailing slash here...
REPO_URL="/pub/dell/repo/hardware/dsu/"
REPO_NAME="dell-system-update"
GPG_KEY_REPO="/pub/dell/repo/pgp_pubkeys/"
# these are 'eval'-ed to do var replacement later.
GPG_KEY_LOCATION=${SERVER}${GPG_KEY_REPO}
#DSU_INSTALL_PATH='/usr/libexec/dell_dup/'

Then do the install on the client

sudo bash bootstrap.cgi

If successful edit the Dell repo added:

sudo vim /etc/yum.repos.d/dell-system-update.repo

You should see the internal repo you setup. It's wonderful isn't it!!!! Just disable the [dell-system-update_dependent] repo since we won't be using OMSA

[dell-system-update_dependent]
.....
enabled=0

Then install Dell System Update

sudo dnf install dell-system-update

Finally run DSU

sudo dsu

Create Virtual Hosts

Setup virtual host config files

sudo vim /etc/httpd/conf.d/mirror.example.conf

Add the following content. Make sure the website directory path, server name, and PHP version match your setup:

<Directory /var/mirror>
        Require all granted
        Options +Indexes
</Directory>
<VirtualHost *:80>
     ServerAdmin webmaster@example.com
     ServerName mirror.example.com
     DocumentRoot /var/mirror
     ErrorLog /var/log/httpd/mirror.example.com_error.log
     CustomLog /var/log/httpd/mirror.example.com_access.log combined
</VirtualHost>
<IfModule mod_autoindex.c>
  IndexOptions NameWidth=*
</ifModule>

Save and close the file when you are finished. Then check the Apache configuration file for any syntax errors:

sudo httpd -t

You’ll see the following output:

Output
Syntax OK

Disable Exposing Apache Server Info

https://www.if-not-true-then-false.com/2009/howto-hide-and-modify-apache-server-information-serversignature-and-servertokens-and-hide-php-version-x-powered-by/
This will help to prevent being the target of a an attack if you're running an out of date or vunerable release

sudo vim /etc/httpd/conf/httpd.conf

Add the following:

ServerSignature Off
ServerTokens Prod

delete the welconf.conf to disable the default apache welcome page.

sudo rm /etc/httpd/conf.d/welcome.conf

Remove Default Access from httpd.conf

This is important as it would allow access to the /var/www/html path via HTTP.

sudo vim /etc/httpd/conf/httpd.conf

Modify the following statements to mirror the examples below:

DocumentRoot "/var/www/html"

<Directory "/var/www">
    AllowOverride None
    Require all denied
    Options None
</Directory>

<Directory "/var/www/html">
    AllowOverride None
    Require all denied
    Options None
</Directory>

<Directory "/var/www/cgi-bin">
    AllowOverride None
    Require all denied
    Options None
</Directory>

Set Owner and File Permissons

sudo chown -R apache:apache /var/mirror
sudo chmod -R 755 /var/mirror

Set SElinux Permissions

https://www.redhat.com/sysadmin/apache-yum-dnf-repo

sudo chcon -Rt httpd_sys_content_t /var/mirror/

Finally, restart the Apache service to implement your changes:

sudo systemctl restart httpd

Automatic Updates for Rocky Linux

https://www.tecmint.com/dnf-automatic-install-security-updates-automatically-in-centos-8/

sudo dnf install dnf-automatic
sudo vim /etc/dnf/automatic.conf

Set:

upgrade_type = default
download_updates = yes
apply_updates = yes
reboot = when-needed
system_name = (your system name)
emit_via = motd

Enable the auto-update timer

sudo systemctl enable --now dnf-automatic.timer

fail2ban

https://idroot.us/install-fail2ban-centos-8/ https://www.digitalocean.com/community/tutorials/how-to-protect-an-apache-server-with-fail2ban-on-ubuntu-14-04

sudo dnf install epel-release
sudo dnf install fail2ban

Create a Jail for SSHd

sudo vim /etc/fail2ban/jail.d/sshd.local

Add the following:

[sshd]
enabled = true
port    = ssh
logpath = %(sshd_log)s
backend = %(sshd_backend)s

[selinux-ssh]
enabled  = true
port     = ssh
logpath  = %(auditd_log)s

Create a Jail for Apache

sudo vim /etc/fail2ban/jail.d/apache.local

Add the following:

[apache-auth]
port     = http,https
logpath  = %(apache_error_log)s


[apache-badbots]
# Ban hosts which agent identifies spammer robots crawling the web
# for email addresses. The mail outputs are buffered.
port     = http,https
logpath  = %(apache_access_log)s
bantime  = 48h
maxretry = 1

[apache-noscript]
port     = http,https
logpath  = %(apache_error_log)s


[apache-overflows]
port     = http,https
logpath  = %(apache_error_log)s
maxretry = 2


[apache-nohome]
port     = http,https
logpath  = %(apache_error_log)s
maxretry = 2

[apache-botsearch]
port     = http,https
logpath  = %(apache_error_log)s
maxretry = 2


[apache-fakegooglebot]
port     = http,https
logpath  = %(apache_access_log)s
maxretry = 1
ignorecommand = %(fail2ban_confpath)s/filter.d/ignorecommands/apache-fakegooglebot <ip>


[apache-modsecurity]
port     = http,https
logpath  = %(apache_error_log)s
maxretry = 2

[apache-shellshock]
port    = http,https
logpath = %(apache_error_log)s
maxretry = 1

sudo systemctl start fail2ban
sudo systemctl enable fail2ban
sudo fail2ban-client status sshd

Configure Client Repos

CentOS 7

Edit the Repo Config

sudo vim /etc/yum.repos.d/CentOS-Base.repo

Comment out any line like

mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=os&infra=$infra

Uncomment the baseurl lines and change the FQDN to your FQDN or IP of the repo server

baseurl=http://10.222.0.21/centos/$releasever/os/$basearch/

Now you should be able to do a yum update.

Rocky 8

sudo vim /etc/yum.repos.d/Rocky-AppStream.repo

Modify as such, commenting out mirror and uncommenting baseurl then modifying the url:

# Rocky-AppStream.repo
#
# The mirrorlist system uses the connecting IP address of the client and the
# update status of each mirror to pick current mirrors that are geographically
# close to the client.  You should use this for Rocky updates unless you are
# manually picking other mirrors.
#
# If the mirrorlist does not work for you, you can try the commented out
# baseurl line instead.

[appstream]
name=Rocky Linux $releasever - AppStream
#mirrorlist=https://mirrors.rockylinux.org/mirrorlist?arch=$basearch&repo=AppStream-$releasever
baseurl=http://10.222.0.21/$contentdir/$releasever/AppStream/$basearch/os/
gpgcheck=1
enabled=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-rockyofficial

Repeat for /etc/yum.repos.d/Rocky-BaseOS.repo and any other branches you've included and will use from your local mirror.

For Rocky 9, rocky.repo and rock-extras.repo

Table of Contents

Repo Mirror Using Rocky Linux 9

Base Install

Install Misc Tools, Create Sudo User and Reboot

Disallow root login over SSH

Setup/Sync Repos CentOS/Rocky

Repo Mirror Script for CentOS

Repo Mirror Script for Rocky Linux

EPEL

Setup/Sync Repos Debian

debian-archive-keyring.gpg

ZFS on Linux

Install Apache

Configure hosts

Install Apache and Configure Firewall

Dell Server Updates (not working, not sure how to use DSU with a local mirror)

Create Virtual Hosts

Setup virtual host config files

Disable Exposing Apache Server Info

Remove Default Access from httpd.conf

Set Owner and File Permissons

Set SElinux Permissions

Automatic Updates for Rocky Linux

fail2ban

Create a Jail for SSHd

Create a Jail for Apache

Configure Client Repos

CentOS 7