Hello, boys and girls! This is a follow-up to an article I wrote a while back about why sysadmins and developers use packages to distribute software.
On today’s episode of “NO”, we’re going to go over how to create your own RPM repo. As it turns out, this year I’ve ended up with tons, and tons, and tons of software I want to distribute internally and externally with all these projects and need a clean way of managing it. I even have a need now for a CICD pipeline, which is a thing all of its own.
This is an often debated topic among some of my circles, where, in many cases, the amount of work in building packages and setting up a repo just frankly outweighs the benefits that packaging software for internal distribution and then distributing from a repo has.
I can certainly sympathize with that, so, I thought I’d list out the steps so that I can keep a record of breadcrumbs along this treacherous minefield of hardships towards:
- Setting up an RPM repository.
- Configuring client machines to use that repository.
- Understanding CI/CD benefits with RPM.
I’ve certainly got my work ahead of me on this one as I’ve heard this is just an enormous and time consuming amount of work.
I’ll have to break it down into smaller pieces I think to get it all done this year.
Key: Items colored in purple are entry-level system administrator tasks that should be second nature to you by the time you really need to have a custom repo. Items colored in red are specific to the task of setting up an RPM repository or a client to use an RPM repository. Items colored in black are mostly sarcasm and general assery. Items colored in yellow are commands that you type. Items colored white are the output of those commands you should expect. *I'm extremely colorblind, so, if I got the shades wrong, let me know so that I can tell you how much I really, really....really care.
1.1 Get a Server
Get a machine to host the repo on. In my case I used a linode VPS. This is a straight forward process that I can’t explain for you much more than just to say “Pick a VM size, pick a distro, deploy the image, start the server”. It’s not very hard.
I picked Fedora 26 for my OS which is a kind of testing distro for next-generation RHEL. The commands will be identical on any redhat-based distribution, such as CentOS, or Fedora, or Oracle, or whatever you are using.
1.2 Point your DNS at it.
This may be a little more advanced for some but I can’t explain how to set up DNS for your environment. In my case I used repo.silogroup.org by creating an A record pointing to the server IP, and then setting up reverse DNS on the linode.
1.3 Install Apache.
Now remember on Fedora you’re using DNF now which is the next generation of yum. DNF has legacy symlinks and CLI options for yum but it’s better to get in the habit of typing it now as opposed to later so that you’re able to transition, and, there’s no guarantee it’ll just be there waiting for you in a future version of your distribution.
sudo dnf -y install httpd
If you’re on CentOS or similar:
sudo yum -y install httpd
1.4 Create some directories.
You’ll need a place to store your packages.
[phanes@repo ~]$ sudo mkdir -p /repos/{CentOS,Fedora,SURRO}
[phanes@repo ~]$ sudo mkdir -p /repos/CentOS/{5,6,7}/Packages
[phanes@repo ~]$ sudo mkdir -p /repos/Fedora/26/Packages
[phanes@repo ~]$ sudo mkdir -p /repos/SURRO/PRE-ALPHA/Packages
[phanes@repo ~]$ tree /repos
/repos
├── CentOS
│ ├── 5
│ │ └── Packages
│ ├── 6
│ │ └── Packages
│ └── 7
│ └── Packages
├── Fedora
│ └── 26
│ └── Packages
└── SURRO
└── PRE-ALPHA
└── Packages
13 directories, 0 files
Ok, so, I lied. In this guide we’re making five repos instead of one.
This is going to be a crazy amount of work, right? I always do this, I blow my project scope with additional considerations that’ll save me time later, like I’m planning ahead or something.
1.5 Initialize the repository
Repositories, as should be expected, keep some metadata about the packages they’re hosting so that clients can connect and do the awesome things that package managers do.
The grueling process of initializing the repo database is as follows:
[phanes@repo ~]$ sudo createrepo /repos/CentOS/6 Saving Primary metadata Saving file lists metadata Saving other metadata Generating sqlite DBs Sqlite DBs complete [phanes@repo ~]$ sudo createrepo /repos/CentOS/5 Saving Primary metadata Saving file lists metadata Saving other metadata Generating sqlite DBs Sqlite DBs complete [phanes@repo ~]$ sudo createrepo /repos/CentOS/7 Saving Primary metadata Saving file lists metadata Saving other metadata Generating sqlite DBs Sqlite DBs complete [phanes@repo ~]$ sudo createrepo /repos/Fedora/26 Saving Primary metadata Saving file lists metadata Saving other metadata Generating sqlite DBs Sqlite DBs complete [phanes@repo ~]$ sudo createrepo /repos/SURRO/PRE-ALPHA Saving Primary metadata Saving file lists metadata Saving other metadata Generating sqlite DBs Sqlite DBs complete
Man, this is exhausting. So at this point all of the databases for all of the repos are initialized. If you’re tired at this point it’s advisable to take a break, because, like, the rest is really, really hard comparatively.
1.6 Open port 80 and 443 in FirewallD.
This is standard for any installation of apache on a redhat-based distribution so you should be used to it if you’re familiar with these systems already (and here is the code if you’re not):
[phanes@repo ~]$ sudo firewall-cmd --permanent --add-port={80,443}/tcp --permanent success [phanes@repo ~]$ sudo firewall-cmd --reload success
1.7 Symlink the Repos to THE Apache DIR
Before you bring up Apache it’ll need something to actually serve, though.
For reasons I won’t get into here, I’ll just say that this step is needed and do not just plop the rpm’s into Apache’s existing document root.
[phanes@repo ~]$ sudo ln -s /repos/CentOS /var/www/html/CentOS [phanes@repo ~]$ sudo ln -s /repos/Fedora /var/www/html/Fedora [phanes@repo ~]$ sudo ln -s /repos/SURRO /var/www/html/SURRO [phanes@repo ~]$ tree /var/www/html /var/www/html ├── CentOS -> /repos/CentOS ├── Fedora -> /repos/Fedora └── SURRO -> /repos/SURRO
….symlinks, yay!
1.8 Enable the Apache Service
Like I covered during my disgustingly over-explained systemd pattern series, we will want to enable the Apache service.
I’m only using one mode and set of configs for Apache in this case because I have a dedicated server for a repo. If you’re lumping your RPM repos in with an already existing Apache you will probably want to do something different (something else, like, not doing that, amirite?).
Just in case:
[phanes@repo ~]$ sudo systemctl enable httpd
1.9 Start the Apache Service
Riveting! Just in case:
[phanes@repo ~]$ sudo systemctl start httpd
2.0 Adding Packages
At this point you’re probably wondering “Ok, I’ve got an empty repo. Now how do I get packages to it?”.
This is up to you and largely depends on how you’re creating your RPMs, to be honest. In my case, I am using a local VM running a Jenkins instance to build some project-specific RPMs but that’s out of scope for this guide.
I’m also set up to use Jenkins to stage the rpm server which hides all the crazy complex work to do that, but for the purpose of demonstration I won’t be doing that:
2.1 Copy the packages to repo
[jenkins@jenkins rpmbuild]# pwd /var/lib/jenkins/rpmbuild [jenkins@jenkins rpmbuild]# scp RPMS/noarch/tenta-0.20171015.91-1.fc26.noarch.rpm repo.silogroup.org:/repos/Fedora/26/Packages/ tenta-0.20171015.91-1.fc26.noarch.rpm 100% 13KB 158.2KB/s 00:00
As you can see, while it may be craaaazy complicated, I used scp to move the package to the repo.
I then issued the command needed to update the repo database to make the package available as seen below.
2.2 Update the repo Database
[phanes@repo 26]$ sudo createrepo --update /repos/Fedora/26
Spawning worker 0 with 1 pkgs
Workers Finished
Saving Primary metadata
Saving file lists metadata
Saving other metadata
Generating sqlite DBs
Sqlite DBs complete
You really should have your CI server issue this command if you’re using one.
And there you have it. You are now a lean**, mean, packaging-ready machine.*
USING THE REPO / Client Machines
I know what you’re thinking: “He said this would be exhausting and time consuming and expensive but that only took like five minutes and only needed one person. What gives?”.
Here’s where the work is at:
You now need to configure machines to use your repo if you actually want to use it like a repo. Scary, amirite?
Fear not, bromance***, I’ve got that mapped out for your project too to traverse the angry seas of package management!
3.0 Configuring Repo on Client Machine
In this case I’m deploying super-seekret mind control technology to a server I don’t want people to use so I’m redacting everything you don’t need to know.
3.1 Create a yum repo file
First, create a .repo file in /etc/yum.repos.d:
[ phanes@[seekret-removed ] << ~ >>
[- sudo vim /etc/yum.repos.d/silogroup.repo
Second, add these five lines:
[silogroup]
name=SILO GROUP Fedora RPM Repo
baseurl=http://repo.silogroup.org/Fedora/26
gpgcheck=0
enabled=1
These are pretty self explanatory. You’ve got a section header, a name, a baseurl, a gpgcheck, and an enabled.
The section header is what shows up in yum as a handle for the repo.
The name value is the descriptive name of the repo. This will also show up in yum.
The baseurl value is the actual url for your repo. This can take http, https, and FTP (and I think some others but don’t do that).
The gpgcheck value deserves being described. This is a way of ensuring the package integrity by the yum client during download/install. If you gpg-sign the packages that’s good. If you host your repo via https that’s also good. These are both things you should do that I will not tell you how to do in this article (maybe next week?).
The enabled value tells yum on the client machine whether or not to actually use the repo. Handy, amirite?
So check this out, all ye bromancers of the stones, the demo you have been exhaustingly working your way towards, using a client I wrote in python called tenta that I packaged just for this demo:
4.0 Install the package with yum:
Or do it via a configuration management system which is actually still using yum under the hood. I won’t be for this demonstration.
[ phanes@super-seekret-removed ] << ~ >> [- sudo dnf install tenta Last metadata expiration check: 0:03:50 ago on Sat 21 Oct 2017 09:02:35 PM UTC. Dependencies resolved. ================================================================================================ Package Arch Version Repository Size ================================================================================================ Installing: tenta noarch 0.20171015.91-1.fc26 silogroup 13 k Transaction Summary ================================================================================================ Install 1 Package Total download size: 13 k Installed size: 28 k Is this ok [y/N]: y Downloading Packages: tenta-0.20171015.91-1.fc26.noarch.rpm 46 kB/s | 13 kB 00:00 ------------------------------------------------------------------------------------------------ Total 46 kB/s | 13 kB 00:00 Running transaction check Transaction check succeeded. Running transaction test Transaction test succeeded. Running transaction Preparing : 1/1 Installing : tenta-0.20171015.91-1.fc26.noarch 1/1 Running scriptlet: tenta-0.20171015.91-1.fc26.noarch 1/1 Verifying : tenta-0.20171015.91-1.fc26.noarch 1/1 Installed: tenta.noarch 0.20171015.91-1.fc26 Complete!
Super hard, right? It took longer to write this guide and the guide only took about 10 minutes.
And here’s where the files from the demo package ended up on the system:
[ phanes@sekret-server ] << ~ >>
[- rpm -ql tenta
/etc/tenta
/etc/tenta/nerve.conf
/etc/tenta/tentacle.conf
/usr/bin/nerve
/usr/bin/tentacle
/usr/lib/systemd/system/nerve.service
/usr/lib/systemd/system/tentacle.service
/usr/share/man/man1/tenta.1.gz
/var/log/tenta
/var/log/tenta/error.log
/var/log/tenta/producer_source.log
/var/log/tenta/tentacle.log
I am also able to determine now if a file has been edited since the package was installed, and much more.
Notes
*In Fedora 26, the default Apache configuration will still show a splash page for Apache at this point. You may want to add a landing page at /var/www/html/index.html just to polish it out. In my case I just commented the "Options -Indexes" line in /etc/httpd/conf.d/welcome.conf because I’m using a dedicated server for this and literally only want directory listings.
** Not to be confused with Lean Transformation Model, which destroys organizations, their cultures, costs more money than it saves in most implementations and suppresses talent in workforces amidst a new atmosphere of fear and instability while somehow sounding really cool to stakeholders. Lean is a westernization of the Toyota Production System used in the 1950s in Japan and doesn’t translate well to western business models (and is extremely harmful to software engineering process), but it did create an entire market of consultancy for how to implement it.
*** This is sarcasm. This is not hard, not complex at all, and it saves you tons of time.
How I use it
In my case, the servers intended to use the packages are controlled by a configuration management (CM) system. The CM’s manifest has an entry to ensure the latest version of the package from the repo at all times. This will force the machine to update the package automatically. This is only for my pre-production environment — I version lock it in my production environment manifest to give me a scheduled release cycle.
For preproduction CI, the Jenkins server is set to listen for changes to my git repository’s master branch to automatically generate an RPM build immediately and then pushes that RPM to the RPM repository, and then updates that repository for consumption by the client.
This way I can commit my code, install my code in a clean way that’s auditable and manageable even if I had tens, or hundreds, or thousands of packages going, I can see the changes automatically in my pre-production environment for rapid testing, and I can accommodate reliable release management with almost no effort at all– well, a little effort — you just read all of it. Oh– and I’ll never have a single mistake in deployment.
If you’re curious what the jenkins job looks like, too bad, but, I can give you this early version to build on that does almost that. The filename and version are where some magic happens. You’ll want to choose between timestamps + build number or git revision numbers.
You’ll have to restructure the order of variable assignment if you want the latter (trust me both you and your devs want the latter but this is a freebie).
#!/usr/bin/bash
set -m
###
# PREWORK
###
# RPMBUILD ROOT CONTEXT
TOPDIR=$(rpm --eval='%{_topdir}')
if [ ! -d "$TOPDIR" ]
then
mkdir -p $HOME/rpmbuild/{BUILD,BUILDROOT,RPMS,SOURCES,SPECS,SRPMS,tmp}
mkdir -p $HOME/rpmbuild/RPMS/{arm7,arm9,i386,i686,mips,miple,noarch,ppc,ppc64,x86_64}
echo "%_topdir $HOME/rpmbuild" >> ~/.rpmmacros
TOPDIR=$(rpm --eval='%{_topdir}')
fi
echo "TOPDIR $TOPDIR"
# TMP PATH
TMPPATH=$(rpm --eval='%{_tmppath}')
if [ ! -d "$TMPPATH" ] || [ "$TMPPATH" = "/var/tmp" ]
then
mkdir -p "$TOPDIR/tmp"
echo "%_tmppath $TOPDIR/tmp" >> ~/.rpmmacros
TMPPATH=$(rpm --eval='%{_tmppath}')
fi
echo "TMPPATH $TMPPATH"
GIT_REPO_BASENAME="tenta"
GIT_REPO="git@source.surroindustries.com:ircthulu/tenta.git"
echo "GIT_REPO_BASENAME $GIT_REPO_BASENAME"
echo "GIT_REPO $GIT_REPO"
# The name of the RPM that is being built
APPNAME="tenta"
echo "APPNAME $APPNAME"
# version number for this build
VERSION=0.$(date +%Y%m%d)
echo "VERSION $VERSION"
# The directory containing .spec files
SPEC_DIR="$TOPDIR/SPECS"
echo "SPEC_DIR $SPEC_DIR"
# the target spec file
SPEC_FILE="$SPEC_DIR/$APPNAME.spec"
echo "SPEC_FILE $SPEC_FILE"
# the workspace dir
prearchive_dir=$APPNAME-$VERSION
echo "prearchive_dir $prearchive_dir"
# the source spec file template
PRE_SPEC_FILE="$prearchive_dir/$APPNAME.spec"
echo "PRE_SPEC_FILE $PRE_SPEC_FILE"
###
# WORK
###
rm -Rf ./*
# create a dest dir (ensuring write access to target fs)
mkdir -p "$prearchive_dir"
printf "\n***\n*** OBTAINING AND HOMOGENIZING SOURCE ***\n***\n"
# Get the source
git clone "$GIT_REPO"
# deglove the unnecessary git repo and child dir
mv $GIT_REPO_BASENAME/* $prearchive_dir &&
rm -Rf $GIT_REPO_BASENAME
echo "GIT REPO $GIT_REPO_BASENAME => $prearchive_dir"
# cat the spec file for the log
printf "\n***\n*** SPEC FILE TEMPLATE ***\n***\n"
cat $PRE_SPEC_FILE
###
## SPEC VARS
## These are variables that are expected to be in the SPEC file.
###
SPEC_VERSION="__VERSION__"
###
# Transform the SPEC file
###
mv $PRE_SPEC_FILE $SPEC_FILE
sed -i "s,$SPEC_VERSION,$VERSION," $SPEC_FILE
printf "\n***\n*** SPEC FILE INTERPOLATED ***\n***\n"
# cat the SPEC file for the log
cat $SPEC_FILE
# bundle up the now distributable source tarball to the rpmbuild SOURCES dir
printf "\n***\n*** SOURCE ARCHIVE CONTENTS ***\n***\n"
tar cvzf $TOPDIR/SOURCES/$prearchive_dir.tar.gz ./$prearchive_dir &&
rm -rf $prearchive_dir
echo "Archive location: $TOPDIR/SOURCES/$prearchive_dir.tar.gz"
printf "\n***\n*** RPM GENERATION ***\n***\n"
rpmbuild -ba $SPEC_FILE
printf "\n***\n*** STAGING RPM to Repository ***\n***\n"
scp $TOPDIR/RPMS/noarch/$APPNAME-$VERSION-*.fc26.noarch.rpm repo.silogroup.org:/repos/Fedora/26/Packages/
printf "\n***\n*** Updating RPM Repository Metadata ***\n***\n"
ssh repo.silogroup.org createrepo --update /repos/Fedora/26