Puppet, Round II. Mistakes shape Successes.

Since I can’t stand a server of mine that isn’t pristine, I’d rather set it on fire and reinstall the OS before I capture a backup so I don’t have to have historical knowledge of the server and critical services to be able to do work on it.  You need an origin as a fixed point to refer to during rollbacks and future changes.

From there, I want documented, controlled, change management in place, so that instead of relying on memory (which I destroyed hopelessly beyond repair in college) I can refer to sane, clear, simple documentation.  I set up everything I touch in a way that someone who’s never seen it before but knows Linux standards can use it — just like I do for every piece of software I write — respect the standards.  I’ve had to use systems before that did not do this and that’s a special kind of hell — and, I’d like to bring other people into this once I get it built out and capable of producing software.

With another new fresh install on the same host, I can’t help but think I would have been better off if instead of installing ubuntu server directly on the rack server, if I’d gone with something like ESXi or another hypervisor, and installed oldhorse as a VM using nonvirtualized resources.  Too little too late, maybe next time.

I’ve got a PowerEdge 2850 currently in hardware raid 0 with six blades spinning at 15k RPM so the install went pretty quickly — half the process was waiting for the big turd to cycle on reboot.  I’ve decided to have a local user that has sudoers access named ‘surro’ for emergency management and then limit the hosts from which that user can connect.

Put on my Sonic Area playlist for the documentation meltdown I’m expecting while I steam the lab as this is definitely a latenight mission:

These guys are great.

Alright, Ubuntu 16.04.1 LTS (Xenial Xerus) Server, fresh install.  Working user is surro since we haven’t set up LDAP yet.

Tiredness is kicking in.  This is actually productive, as, oddly enough, when I’m tired I get a little more put together and intelligent than when I’m rested.  Time to pour some motivation coffee and knock this out.

We have the proper documentation this time:

https://docs.puppet.com/puppet/4.7/reference/

Dammit.

The modern documentation is quite different.  Still a little scattered but might actually be useful.  Looks like Xenial is actually supported.

Note: We will be utilizing puppetDB for the master’s data storage.

Because of this, we will want to use Puppet Collection 1:

• puppet-agent
• puppetserver
• puppetdb
• puppetdb-termini (puppet<>puppetdb interface

wget https://apt.puppetlabs.com/puppetlabs-release-pc1-xenial.deb
sudo dpkg -i puppetlabs-release-pc1-xenial.deb
sudo apt update

According to the docs.

surro@oldhorse:~$ wget https://apt.puppetlabs.com/puppetlabs-release-pc1-xenial.deb && sudo dpkg -i puppetlabs-release-pc1-xenial.deb && sudo apt update--2016-09-26 02:53:41-- https://apt.puppetlabs.com/puppetlabs-release-pc1-xenial.deb
Resolving apt.puppetlabs.com (apt.puppetlabs.com)... 198.58.114.168, 2600:3c00::f03c:91ff:fe69:6bf0
Connecting to apt.puppetlabs.com (apt.puppetlabs.com)|198.58.114.168|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 13662 (13K) [application/x-debian-package]
Saving to: ‘puppetlabs-release-pc1-xenial.deb’

puppetlabs-release-pc1-xenial.deb 100%[=========================================================================>] 13.34K --.-KB/s in 0.001s

2016-09-26 02:53:41 (12.2 MB/s) - ‘puppetlabs-release-pc1-xenial.deb’ saved [13662/13662]

[sudo] password for surro:
Selecting previously unselected package puppetlabs-release-pc1.
(Reading database ... 60846 files and directories currently installed.)
Preparing to unpack puppetlabs-release-pc1-xenial.deb ...
Unpacking puppetlabs-release-pc1 (1.1.0-2xenial) ...
Setting up puppetlabs-release-pc1 (1.1.0-2xenial) ...
Hit:1 https://us.archive.ubuntu.com/ubuntu xenial InRelease
Get:2 https://us.archive.ubuntu.com/ubuntu xenial-updates InRelease [95.7 kB]
Get:3 https://security.ubuntu.com/ubuntu xenial-security InRelease [94.5 kB]
Ign:4 https://apt.puppetlabs.com xenial InRelease
Hit:5 https://us.archive.ubuntu.com/ubuntu xenial-backports InRelease
Get:6 https://apt.puppetlabs.com xenial Release [13.3 kB]
Get:7 https://apt.puppetlabs.com xenial Release.gpg [836 B]
Get:8 https://apt.puppetlabs.com xenial/PC1 amd64 Packages [7,937 B]
Get:9 https://apt.puppetlabs.com xenial/PC1 i386 Packages [7,462 B]
Get:10 https://apt.puppetlabs.com xenial/PC1 all Packages [4,784 B]
Fetched 224 kB in 0s (231 kB/s)
Reading package lists... Done
Building dependency tree
Reading state information... Done
62 packages can be upgraded. Run 'apt list --upgradable' to see them.
surro@oldhorse:~$

So far so good.   Now we’re installing the puppetserver package:

surro@oldhorse:~$ sudo apt-get install puppetserver
Reading package lists... Done
Building dependency tree 
Reading state information... Done
The following additional packages will be installed:
 ca-certificates-java fontconfig-config fonts-dejavu-core java-common libcups2 libfontconfig1 liblcms2-2 libpcsclite1 libxi6 libxrender1 libxtst6
 openjdk-8-jre-headless puppet-agent x11-common
Suggested packages:
 default-jre cups-common liblcms2-utils pcscd openjdk-8-jre-jamvm libnss-mdns fonts-dejavu-extra fonts-ipafont-gothic fonts-ipafont-mincho
 ttf-wqy-microhei | ttf-wqy-zenhei fonts-indic
The following NEW packages will be installed:
 ca-certificates-java fontconfig-config fonts-dejavu-core java-common libcups2 libfontconfig1 liblcms2-2 libpcsclite1 libxi6 libxrender1 libxtst6
 openjdk-8-jre-headless puppet-agent puppetserver x11-common
0 upgraded, 15 newly installed, 0 to remove and 62 not upgraded.
Need to get 75.7 MB of archives.
After this operation, 225 MB of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 https://us.archive.ubuntu.com/ubuntu xenial/main amd64 liblcms2-2 amd64 2.6-3ubuntu2 [137 kB]
Get:2 https://apt.puppetlabs.com xenial/PC1 amd64 puppet-agent amd64 1.7.0-1xenial [13.9 MB]
Get:3 https://us.archive.ubuntu.com/ubuntu xenial/main amd64 x11-common all 1:7.7+13ubuntu3 [22.4 kB]
Get:4 https://us.archive.ubuntu.com/ubuntu xenial/main amd64 libxtst6 amd64 2:1.2.2-1 [14.1 kB]
Get:5 https://us.archive.ubuntu.com/ubuntu xenial/main amd64 ca-certificates-java all 20160321 [12.9 kB]
Get:6 https://us.archive.ubuntu.com/ubuntu xenial/main amd64 java-common all 0.56ubuntu2 [7,742 B]
Get:7 https://us.archive.ubuntu.com/ubuntu xenial/main amd64 libcups2 amd64 2.1.3-4 [197 kB]
Get:8 https://us.archive.ubuntu.com/ubuntu xenial/main amd64 fonts-dejavu-core all 2.35-1 [1,039 kB]
Get:9 https://apt.puppetlabs.com xenial/PC1 amd64 puppetserver all 2.6.0-1puppetlabs1 [33.3 MB]
Get:10 https://us.archive.ubuntu.com/ubuntu xenial-updates/main amd64 fontconfig-config all 2.11.94-0ubuntu1.1 [49.9 kB] 
Get:11 https://us.archive.ubuntu.com/ubuntu xenial-updates/main amd64 libfontconfig1 amd64 2.11.94-0ubuntu1.1 [131 kB] 
Get:12 https://us.archive.ubuntu.com/ubuntu xenial/main amd64 libpcsclite1 amd64 1.8.14-1ubuntu1 [21.4 kB] 
Get:13 https://us.archive.ubuntu.com/ubuntu xenial/main amd64 libxi6 amd64 2:1.7.6-1 [28.6 kB] 
Get:14 https://us.archive.ubuntu.com/ubuntu xenial/main amd64 libxrender1 amd64 1:0.9.9-0ubuntu1 [18.5 kB] 
Get:15 https://us.archive.ubuntu.com/ubuntu xenial-updates/main amd64 openjdk-8-jre-headless amd64 8u91-b14-3ubuntu1~16.04.1 [26.9 MB] 
Fetched 75.7 MB in 12s (6,103 kB/s) 
Selecting previously unselected package liblcms2-2:amd64.
(Reading database ... 60851 files and directories currently installed.)
Preparing to unpack .../liblcms2-2_2.6-3ubuntu2_amd64.deb ...
Unpacking liblcms2-2:amd64 (2.6-3ubuntu2) ...
Selecting previously unselected package x11-common.
Preparing to unpack .../x11-common_1%3a7.7+13ubuntu3_all.deb ...
Unpacking x11-common (1:7.7+13ubuntu3) ...
Selecting previously unselected package libxtst6:amd64.
Preparing to unpack .../libxtst6_2%3a1.2.2-1_amd64.deb ...
Unpacking libxtst6:amd64 (2:1.2.2-1) ...
Selecting previously unselected package ca-certificates-java.
Preparing to unpack .../ca-certificates-java_20160321_all.deb ...
Unpacking ca-certificates-java (20160321) ...
Selecting previously unselected package java-common.
Preparing to unpack .../java-common_0.56ubuntu2_all.deb ...
Unpacking java-common (0.56ubuntu2) ...
Selecting previously unselected package libcups2:amd64.
Preparing to unpack .../libcups2_2.1.3-4_amd64.deb ...
Unpacking libcups2:amd64 (2.1.3-4) ...
Selecting previously unselected package fonts-dejavu-core.
Preparing to unpack .../fonts-dejavu-core_2.35-1_all.deb ...
Unpacking fonts-dejavu-core (2.35-1) ...
Selecting previously unselected package fontconfig-config.
Preparing to unpack .../fontconfig-config_2.11.94-0ubuntu1.1_all.deb ...
Unpacking fontconfig-config (2.11.94-0ubuntu1.1) ...
Selecting previously unselected package libfontconfig1:amd64.
Preparing to unpack .../libfontconfig1_2.11.94-0ubuntu1.1_amd64.deb ...
Unpacking libfontconfig1:amd64 (2.11.94-0ubuntu1.1) ...
Selecting previously unselected package libpcsclite1:amd64.
Preparing to unpack .../libpcsclite1_1.8.14-1ubuntu1_amd64.deb ...
Unpacking libpcsclite1:amd64 (1.8.14-1ubuntu1) ...
Selecting previously unselected package libxi6:amd64.
Preparing to unpack .../libxi6_2%3a1.7.6-1_amd64.deb ...
Unpacking libxi6:amd64 (2:1.7.6-1) ...
Selecting previously unselected package libxrender1:amd64.
Preparing to unpack .../libxrender1_1%3a0.9.9-0ubuntu1_amd64.deb ...
Unpacking libxrender1:amd64 (1:0.9.9-0ubuntu1) ...
Selecting previously unselected package openjdk-8-jre-headless:amd64.
Preparing to unpack .../openjdk-8-jre-headless_8u91-b14-3ubuntu1~16.04.1_amd64.deb ...
Unpacking openjdk-8-jre-headless:amd64 (8u91-b14-3ubuntu1~16.04.1) ...
Selecting previously unselected package puppet-agent.
Preparing to unpack .../puppet-agent_1.7.0-1xenial_amd64.deb ...
Unpacking puppet-agent (1.7.0-1xenial) ...
Selecting previously unselected package puppetserver.
Preparing to unpack .../puppetserver_2.6.0-1puppetlabs1_all.deb ...
Unpacking puppetserver (2.6.0-1puppetlabs1) ...
Processing triggers for systemd (229-4ubuntu7) ...
Processing triggers for ureadahead (0.100.0-19) ...
ureadahead will be reprofiled on next reboot
Processing triggers for man-db (2.7.5-1) ...
Processing triggers for ca-certificates (20160104ubuntu1) ...
Updating certificates in /etc/ssl/certs...
0 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
done.
Processing triggers for libc-bin (2.23-0ubuntu3) ...
Setting up liblcms2-2:amd64 (2.6-3ubuntu2) ...
Setting up x11-common (1:7.7+13ubuntu3) ...
update-rc.d: warning: start and stop actions are no longer supported; falling back to defaults
Setting up libxtst6:amd64 (2:1.2.2-1) ...
Setting up java-common (0.56ubuntu2) ...
Setting up libcups2:amd64 (2.1.3-4) ...
Setting up fonts-dejavu-core (2.35-1) ...
Setting up fontconfig-config (2.11.94-0ubuntu1.1) ...
Setting up libfontconfig1:amd64 (2.11.94-0ubuntu1.1) ...
Setting up libpcsclite1:amd64 (1.8.14-1ubuntu1) ...
Setting up libxi6:amd64 (2:1.7.6-1) ...
Setting up libxrender1:amd64 (1:0.9.9-0ubuntu1) ...
Setting up puppet-agent (1.7.0-1xenial) ...
Created symlink from /etc/systemd/system/multi-user.target.wants/puppet.service to /lib/systemd/system/puppet.service.
Created symlink from /etc/systemd/system/multi-user.target.wants/mcollective.service to /lib/systemd/system/mcollective.service.
Created symlink from /etc/systemd/system/multi-user.target.wants/pxp-agent.service to /lib/systemd/system/pxp-agent.service.
Removed symlink /etc/systemd/system/multi-user.target.wants/pxp-agent.service.
Setting up ca-certificates-java (20160321) ...
Adding debian:Go_Daddy_Root_Certificate_Authority_-_G2.pem
Adding debian:Certinomis_-_Autorité_Racine.pem
Adding debian:Verisign_Class_2_Public_Primary_Certification_Authority_-_G2.pem
Adding debian:TWCA_Global_Root_CA.pem
Adding debian:Network_Solutions_Certificate_Authority.pem
Adding debian:QuoVadis_Root_CA_2_G3.pem
Adding debian:TWCA_Root_Certification_Authority.pem
Adding debian:COMODO_Certification_Authority.pem
Adding debian:AffirmTrust_Networking.pem
Adding debian:Camerfirma_Global_Chambersign_Root.pem
Adding debian:Camerfirma_Chambers_of_Commerce_Root.pem
Adding debian:GeoTrust_Primary_Certification_Authority.pem
Adding debian:Comodo_Trusted_Services_root.pem
Adding debian:RSA_Security_2048_v3.pem
Adding debian:CNNIC_ROOT.pem
Adding debian:WoSign.pem
Adding debian:GeoTrust_Global_CA.pem
Adding debian:Hellenic_Academic_and_Research_Institutions_RootCA_2011.pem
Adding debian:Go_Daddy_Class_2_CA.pem
Adding debian:Comodo_AAA_Services_root.pem
Adding debian:AddTrust_Public_Services_Root.pem
Adding debian:VeriSign_Class_3_Public_Primary_Certification_Authority_-_G4.pem
Adding debian:Verisign_Class_3_Public_Primary_Certification_Authority_-_G2.pem
Adding debian:GlobalSign_Root_CA_-_R2.pem
Adding debian:Actalis_Authentication_Root_CA.pem
Adding debian:Chambers_of_Commerce_Root_-_2008.pem
Adding debian:Autoridad_de_Certificacion_Firmaprofesional_CIF_A62634068.pem
Adding debian:EE_Certification_Centre_Root_CA.pem
Adding debian:China_Internet_Network_Information_Center_EV_Certificates_Root.pem
Adding debian:Root_CA_Generalitat_Valenciana.pem
Adding debian:GeoTrust_Global_CA_2.pem
Adding debian:Entrust_Root_Certification_Authority_-_EC1.pem
Adding debian:DigiCert_Assured_ID_Root_G2.pem
Adding debian:PSCProcert.pem
Adding debian:Certum_Root_CA.pem
Adding debian:AffirmTrust_Commercial.pem
Adding debian:GeoTrust_Primary_Certification_Authority_-_G2.pem
Adding debian:Certigna.pem
Adding debian:DigiCert_Assured_ID_Root_G3.pem
Adding debian:Verisign_Class_1_Public_Primary_Certification_Authority_-_G3.pem
Adding debian:NetLock_Notary_=Class_A=_Root.pem
Adding debian:UTN_USERFirst_Email_Root_CA.pem
Adding debian:certSIGN_ROOT_CA.pem
Adding debian:COMODO_ECC_Certification_Authority.pem
Adding debian:DigiCert_Global_Root_G2.pem
Adding debian:EBG_Elektronik_Sertifika_Hizmet_Sağlayıcısı.pem
Adding debian:Equifax_Secure_Global_eBusiness_CA.pem
Adding debian:CFCA_EV_ROOT.pem
Adding debian:Certinomis_-_Root_CA.pem
Adding debian:T-TeleSec_GlobalRoot_Class_3.pem
Adding debian:AddTrust_Qualified_Certificates_Root.pem
Adding debian:Comodo_Secure_Services_root.pem
Adding debian:Verisign_Class_3_Public_Primary_Certification_Authority.pem
Adding debian:Staat_der_Nederlanden_EV_Root_CA.pem
Adding debian:AffirmTrust_Premium.pem
Adding debian:Starfield_Root_Certificate_Authority_-_G2.pem
Adding debian:Security_Communication_Root_CA.pem
Adding debian:StartCom_Certification_Authority.pem
Adding debian:StartCom_Certification_Authority_G2.pem
Adding debian:ACEDICOM_Root.pem
Adding debian:QuoVadis_Root_CA_1_G3.pem
Adding debian:QuoVadis_Root_CA_3_G3.pem
Adding debian:DST_ACES_CA_X6.pem
Adding debian:thawte_Primary_Root_CA_-_G2.pem
Adding debian:Verisign_Class_1_Public_Primary_Certification_Authority_-_G2.pem
Adding debian:Verisign_Class_2_Public_Primary_Certification_Authority_-_G3.pem
Adding debian:Equifax_Secure_eBusiness_CA_1.pem
Adding debian:TÜRKTRUST_Elektronik_Sertifika_Hizmet_Sağlayıcısı_H5.pem
Adding debian:ApplicationCA_-_Japanese_Government.pem
Adding debian:NetLock_Express_=Class_C=_Root.pem
Adding debian:COMODO_RSA_Certification_Authority.pem
Adding debian:SwissSign_Platinum_CA_-_G2.pem
Adding debian:Starfield_Services_Root_Certificate_Authority_-_G2.pem
Adding debian:SecureSign_RootCA11.pem
Adding debian:TURKTRUST_Certificate_Services_Provider_Root_2007.pem
Adding debian:USERTrust_ECC_Certification_Authority.pem
Adding debian:Entrust.net_Premium_2048_Secure_Server_CA.pem
Adding debian:CA_Disig_Root_R1.pem
Adding debian:Staat_der_Nederlanden_Root_CA_-_G3.pem
Adding debian:Verisign_Class_3_Public_Primary_Certification_Authority_2.pem
Adding debian:WellsSecure_Public_Root_Certificate_Authority.pem
Adding debian:GeoTrust_Primary_Certification_Authority_-_G3.pem
Adding debian:CA_Disig_Root_R2.pem
Adding debian:ACCVRAIZ1.pem
Adding debian:NetLock_Arany_=Class_Gold=_Főtanúsítvány.pem
Adding debian:Buypass_Class_2_CA_1.pem
Adding debian:GlobalSign_ECC_Root_CA_-_R4.pem
Adding debian:thawte_Primary_Root_CA_-_G3.pem
Adding debian:Visa_eCommerce_Root.pem
Adding debian:ePKI_Root_Certification_Authority.pem
Adding debian:S-TRUST_Universal_Root_CA.pem
Adding debian:QuoVadis_Root_CA.pem
Adding debian:IdenTrust_Commercial_Root_CA_1.pem
Adding debian:TC_TrustCenter_Class_3_CA_II.pem
Adding debian:Hongkong_Post_Root_CA_1.pem
Adding debian:AddTrust_Low-Value_Services_Root.pem
Adding debian:Entrust_Root_Certification_Authority_-_G2.pem
Adding debian:NetLock_Business_=Class_B=_Root.pem
Adding debian:Verisign_Class_3_Public_Primary_Certification_Authority_-_G3.pem
Adding debian:AC_Raíz_Certicámara_S.A..pem
Adding debian:Starfield_Class_2_CA.pem
Adding debian:GlobalSign_ECC_Root_CA_-_R5.pem
Adding debian:WoSign_China.pem
Adding debian:Baltimore_CyberTrust_Root.pem
Adding debian:GeoTrust_Universal_CA_2.pem
Adding debian:Certification_Authority_of_WoSign_G2.pem
Adding debian:ComSign_CA.pem
Adding debian:S-TRUST_Authentication_and_Encryption_Root_CA_2005_PN.pem
Adding debian:Security_Communication_EV_RootCA1.pem
Adding debian:EC-ACC.pem
Adding debian:VeriSign_Class_3_Public_Primary_Certification_Authority_-_G5.pem
Adding debian:VeriSign_Universal_Root_Certification_Authority.pem
Adding debian:T-TeleSec_GlobalRoot_Class_2.pem
Adding debian:DigiCert_Global_Root_G3.pem
Adding debian:Staat_der_Nederlanden_Root_CA_-_G2.pem
Adding debian:Staat_der_Nederlanden_Root_CA.pem
Adding debian:USERTrust_RSA_Certification_Authority.pem
Adding debian:TeliaSonera_Root_CA_v1.pem
Adding debian:QuoVadis_Root_CA_2.pem
Adding debian:Deutsche_Telekom_Root_CA_2.pem
Adding debian:SwissSign_Silver_CA_-_G2.pem
Adding debian:Secure_Global_CA.pem
Adding debian:DigiCert_Global_Root_CA.pem
Adding debian:GeoTrust_Universal_CA.pem
Adding debian:CA_Disig.pem
Adding debian:TÜBİTAK_UEKAE_Kök_Sertifika_Hizmet_Sağlayıcısı_-_Sürüm_3.pem
Adding debian:Swisscom_Root_CA_1.pem
Adding debian:Trustis_FPS_Root_CA.pem
Adding debian:DigiCert_Assured_ID_Root_CA.pem
Adding debian:Entrust_Root_Certification_Authority.pem
Adding debian:E-Tugra_Certification_Authority.pem
Adding debian:OISTE_WISeKey_Global_Root_GA_CA.pem
Adding debian:Izenpe.com.pem
Adding debian:SecureTrust_CA.pem
Adding debian:Certplus_Class_2_Primary_CA.pem
Adding debian:DST_Root_CA_X3.pem
Adding debian:D-TRUST_Root_Class_3_CA_2_2009.pem
Adding debian:Certum_Trusted_Network_CA.pem
Adding debian:GlobalSign_Root_CA_-_R3.pem
Adding debian:D-TRUST_Root_Class_3_CA_2_EV_2009.pem
Adding debian:Taiwan_GRCA.pem
Adding debian:Verisign_Class_1_Public_Primary_Certification_Authority.pem
Adding debian:UTN_USERFirst_Hardware_Root_CA.pem
Adding debian:Microsec_e-Szigno_Root_CA.pem
Adding debian:Buypass_Class_3_Root_CA.pem
Adding debian:AddTrust_External_Root.pem
Adding debian:Sonera_Class_1_Root_CA.pem
Adding debian:Atos_TrustedRoot_2011.pem
Adding debian:XRamp_Global_CA_Root.pem
Adding debian:Swisscom_Root_EV_CA_2.pem
Adding debian:DigiCert_Trusted_Root_G4.pem
Adding debian:thawte_Primary_Root_CA.pem
Adding debian:Sonera_Class_2_Root_CA.pem
Adding debian:SwissSign_Gold_CA_-_G2.pem
Adding debian:Equifax_Secure_CA.pem
Adding debian:CA_WoSign_ECC_Root.pem
Adding debian:Buypass_Class_2_Root_CA.pem
Adding debian:TÜRKTRUST_Elektronik_Sertifika_Hizmet_Sağlayıcısı_H6.pem
Adding debian:Global_Chambersign_Root_-_2008.pem
Adding debian:IdenTrust_Public_Sector_Root_CA_1.pem
Adding debian:IGC_A.pem
Adding debian:StartCom_Certification_Authority_2.pem
Adding debian:OISTE_WISeKey_Global_Root_GB_CA.pem
Adding debian:Microsec_e-Szigno_Root_CA_2009.pem
Adding debian:QuoVadis_Root_CA_3.pem
Adding debian:Juur-SK.pem
Adding debian:Cybertrust_Global_Root.pem
Adding debian:AffirmTrust_Premium_ECC.pem
Adding debian:NetLock_Qualified_=Class_QA=_Root.pem
Adding debian:GlobalSign_Root_CA.pem
Adding debian:DigiCert_High_Assurance_EV_Root_CA.pem
Adding debian:Security_Communication_RootCA2.pem
Adding debian:Swisscom_Root_CA_2.pem
done.
Processing triggers for ca-certificates (20160104ubuntu1) ...
Updating certificates in /etc/ssl/certs...
0 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...

done.
done.
Setting up openjdk-8-jre-headless:amd64 (8u91-b14-3ubuntu1~16.04.1) ...
update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/rmid to provide /usr/bin/rmid (rmid) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java to provide /usr/bin/java (java) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/keytool to provide /usr/bin/keytool (keytool) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/jjs to provide /usr/bin/jjs (jjs) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/pack200 to provide /usr/bin/pack200 (pack200) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/rmiregistry to provide /usr/bin/rmiregistry (rmiregistry) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/unpack200 to provide /usr/bin/unpack200 (unpack200) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/orbd to provide /usr/bin/orbd (orbd) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/servertool to provide /usr/bin/servertool (servertool) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/tnameserv to provide /usr/bin/tnameserv (tnameserv) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/jexec to provide /usr/bin/jexec (jexec) in auto mode
Setting up puppetserver (2.6.0-1puppetlabs1) ...
usermod: no changes
Processing triggers for libc-bin (2.23-0ubuntu3) ...
Processing triggers for systemd (229-4ubuntu7) ...
Processing triggers for ureadahead (0.100.0-19) ...
surro@oldhorse:~$

Flawless so far.

Starting the service:

surro@oldhorse:~$ service puppetserver start
 ==== AUTHENTICATING FOR org.freedesktop.systemd1.manage-units ===
 Authentication is required to start 'puppetserver.service'.
 Authenticating as: surro,,, (surro)
 Password:
 ==== AUTHENTICATION COMPLETE ===

Check out that hilarious typo...

Now for PuppetDB.

The PuppetDB doc for people just getting started and are using the packages is here:

https://docs.puppet.com/puppetdb/4.2/install_from_packages.html

Well, as far as I can tell, that one just sends you in a documentation link loop.  The actual docs are here:

https://docs.puppet.com/puppetdb/4.2/install_via_module.html

Man, I seriously hate bad documentation.  This is where they kind of leave you on your own.  Next you need to install the puppetdb and puppetdb-termini packages:

surro@oldhorse:~$ sudo apt-get install puppetdb puppetdb-termini
[sudo] password for surro: 
Reading package lists... Done
Building dependency tree 
Reading state information... Done
The following NEW packages will be installed:
 puppetdb puppetdb-termini
0 upgraded, 2 newly installed, 0 to remove and 62 not upgraded.
Need to get 21.2 MB of archives.
After this operation, 25.6 MB of additional disk space will be used.
Get:1 https://apt.puppetlabs.com xenial/PC1 amd64 puppetdb all 4.2.2-1puppetlabs1 [21.2 MB]
Get:2 https://apt.puppetlabs.com xenial/PC1 amd64 puppetdb-termini all 4.2.2-1puppetlabs1 [18.5 kB]
Fetched 21.2 MB in 3s (6,309 kB/s) 
Selecting previously unselected package puppetdb.
(Reading database ... 67388 files and directories currently installed.)
Preparing to unpack .../puppetdb_4.2.2-1puppetlabs1_all.deb ...
Unpacking puppetdb (4.2.2-1puppetlabs1) ...
Selecting previously unselected package puppetdb-termini.
Preparing to unpack .../puppetdb-termini_4.2.2-1puppetlabs1_all.deb ...
Unpacking puppetdb-termini (4.2.2-1puppetlabs1) ...
Setting up puppetdb (4.2.2-1puppetlabs1) ...
Config archive not found. Not proceeding with migration
PEM files in /etc/puppetlabs/puppetdb/ssl are missing, we will move them into place for you
Copying files: /etc/puppetlabs/puppet/ssl/certs/ca.pem, /etc/puppetlabs/puppet/ssl/private_keys/oldhorse.surroindustries.com.pem and /etc/puppetlabs/puppet/ssl/certs/oldhorse.surroindustries.com.pem to /etc/puppetlabs/puppetdb/ssl
Backing up /etc/puppetlabs/puppetdb/conf.d/jetty.ini to /etc/puppetlabs/puppetdb/conf.d/jetty.ini.bak.1474875662 before making changes
Updated default settings from package installation for ssl-host in /etc/puppetlabs/puppetdb/conf.d/jetty.ini.
Updated default settings from package installation for ssl-port in /etc/puppetlabs/puppetdb/conf.d/jetty.ini.
Updated default settings from package installation for ssl-key in /etc/puppetlabs/puppetdb/conf.d/jetty.ini.
Updated default settings from package installation for ssl-cert in /etc/puppetlabs/puppetdb/conf.d/jetty.ini.
Updated default settings from package installation for ssl-ca-cert in /etc/puppetlabs/puppetdb/conf.d/jetty.ini.
Setting up puppetdb-termini (4.2.2-1puppetlabs1) ...
surro@oldhorse:~$

From here, they barge deep into puppet concept with no explanation talking about assigning classes to nodes.  What a clusterfuck.

Ok, in the next step on the install from package branch of the documentation, they want a puppet agent installed and ready to receive a certificate for the node.  Since this puppet master will also be a node that’s fine.  This puppetmaster will prior to that need verified to be able to issue the correct certs since it was my understanding it will need to act as a certificate CA.

So skipping that requirement for now, I’ll need to install the puppet agent (the flow is a little crazy here).

On to the puppet-agent installation at:

https://docs.puppet.com/puppet/latest/reference/install_linux.html

Seriously, do these guys even use their docs?

Skipping page, after page, after page of steps for other distros that should have each been branched into their own workflow on separate pages, you’ll see the next command is sudo apt-get install puppet-agent:

surro@oldhorse:~$ sudo apt-get install puppet-agent
[sudo] password for surro: 
Reading package lists... Done
Building dependency tree 
Reading state information... Done
puppet-agent is already the newest version (1.7.0-1xenial).
puppet-agent set to manually installed.
0 upgraded, 0 newly installed, 0 to remove and 62 not upgraded.

Ok, looks like this is already installed as a prerequisite to one of the earlier packages, good, the package tree appears sane so far.

Next, we’re going to do some config, by setting the agent to recognize the puppetmaster as the master in the agent’s config.  They reference a configuration change but don’t say which file that is.  Maddening:

root@oldhorse:/etc# ls -1 /etc/ | grep puppet
puppetlabs
root@oldhorse:/etc#

Ok, we’re getting lucky.  There’s only one puppet folder this time.  Let’s see what we’re dealing with:

root@oldhorse:/etc/puppetlabs# ls -1
code
mcollective
puppet
puppetdb
puppetserver
pxp-agent
root@oldhorse:/etc/puppetlabs#

Jackpot.  There’s a code/, mcollective/, puppet/, puppetdb/, puppetserver/,and a pxp-agent/.

So, there’s puppet/ or pxp-agent/, either of which could be the agent service, with puppet/ by naming convention or pxp-agent by the explicitness of the word ‘agent’.  Glad the documentation is clear on this when they completely did not specify in the guide, leaving users to wade through layers of bullshit.

Someone on IRC has suggested we go back to the instructions for installing puppetdb via module:

https://docs.puppet.com/puppetdb/4.2/install_via_module.html

Since he is suggesting we install via module, and we’ve already installed via package, for whatever reason, we’ll need to uninstall the puppetdb package.  Yikesm this is messy and I’m feeling pretty dumb with it.

root@oldhorse:/etc/puppetlabs# apt-get remove puppetdb
Reading package lists... Done
Building dependency tree 
Reading state information... Done
The following packages will be REMOVED:
 puppetdb
0 upgraded, 0 newly installed, 1 to remove and 62 not upgraded.
After this operation, 25.5 MB disk space will be freed.
Do you want to continue? [Y/n] y
(Reading database ... 67543 files and directories currently installed.)
Removing puppetdb (4.2.2-1puppetlabs1) ...
root@oldhorse:/etc/puppetlabs# ls
code mcollective puppet puppetdb puppetserver pxp-agent
root@oldhorse:/etc/puppetlabs# rm -rf puppetdb
root@oldhorse:/etc/puppetlabs#

Package removed.  Config directory removed at the user’s suggestion.

So, going back to the install via module guide, it……tells you to enable the….collection package repository….and or…..grab the packages.

Step 2.  Assign classes to nodes.

Wait, what?  It didn’t tell you how to install by module.  This is configuration of an already installed puppetdb it looks like.  What am I even looking at?

Ah ha!  The instructions were buried on a different subdomain as a passing reference and the contained instructions on the current page were not actually related to installing puppetdb.  The real instructions are here:

https://forge.puppet.com/puppetlabs/puppetdb

puppet module install puppetlabs-puppetdb

Jackpot.

root@oldhorse:/etc/puppetlabs# puppet module install puppetlabs-puppetdb
Notice: Preparing to install into /etc/puppetlabs/code/environments/production/modules ...
Notice: Downloading from https://forgeapi.puppetlabs.com ...
Notice: Installing -- do not interrupt ...
/etc/puppetlabs/code/environments/production/modules
└─┬ puppetlabs-puppetdb (v5.1.2)
 ├── puppetlabs-firewall (v1.8.1)
 ├── puppetlabs-inifile (v1.6.0)
 └─┬ puppetlabs-postgresql (v4.8.0)
 ├── puppetlabs-apt (v2.3.0)
 ├── puppetlabs-concat (v2.2.0)
 └── puppetlabs-stdlib (v4.12.0)
root@oldhorse:/etc/puppetlabs#

Alright, the nightmare might be almost over.

So, from here, we’re at a crossroads.  You have to know all about puppet to even install this component.

The docs wants you to set up a manifest, somewhere, in some unspecified file with absolutely no link explaining what they’re talking about.

IRC says you need to download the VM at learn.puppet.com and go through the katas before actually installing it.

Screw it, it’s 5 AM and I have to work in a few hours.  I can’t believe this is now going to span over into a 3rd weekend.