Upgrading Qubes to latest fedora 29 template on an old (4.0), but not turned on in months laptop

I had qubes-os 4.0 on a laptop I hadn't turned on in months. It was still running fedora-26.

I wanted to update directly to the latest template (fedora-29). A direct in-place upgrade (26->29) isn't recommended by fedora, but totally worked and even says it should work in the Qubes upgrade docs.

Summary

 

I only did two things different than the stock standard directions in these three docs:
https://www.qubes-os.org/doc/templates/
https://www.qubes-os.org/doc/template/fedora/upgrade-28-to-29/
https://www.qubes-os.org/doc/disposablevm-customization/

1. The following command errored, saying I was missing rpmfusion gpg keys. I added them successfully and the upgrade completed.
   sudo dnf --releasever=29 --best --allowerasing distro-sync
2. I am using two template vms so I have a more-trusted standard fedora-29 vm with less installed software exposed in my sys-* vms.

The Qubes documentation is excellent and I saw even more this time around re-reading them while making this blog post. If you have more questions, the mailing list is very helpful and searchable here: https://groups.google.com/forum/#!forum/qubes-users

The purpose of this blog post is to document my step by step process for this upgrade so I can use it as a reference and others can learn what I learned. I'm sure some things can be more efficient and automated, but I'm first learning the GUIs provided. I'm not opposed to cli, just trying to understand the different mangement features provided by qubes.

Upgrading

I wanted to try in this less stressful environment, so I chose this laptop, since I don't need it for my day-to-day work and I wanted to compare and contrast in-place upgrades with downloading a brand new template.

First, I upgraded dom0

> sudo qubes-dom0-upgrade

This got me to: Kernel 4.14.74, xen 4.8.4, and still Qubes 4.0.

I then rebooted because I like rebooting a lot to make sure I am exactly where I expect to be, nothing pending.

I installed fedora-29 two different ways

 

Method 1: Install the new template:

I followed directions here: https://www.qubes-os.org/doc/templates/
> sudo qubes-dom0-update qubes-template-fedora-29

This was super easy. I tested by creating and launching a vm from this template using the gui "Create Qubes vm" launcher item and manually choosing the "fedora-29" VM template. Success. I updated it internally by booting the terminal in the qubesvm and running "sudo dnf update".

Note: I just realized I don't know if I type update or upgrade usually, and it turns out that it's the same command, but update is a deprecated alias for upgrad according to "man dnf".

Method 2: I upgraded my template VM for fedora-26 in-place

I used the same instructions here: https://www.qubes-os.org/doc/template/fedora/upgrade-28-to-29/

Notes and one error I encountered and fixed in my process:

I named my template fedora-29-1 because I didn't want the simultaneous download of the fedora-29 template that dom0 was doing to have any naming issues/errors.
I encountered an error upgrading that stated I didn't have fedora29 rpmfusion free and nonfree signing keys imported. I checked /etc/pki/rpm-gpg and sure enough, they were missing, it automatically added fedora 25-28, but ignored fedora29, strangely enough.

I imported the keys with the following commands:

• [user@fedora-29-1 rpm-gpg]$ sudo rpm --import 'https://rpmfusion.org/keys?action=AttachFile&do=get&target=RPM-GPG-KEY-rpmfusion-free-fedora-29'
• [user@fedora-29-1 rpm-gpg]$ sudo rpm --import 'https://rpmfusion.org/keys?action=AttachFile&do=get&target=RPM-GPG-KEY-rpmfusion-nonfree-fedora-29'

As mentioned might be a necessity in the link, I did need to add disk space after downloading all the files. I got this message after restarting my template upgrade command:

Error Summary
-------------
Disk Requirements:
At least 904MB more space needed on the / filesystem.

I tried copying over the cache folder so I wouldn't have to waste another 20+ minutes waiting for 1.4GB of files to download, but I was unable to get the cache to copy. If someone knows how to copy the cache with a lot of cached downloads, let me know. ** I recommend adding disk space pre-emptively, because it if you've installed much, the stock VM probably doesn't have space for the in-place upgrade. ** It takes an extra 30 seconds up front and saves 10-30 minutes later.

Summary: The documentation is great for this. I recommend preemptively adding the disk-cache. Then import the rpm-fusion keys before upgrading.

So far we haven't changed anything that we're relying on. I love the segmentation of Qubes that allows us to install a new template and upgrade a cloned template in-place, and have nothing I was relying on risked because of this. The segmentation is awesome. Now we switch VMs to the new templates

Qubes Template Manager is AWESOME 

 

Q icon> System Tools > Qubes Template Manager
https://www.qubes-os.org/doc/templates/#how-to-switch-templates-40
It makes switching to a different/new template effortless.
I did have to shutdown some VMs in order to switch them over.
I also rebooted a couple times because I wanted to make sure everything was working properly.

My don't-put-all-my-eggs-in-one-basket sequence that worked:  

First: A spot check

I spot-checked both templates by creating a new-vm from those templates and tested out network connectivity by opening firefox to a new page. I upgrade the templates a well and tested out VLC in the 29-1 template since it should still be installed. All worked. Great.

Second: Move over easy VMs

• Spot check one VM that has persistence. I use a VM that had some files in Downloads that were still there in the migrated AppVM with the new Fedora-29-1 template and I confirmed in the CLI that cat /etc/fedora-release as 29.
• Do the rest of the vms besides sys-* and disposible-vms.
• Reboot

Third: Move sys-* after a reboot

• I'm cautions/still learning how restart services and had trouble letting Qubes Manager let me kill sys-network. I decided to reboot and then update the sys-* vms.
• I booted, shut down anything that had automatically started, and then I was able to shutdown/kill the sys-* vms to update their template.
• I rebooted again to get the full start scripts running in the right order because that seemed like less work than doing it manually and this fells like a fuller-test if they work.

Disposable VMs need a bit extra work:

This link shows the few commands for updating and includes deletion.
https://www.qubes-os.org/doc/disposablevm-customization/

I consciously chose fedora-29-1 for the dvm-29-1 and not the clean, fedora-29 vm because I wanted VLC and a few other disposable applications that I don't want in my sys-vms.

I did end up with two launcher items/VMs for different disposable VMs. I deleted the old one following directions in the link above.

Notes on what I did differently than stock-standard:

I'm running two templates, the upgraded-in-place one (fedora-29-1) and the downloaded fedora-29 template. Anything involving sys-* or display-mgmt vms I set to fedora-29. For things that need installed software, I used fedora-29-1.
Eventually I might get minimal and/or disposible VMs working for these. I wasn't able to get a minimal VM working for network or firewall on my latest attempt. I haven't given up yet though.

I set personal use (work/personal/untrusted/etc) vms to fedora-29-1 because that has extra packages installed that I want to stay around.

Brain dumping multiple subjects for future reference

Good write up about cookie security. One of the clearest I've seen on thevarious cookie flags.: https://www.wst.space/cookies-samesite-secure-httponly/

Ideas for startup administration: 

• OSX admin: Fleetsmith
• Logging/infra analytics: OSquery/Kolide
• Password storage and sharing: Lastpass
• Email: Gmail (preferrable ot office365, since the lowest versions of o365 don't allow 2FA and fleetsmith/kolide prefer gmail integration for auth)
• 2FA solution (because google-authenticator doesn't allow backups:
• Initially (free): Authy
• Later on, for all their integrations and yubikey/ease of use: DUO

Goals for security admins:

• Involve users minimally in setup and configuration of security products.
• Make sec infra as invisible as possible.
• Don't require 100 different passwords. Try to limit it to 3ish. 
• Log from the get-go: https://medium.com/starting-up-security/starting-up-security-87839ab21bae

In general, this writeup is way better than I could have done it: https://medium.com/starting-up-security/starting-up-security-87839ab21bae

Everything Ryan writes here is pretty great: https://magoo.github.io/Blockchain-Graveyard/advice/

EFI/UEFI and OSX link dump

Firmware/EFI and Kernel/OS level issues link dump:

long running bug (fixed?)
http://newosxbook.com/articles/PST2.html

twitter account for osx/efi security person: https://twitter.com/osxreverser

OSX kernel mem leak: https://github.com/ud2/advisories/tree/master/osx/cve-2015-3780
malicious ntfs image dos: https://github.com/ud2/advisories/tree/master/osx/cve-2015-5763

read osx physical memory, including bios, kernel, and possibly smm: https://github.com/gdbinit/readphysmem
exploit using this info: https://github.com/gdbinit/diagnostic_service2

info aobut twpn: http://blog.qwertyoruiop.com/?p=69
twpn: https://github.com/kpwn/tpwn

OSX rootkit detection and prevention:

Rootpipe: https://twitter.com/diodesign/status/630553369078149120
http://www.opensource.apple.com/source/gdb/gdb-1705/src/gdb/macosx/macosx-nat-dyld.c
https://media.defcon.org/DEF%20CON%2023/DEF%20CON%2023%20presentations/Patrick%20Wardle/
updated preso from defcon: https://media.defcon.org/DEF%20CON%2023/DEF%20CON%2023%20presentations/Patrick%20Wardle%20-%20UPDATED/

thunderstrike 2: https://trmm.net/Thunderstrike2_details
http://legbacore.com/Research_files/ts2-blackhat.pdf

somehting about pwning ios devices, firmware, something something:
https://github.com/planetbeing/xpwn
lightweight version: https://github.com/sektioneins/xpwntool-lite

tons of good resources on this site: https://reverse.put.as/2015/08/07/writing-bad-lamware-for-os-x/
their github: https://github.com/Gdbinit/
https://reverse.put.as/archives/


EFI:
EDK II visual studio stuff https://github.com/ionescu007/VisualUefi
thread about said tool: https://twitter.com/aionescu/status/632594173414129664


https://github.com/LongSoft/UEFITool

UEFITool is a cross-platform C++/Qt program for parsing, extracting and modifying UEFI firmware images.

It supports parsing of full BIOS images starting with the flash descriptor or any binary files containing UEFI volumes.

hypervisor stuff:

https://twitter.com/adulau/status/631366664265842688 - small enough to bypass and interfere with patchguard https://github.com/tandasat/Sushi

trustzone fuzzer: https://github.com/laginimaineb/fuzz_zone

bootloader walkthrough: http://www.cs.dartmouth.edu/~bx/blog/2015/09/03/a-toure-of-bootloading.html

Updating the bios on a Lenovo Thinkpad T440s

I like not having to worry about web pages flipping adjacent bits in memory (http://googleprojectzero.blogspot.com/2015/03/exploiting-dram-rowhammer-bug-to-gain.html), so I wanted to update my bios. I didn't want to use the Windows update utility (not running windows) or Lenovo crapware system update center that's on windows, so I went the route of the bootable USB.

From a Ubuntu system running FDE on a Lenovo Thinkpad 440s, I did the following:
Downloaded this: http://support.lenovo.com/us/en/downloads/ds035967
More or less followed directions here: http://www.lenzg.net/archives/358-Updating-the-BIOS-on-my-ThinkPad-T440-without-Windows-or-a-DVD-Drive.html

If you do have windows and don't want to shut down, here's how to query your Lenovo Bios version number using powershell (should work on any version of powershell, definitely 2.0+):
(Get-WmiObject -class "Win32_bios" -namespace "root\CIMV2" -computername ".").SMBIOSBIOSVersion

Here's how to query if you're on ubuntu (and probably other linuxes):
$ sudo dmidecode -s bios-version

Unlike the tutorial linked above, my steps for creating the bootable USB are a little simpler, mostly because the tool, "geteltorito" is installed by default in Ubuntu.

How to update:
Step 1: Create the bootable USB:
**Note, this assumes your usb is /dev/sdb1

• Download the iso here: http://support.lenovo.com/us/en/downloads/ds035967
• Run the following 2 commands:

$ geteltorito -o bios.img gjuj13us.iso
$ sudo dd if=bios.img of=/dev/sdb1

Done, USB should now be bootable.

Updating the Bios with the bootable USB
Then I shutdown, crossed my fingers, and booted from the USB (f12 brings up the boot media selection menu while booting if you haven't disabled that in Bios).

If you don't care about details and just want to finish, select #2, next, Y, yes, yea, sure, do the thing, reboot without removing media, wait for flashing to happen, remove bootable usb, reboot, done.

Detailed version:

When the usb loads, I have three options:

1. Read this first. This actually is pretty darn confusing and didn't really make sense the instructions are in Engrish, or were written by a drunk person.

2. "update system program". This does what I'm here for.
<Y>, yes, I want to continue

Don't remove USB, press <Enter>

System reboots into USB again

Flashing takes about a minute or two, then reboots. I removed the flash drive while it was counting down and it rebooted happily into my disk encryption prompt, then into my OS.

Option 3 was to manually change your serial number. I didn't have a need for this as far as I'm aware. I kind of want to make it "SKYNET" though.

Next
I'm going to enable boot protection, TXTmode, and security auditing on Bios and other strings and then will post an update on updating the bios when all that complication is added to the process. My next bios update should be considerably more complicated.

Let me know if anyone had issues with this method. Your bios image may be different than mine if your computer is not the T440s.

What do I do when I get a new server? A bare-bones simple guide to starting up a secure-enough server.

Lets say I spin up an instance in AWS, or Linode, or install a new OS in a VM at home to be a server of...anything. Each time it's a little bit different, but generally its Linux, although much of this applies to BSDs as well. This is assuming you know how to connect to your machine and have setup a non-root user and strong password for that account.

Step 1
If I can, encrypt the disk. On AWS this is already the case and they have solid disk-erasing as far as I know (TODO: link to the blog/whitepaper where researchers found a bug and amazon fixed it platform-wide afaik). On Ubuntu this is baked into the install script, which you'll probably have to run yourself since most automated/one-click/prepackaged ones don't do it for you.

Step 2
See what's running on the system and listening on the network and kill anything unnecessary. I run ps -ef (process list), top/htop (resource usage), and netstat -ant (network listeners) to do this. FTP doesn't need to be on. It is negated in necessity by the next step.

Step 3
UPGRADE ALL THE THINGS!! This should already be done, but sudo apt-get update && sudo apt-get upgrade -y will be sure you have everything on Ubuntu.

Step 4
Once done installing, use SSH as the sole means of communication and disable root logins.
4.1 Disabling root logins:
4.2 Use ssh-copy-id to get your key trusted by the server.
4.3 Test that sftp is configured. This way you have both SCP and SFTP as options for moving large numbers of bits to/from your server.

Step 5
Install fail2ban: http://www.fail2ban.org/wiki/index.php/Main_Page
-- I'm pretty sure this is in Ubuntu and many other Linux package repos by default, so sudo apt-get install fail2ban should work.
5.1 I up my failed connections form a single IP to 6 from the default of three because sometimes I make bad mistakes.

Step 6
Increase auditing from Ubuntu's default of holding it monthly to hold a year's worth of audit logs on logins and other helpful stuff.

Step 7
After I'm happy with the server's setup I install google-authenticator for 2 factor authentication by following the short steps I detailed in a previous post here: http://fiascoaverted.blogspot.com/2014/03/configuring-two-factor-authentication.html

At this point all you should only need TCP port 22 listening on the network. If you want to keep your log attempts quieter, you can change SSH to listen to an alternate port, like 2222, but this is security through obscurity and only possibly protects you from major stupidity.

From here I install whatever the server needs. This post was how to get me to a clean, safe state, from here I start adding/enabling more stuff until the server is ready.

NFSShell

How to build NFSShell on ubuntu:

wget http://www.cs.vu.nl/pub/leendert/nfsshell.tar.gz
cd nfs/

comment out lines 25-28 #this disables solaris specific compilation
uncomment lines 36-38 #this makes it work on linux
#not sure if the following command is necessary:
sudo apt-get install libtirpc-dev libncurses-dev

make

./nfs

The line numbers and stuff should Just Work(tm) since this is based off of the nfsshell from 1998.
Most that say they add nfsv3 support are broken and break some nfsv2 according to a friend, so they are not included here.

I should fix this up with a script later. And will add it to my github.

Configuring two factor authentication using google-auth for SSH on Ubuntu 13.10

This is really simple, shouldn't take more than 2 minutes and only has 5 steps (read all of this):
Mostly taken and condensed from: http://www.howtogeek.com/121650/how-to-secure-ssh-with-google-authenticators-two-factor-authentication/

1. Install the package: sudo apt-get install libpam-google-authenticator
2. Edit /etc/pam.d/sshd to have the line: auth required pam_google_authenticator.so
3. Edit /etc/ssh/sshd_config to have the line: ChallengeResponseAuthentication yes
4. Make sure you're no longer sudo'd or running as root and that your're whatever user you want to log in as and run google-authenticator from your home directory. Be sure to take down the secret, scratch codes, and/or the QR code.
5. sudo service ssh restart

Now you should be good to go.

In Ubuntu 13.10 for step 2, there will not be that line already in the file.

For step 3, ChallengeResponseAuthentication will be set to 'no'. Just change it to 'yes'.

No promises for other linux distros, but the above should work. If it doesn't you'll have to grab source and compile from the google-authenticator project on google-code.

Note: if you don't even want to have to enter the password (seriously stupid, but could be fun for a CTF or for learning purposes) comment out the following line in your /etc/pam.d/sshd file: @include common-auth

This was the first line of code in my sshd file and the helpful comments explain that it controls "Standard Un*x authentication". Fun stuff. For the love of all that is bacon and blue skies, don't do this in prod unless you're using a different auth mechanism in addition to 2FA.


Next up, google-auth for open-vpn.