Track 2 Workshop PacNOG 7 June 30, 2010 Securing Hosts -------------- 1 What's running? What's running and listing to the network? ---------------------------------------------------------- First you can see what is running on your machine by typing something like: $ sudo ps auxwww You will see lots and lots of stuff go by. So, let's look at this a bit more closely: $ sudo ps auxwww | less or $ sudo ps auxwww | more Now, browsing through all this we can see there are a bunch of initial system processes that start to support our hardware (items in "[ ]") as well as lots of processes associated with the Gnome Display Manager (gdm and gnome). Let's filter all of this out and see what we are left with: $ sudo ps auxwww | grep -v "\[" | grep -v gdm | grep -v gnome [Hint! You might want to copy and paste this in to a command window] What's left? Have a look and see if you can identify everything in the remaining list. Your list of processes should look something like: USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 1 0.0 0.0 2644 1552 ? Ss Jun29 0:01 /sbin/init root 429 0.0 0.0 2152 756 ? S Jun29 0:00 upstart-udev-bridge --daemon root 431 0.0 0.0 2624 1020 ? Snoc.pacnog.bluesky.as:34634 (ESTABLISHED) Again, Google and man to figure out what is going on: $ man sshd $ man cupsd $ man munin-node What's munin? Can you figure it out? What about cupsd. is this necessary during our workshop? Notice that ssh and minion are listening to all incoming connection requests (the "*"). This is a typical, potential security hole. In our case, we will leave ssh and munin up, but we are aware they are running and need to be patched for security updates as they come out. Earlier you locked down ssh a bit by not allowing the root user to log in with a passwords. This is important. As you are not printing from your desktops, let's turn off the cups printing service. Do you remember how to do this? $ cd /etc/init.d $ sudo cups stop $ lsof -i Now we only see: COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME sshd 1005 root 3u IPv4 5150 0t0 TCP *:ssh (LISTEN) sshd 1005 root 4u IPv6 5152 0t0 TCP *:ssh (LISTEN) munin-nod 1170 root 5u IPv4 5478 0t0 TCP *:munin (LISTEN) sshd 10340 root 3r IPv4 18747 0t0 TCP pc4.pacnog.bluesky.as:ssh->noc.pacnog.bluesky.as:34634 (ESTABLISHED) To make this permanent do: $ sudo update-rc.d cups remove 2. Install and configure an Intrusion Detection System (IDS) --------------------------------------------------------- Let's install the fcheck. This is intrusion detection software that is very simple to set up and is preconfigured to do most of what you want: $ sudo apt-get install fcheck This installs the Postfix Mail Transfer Agent (MTA). We already installed this earlier, but if you are asked to configure Postfix, then the steps are: * Choose to the first dialog * Select "Internet Site" and then * The FQDN selected should be fine (pcX.pacnog.bluesky.as), then choose You'll see check building its checksum database (to be explained a bit later). Once this is done you can look around to see how fcheck is configured. By default Ubuntu installs and configures check in a reasonable manner and you probably don't need to do anything else. $ man fcheck Configuration of check is in /etc/fcheck/fcheck.cfg. Let's have a look: $ vi /etc/fcheck/fcheck.cfg Read through the file to see what directories fcheck is checking, which directories are excluded, etc. The check process is run once every two hours on the 1/2 hour. You can view this by going to: $ cd /etc/cron.d And looking at the file fcheck. $ less vi /etc/cron.d/fcheck The text that reads: 30 */2 * * * Is telling our system cron process to run the long check command listed in the file once every 2 hours on the 1/2 hour. Now let's make a change to a file in one of the directories that fcheck is checking. $ sudo vi /etc/hosts Add a blank line somewhere in the file ("o" adds a new line). Save the file. Now do a forced run of check: $ sudo fcheck -a You'll see lots of stuff go by on the screen. Once this is done let's such email for root to see if fcheck noticed that /etc/hosts has changed. $ su - root [enter the root password] $ mail In mail if you see something like: Mail version 8.1.2 01/15/2001. Type ? for help. "/var/mail/root": 2 messages 2 new >N 1 root@noc.pacnog Wed Jun 30 08:47 45/1421 ALERT: [fcheck] noc.pacnog Just press enter to read the first message, or type the message number (1, 2, etc.). Inside the message scroll through it by pressing the space bar. You should see a line that looks like WARNING: [noc] /etc/hosts [Inodes: 57616 - 57589, Sizes: 316 - 317, Times: Jun 30 19:34 2010 - Jun 30 19:46 2010, CRCs: 1b38420961b94b69cc87a8fa8d777ebb9bedf19ed212fae1f7ce5b451854fb33 - a76ec8d6905a6565b45903ae82c30e960e4db01fad3372b9f050ba66df14560d] This tells you that the file /etc/hosts has changed, shows you the hash of the old file, the new file and all the pertinent file information. To exit from mail first press "q", then type "exit" 3. Turn on automatic installation of security updates -------------------------------------------------- There is a meta package called unattended-upgrades to do this. To install: $ sudo apt-get install unattended-upgrades That's it. Any time a security update is placed in the Ubuntu repositories it will be automatically installed on your system. You will probably want to look at how unattended-upgrades is configured. $ cd /etc/apt/apt.conf.d This package is configured in the file 50unattended-upgrades. Let's have a look and we will make a change to the configuration: $ sudo vi 50unattended-upgrades Note at the very top of the file. If you were to change this: // Automatically upgrade packages from these (origin, archive) pairs Unattended-Upgrade::Allowed-Origins { "Ubuntu karmic-security"; // "Ubuntu karmic-updates"; }; To look like: // Automatically upgrade packages from these (origin, archive) pairs Unattended-Upgrade::Allowed-Origins { "Ubuntu karmic-security"; "Ubuntu karmic-updates"; }; Then all software package updates would be installed as well. You may, or may not, want to do this. This is generally more useful for user desktops than for servers. Let's change this line: //Unattended-Upgrade::Mail "root@localhost"; To be: Unattended-Upgrade::Mail "root@localhost"; That way your root account will get an email when an update is installed. Note that you can even have your machine automatically reboot if required after an update. Save the file and exit: :wq That's it. If a security update is applied you will be notified. 4. Scan your machine remotely using nmap ------------------------------------- It's usually a good idea to see how your machine looks to other users. Log in on another machine in the classroom (pc1-pc10, it does not matter which one). $ ssh pacnog@pcX [X = 1, 2, 3, etcÉ] Now let's scan your box from this remote box using the nmap command: $ sudo nmap -sV pcX [Where "pcX" is _your_ pc] You should see something like: Starting Nmap 5.00 ( http://nmap.org ) at 2010-06-30 09:59 SST Interesting ports on pc2.pacnog.bluesky.as (67.218.55.102): Not shown: 998 closed ports PORT STATE SERVICE VERSION 22/tcp open ssh OpenSSH 5.1p1 Debian 6ubuntu2 (protocol 2.0) 25/tcp open smtp Postfix smtpd MAC Address: 00:0F:1F:E6:62:94 (WW Pcba Test) Service Info: Host: pc2.pacnog.bluesky.as; OS: Linux Service detection performed. Please report any incorrect results at http://nmap.org/submit/ . Nmap done: 1 IP address (1 host up) scanned in 1.76 seconds This looks reasonable. The remote machine is exposing smtp and ssh to the world as well as the type of OS that it is running. Now let's scan a bit more aggressively on your machine: $ sudo nmap -A -T4 pcX Take a look at the information presented. This will take some time, but it will contain more detail. You can use nmap to scan entire networks and find all the machines and services that are running. This is what network attack scripts do - but, they usually scan for a specific port and service, then they launch an attack when they find a service that they think they can break. Be careful with nmap! If you scan aggressively or against an entire network you will likely set off detection alarms and you could get in trouble. Let people know before you scan if you are not in charge of the remote machines. Log out of the remote box: $ exit Now read about nmap to understand what -sV, -A, -T4 and -F are doing: $ man nmap To find what these specific items do more quickly in "man" do: /-A to search for the "-A" string. Press "n" to go to the next occurrence. Do the same for "-T4", "-V", etc... 5. run a root kit checker ---------------------- There is a nice tool called "chkrootkit" - This is used to see if a machine has been compromised with known software kits that install once security has been breached. You can read about this software here: http://www.chkrootkit.org/ To install do this: $ sudo apt-get install chkrootkit To use chkrootkit do: $ sudo chkrootkit You should not see anything found or infected (hopefully!). If you do, then this is serious. You can go back to the http://www.chkrootkit.org/ web site for more information in the README and FAQ pages and you can use Google. But, in reality, if chkrootkit finds anything installed on your system then you probably need to format your hard drive, reinstall and restoreÉ Let's do something to make chkrootkit give you a warning: Place our ethernet interfaces in to promiscuous mode (i.e. it listens for _all_ packets on the network, not just packets coming to your machine). $ sudo ifconfig lo promisc $ sudo ifconfig eth0 promisc Now let's re-run chkrootkit: $ chkrootkit and you will see that it detects that the loopback network interface (lo) and the first network card (eth0) are now in promiscuous mode. To just see this vs. all the other messages do: $ chkrootkit | grep PROMISC Turn off promiscuous mode: $ sudo ifconfig lo -promisc $ sudo ifconfig eth0 -promisc 6. Turn on log reporting --------------------- First, we'll install a different logging program than what comes by default with Ubuntu: $ sudo apt-get install syslog-ng Now let's tell this logging facility to create new log file called /var/log/everything that will contain _all_ our log messages: Edit /etc/syslog-ng/syslog-ng.conf, $ sudo vi /etc/syslog-ng/syslog-ng.conf Go to the end of the file and add: >>>> destination everything { file("/var/log/everything" template("$DATE <$FACILITY.$PRIORITY> $HOST $MSG\n") template_escape(no) ); }; log { source(s_all); destination(everything); }; <<<<< Exit and save from the file. I strongly suggest you copy and paste to do this! Now let's create a script to tell our log rotation process to not let this file get too big. $ cd /etc/logrotate.d $ sudo vi everything In this blank file add the following: >>>> /var/log/everything { daily copytruncate rotate 1 postrotate /etc/init.d/swatch restart endscript } <<<<< Exit and save from the file. Now restart the syslog-ng logging process to pick up all these changes: $ sudo /etc/init.d/syslog-ng restart Now we can install the swatch log monitoring process: $ sudo apt-get install swatch It may already be installed on your machine. Next let's create a rule for swatch to listen for by creating file /etc/swatch.conf $ sudo vi /etc/swatch.conf In this file place the following: >>>> # Bad login attempts watchfor = /FAILED|FAIL|fail|failures / echo bold mail addresses=root\@localhost,subject=Failed_login throttle 01:00 watchfor /(su|sudo):/ echo bold <<<< Save and exit from the file. Now let's start swatch having it use the new swatch.conf file and watching the /var/log/everything log file: $ sudo swatch -c /etc/swatch.conf --tail-file=/var/log/everything --daemon Now let's see if we can get the swatch rule to "trip". Go to one of your consoles: alt-ctrl-f1 Now try logging in as the user "pacnog" but enter in the _wrong_ password for each log in attempt. Do this a few times. Go back to your desktop by pressing alt-ctrl-f7. Now in a terminal window do: $ su - root $ mail And see if you have received on email from Swatch telling you of the attempted and failed logins. 7. Are you being monitored? ------------------------ We are monitoring your machines remotely on our noc box. We are using the network monitoring tools munin, Nagios and Smokeping. http://noc/munin/ http://noc/nagios3 [log in as user "guest" with password of "guest"] http://noc/cgi-bin/smokeping.cgi This is another workshop worth of information. But, in general, it is good to have someone else looking at your system. If your machine goes off the network, exceeds certain CPU usage, stops serving up web pages, etc. - having a remote monitoring service can warn you of this. 8. Partitions ---------- We installed your machine with the following partition scheme: /boot [Kernel and initial bootstrap files] / [everything else] SWAP [your virtual memory] In theory you may want to install and partition more along these lines: /boot [100MB[ /home [large enough for your home users. Be aware of mail] /var [large enough for logs, printer files, email] /usr [all the system program, webs sites, user accounts and databases] / [everything else. usually pretty small] /tmp [how much temporary space you want to give to people] In UNIX the /usr partition may contain web sites and user accounts. If you have /tmp defined you can choose to run this in memory instead. In addition in the file /etc/fstab you might want to define the /tmp partition to be nodev,nosuid,noexec something like this: file system mount point type options dump pass ----------- ----------- ---- ------- ---- ---- /dev/sda5 /tmp ext4 errors=remount-ro,nodev,nosuid,noexec 0 0 For details on what this all means a decent discussion is available here: http://en.wikipedia.org/wiki/Fstab 9. Use chroot for potentially vulnerable services ---------------------------------------------- When we install the DNS server software, BIND, we will may run this service using the chroot facility. To learn about chroot do: $ man chroot To read about all the details of using BIND in a chroot environment have a look at: http://www.faqs.org/docs/Linux-HOWTO/Chroot-BIND-HOWTO.html And, to read about chroot some more see: http://en.wikipedia.org/wiki/Chroot 10. Firewalls --------- We will learn about and practice using the iptables firewall service tomorrow.