We have developed a technical test environment for capturing and decrypting WiFi data transmissions. The code for it lives on Github.
Specifically, we have created a WiFi hotspot that is continually collecting all the packets sent over it. All connected clients’ HTTPS communications are subjected to a “Man-in-the-middle” attack, whereby they can later be decrypted for analysis [1].
This article presents a detailed look at how this test environment works and then gets into how to set one up for your own testing purposes, including a list of required parts. Setting one up yourself is not user-friendly, and requires familiarity with the unix command line and networking concepts.
Motivation
Researchers and end-users alike often seek to understand what data their mobile device is sending to third parties. Unfortunately, monitoring one’s phone to see what, and to whom, data is sent is not exactly simple. Using packet capture software on Android is impossible without first rooting the device, and even then, difficult to use and export saved data. There are no applications to capture packets on iOS.
Our motivation for creating the test environment described herein is to make it incredibly easy to capture packets for any device with a WiFi connection, with very little client configuration needed.
How it works
In our environment, dubbed Snifflab, a researcher simply connects to the Snifflab WiFi network, is prompted to install a custom certificate authority on the device, and then can use their device as needed for the test.
Snifflab architecture
All traffic on the network is logged by a Raspberry Pi dedicated to that task (“PCAP Collecting Machine”, in the Figure). The traffic is cloned by a Great Scott Gadgets Throwing Star LAN Tap, which routes it both to its destination, and to our Raspberry Pi. The Pi continually collects packet data, creating new packet capture (pcap) files at a regular interval, or once the active file reaches a configurable size. Saved files are regularly transferred to another machine (“Backup Machine”) for persistent storage. Users with SSH access to the Pi can also manually restart the pcap service, to get instant access to the captured packets, instead of waiting for the interval.
The custom certificate that each client must install enables the proxy server (“MITM Proxy Machine”) through which Snifflab routes its traffic to intercept HTTPS requests to the outside world, and re-encrypt them using certificates generated on-the-fly. This allows for the researcher to later decrypt most captured network traffic sent over HTTPS.
On the backup machine, the researcher has access to all previously-collected PCAPs, organized into folders by date, with each file named by the unix time at which the capture began.
The researcher may then open up the collected PCAP(s) in Wireshark or their utility of choice to analyze and decrypt the traffic.
On packet captures
A Packet capture (pcap) is a widely used data format for storing low-level network data transmission information. The packet is the base unit of data transmission on networks. To send a message from one computer to another, networking software breaks up the message into small packet files, each with metadata that — among other things — describes the source of the data, the destination, and the specific packet’s ID so that packets can be reassembled correctly at the destination. A pcap file is a collection of packets sent over a network. pcaps are created using software that “listens” to one or more network interfaces running on a given device, and dumps all the data packets it detects into a pcap file for future analysis. For example, one could listen on a computer’s WiFi interface, or the ethernet interface, or both.
How-to
This section describes the hardware, software, and configuration we used to set up Snifflab. It should be sufficient information to guide the creation of a new Snifflab instance from scratch.
Snifflab router, PCAP machine, and LAN Tap
Parts needed
- 1 router (in addition to your primary one), capable of running DD-WRT standard firmware
- 1 Raspberry Pi 2 Model B+
- 1 server running all the time (Ubuntu) for backups and running the MITM proxy
- 1 Great Scott Gadgets Throwing Star LAN Tap
- 1 USB LAN adapter (TRENDnet TU3-ETG)
- 1 USB WiFi adapter (TP-Link TL-WN725N)
- Many Ethernet Cables
The testing network
We used a Cisco WRT54GL router to administer the Snifflab access point. We installed the DD-WRT Standard build firmware on this router. Please consult the DD-WRT guide to ensure your router is supported, lest you risk bricking your device.
We connected this router’s internet port to our pre-existing LAN, creating a subnet with its own IP space. However, we don’t directly connect the router to the LAN. Between the LAN ethernet and the Snifflab router, we place a Great Scott Gadgets Throwing Star LAN Tap. This device allows us to passively sniff all traffic passing through it. In this manner, a copy of all network traffic is copied and directed to our PCAP collecting machine.
The LAN tap, unfortunately, has two separate ethernet ports for sniffing traffic. One is for inbound traffic, and the other, outbound. As such, the PCAP collecting machine needs to have two ethernet interfaces on which to listen for packets. This is discussed further below.
Transparently proxying traffic to get MITM’d
The router’s iptables must also be configured to transparently forward packets to the MITM proxy device. To do that, we login as administrator to the DD-WRT network portal, and navigate to Administration > Commands. There, we enter in the following commands, replacing the value of the PROXYIP variable with the IP address of your MITM proxy machine:
PROXYIP=192.168.0.2 iptables -t mangle -A PREROUTING -j ACCEPT -p tcp -m multiport --dports 80,443 -s $PROXYIP iptables -t mangle -A PREROUTING -j MARK --set-mark 3 -p tcp -m multiport --dports 80,443 ip rule add fwmark 3 table 2 ip route add default via $PROXYIP dev vlan1 table 2
The last line in the above code refers to the vlan1 network interface; Make sure the network interface your router uses to communicate with the WAN is in fact vlan1 for your router, and adjust if needed. For example, this guide to setting up transparent proxies mentions br0 instead.
Make it easy to get MITM’d
The other thing we’d like our Snifflab router to do is to act as a convenient delivery service for our mitmproxy CA certificate. The method we chose is to make Snifflab a captive WiFi portal, meaning that users must visit a sign-in splash page before being able to use the network. We create a custom splash page for the user to sign into, and a redirect page from where the certificate can be downloaded after sign-in.
A captive WiFi portal screen makes it simple to install the MITM certificate on a client device.
To configure a captive WiFi portal, we login to the DD-WRT admin interface on Snifflab’s default gateway, and navigate to Services>Hotspot. We then enable NoCatSplash.
Set up the NoCatSplash parameters to point the home page and splash URLS to a web server (hopefully running on your WAN) that serves redirect.html and splash.html as contained in the Git repository described below. Ensure that Homepage Redirection is turned on. Set a login timeout.
The PCAP collecting machine
In our system, a Raspberry Pi 2 Model B+ functions as the PCAP collecting machine. Setting up this Pi for sniffing network traffic sent to it from the LAN tap, and then backing up those captured packets to another machine, requires the configuration of multiple network interfaces.
Since the LAN tap splits inbound traffic to one ethernet port, and outbound to another, we need two ethernet interfaces on the Pi to capture both directions. However, packet capturing is much more straightforward if done on a single network interface. As such, we’ll have to bond the two ethernet connections into one network interface. To do that, install ifenslave and set networking interface commands as follows:
sudo apt-get install ifenslave-2.6
/etc/network/interfaces
auto lo iface lo inet loopback iface eth0 inet manual iface eth1 inet manual allow-hotplug wlan0 iface wlan0 inet dhcp wpa-conf /etc/wpa_supplicant/wpa_supplicant.conf iface default inet dhcp auto bond0 iface bond0 inet dhcp bond-mode 3 bond-miimon 100 slaves eth0 eth1
Make sure bond-mode is set to 3, a “broadcast” policy, meaning all packets from all interfaces are transmitted via the bonding interface, otherwise packets may get dropped.
WiFi
Setting up WiFi driver for TP-Link TL-WN725N
mkdir driver cd driver wget https://dl.dropboxusercontent.com/u/80256631/8188eu-v7-20150713.tar.gz tar xvzf 8188eu-v7-20150713.tar.gz sudo ./install.sh cd ../ rm -rf driver sudo reboot
/etc/wpa_supplicant/wpa_supplicant.conf
ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
update_config=1
network={
ssid=""
psk=hashofyourpassword
proto=RSN
key_mgmt=WPA-PSK
pairwise=TKIP
auth_alg=OPEN
}
Customize this script to suit your particular WiFi connection. It may be easier to generate a config file through the Raspberry Pi GUI. It’s also best practice to not store the plaintext version of your WiFi password in this file. Instead, run wpa_passphrase to generate a hash of the password.
A WiFi connection is necessary since the two ethernet ports only receive traffic cloned from the LAN tap, and don’t actually connect to the network. Thus, a networking interface is needed so that one can SSH into your machine, and transmit pcap files to the backup machine.
Getting the network running correctly on boot
By default, on Raspberry Pis, if ethernet is plugged in then WiFi will be automatically disabled.
First, turn off ethernet hotplugging, which causes WiFi to get disabled.
/etc/default/ifplugd
INTERFACES="eth0" HOTPLUG_INTERFACES="eth0" ARGS="-q -f -u0 -d10 -w -I" SUSPEND_ACTION="stop"
Next, the below script runs at startup to ensure that multiple networking interfaces can operate simultaneously. We run the ifup command on each network interface to force start all of them. We also set the two ethernet interfaces to promiscuous mode to ensure they process all packets being routed to them, for complete sniffing.
/etc/init.d/network.sh
#!/bin/sh ### BEGIN INIT INFO # Provides: network.sh # Short-Description: Ensure WiFi as well as Ethernet interfaces are up # Description: # Default-Start: 2 3 4 5 # Default-Stop: 0 1 6 # Required-Start: $remote_fs $syslog # Required-Stop: $remote_fs $syslog ### END INIT INFO sudo ifplugd eth0 --kill sudo ifup wlan0 sudo ifup eth0 sudo ifup eth1 sudo ifconfig eth1 promisc sudo ifconfig eth0 promisc exit 0
Dependencies for collecting packets
Ensure you setup your ~/.ssh/config to connect to your analysis machine without the need for a password, otherwise the backup script will fail.
ssh-keygen -t rsa cat ~/.ssh/id_rsa.pub | ssh user@server 'cat >> .ssh/authorized_keys
Installing libpcap
sudo apt-get install libpcap0.8 libpcap0.8-dev libpcap-dev
Installing pip
sudo apt-get install python3-pip sudo apt-get install python-pip
Installing pcapy
wget https://pypi.python.org/packages/source/p/pcapy/pcapy-0.10.8.tar.gz tar xvzf pcapy-0.10.8.tar.gz cd pcapy-0.10.8/ python setup.py install
Installing dpkt
pip install dpkt pip install dpkt --upgrade
Make sure timezone is set correctly
tzconfig sudo dpkg-reconfigure tzdata
Capturing Packets
Get the latest code from Github. This guide refers to a copy of the repo housed at /home/pi/snifflab. This repository contains sniffer.py, a Python application developed to capture packets on an interface into PCAPs, creating new ones at a fixed interval, or when a file size limit is reached. The interface, interval, and filesize limit are all configurable command line parameters:
-i (specify the network interface) -s (specify the file size limit) -t (specify the time interval, in seconds, between new PCAP files) -f (specify a filename suffix to append to each PCAP. -u (specify a ssh username for a remote backup) -h (specify a ssh host for remote backup) -p (specify the path on the remote host for backup)
The script also backs up old PCAPs to a separate machine by launching an external bash script upon the creation of a new PCAP file. As described above, there are also parameters to specify the remote backup server username, hostname, and path for SCPing the PCAPs:
Move the repository’s splash.html and redirect.html to the WAN web server of your choice, as described above. Edit the link in splash.html to correspond to your Snifflab router’s IP address.
Configure the backup script
Make sure to add the PCAP machine’s public SSH key to the known_hosts file on the remote machine, or else movescript will prompt the user for a password.
Start capturing packets on startup — create a sniffer service
The simplest way to ensure reliably consistent packet capturing is to wrap sniffer.py in an upstart service, so it can easily be started on boot, and restarted as needed. To do that, create and edit the file below:
/etc/init/sniffer.conf
#sniffer.conf start on runlevel [2345] stop on runlevel [016] script exec echo "hi" cd /home/pi/mitm if [ -z "$filenamesuffix" ]; then exec python sniffer.py -i bond0 -s 100 -t 120 else exec python sniffer.py -i bond0 -s 100 -t 120 -f $filenamesuffix fi end script
Make sure to edit the parameters of sniffer.py to suit your needs, based on the previous section.
With this service all set up, it is easy to start / stop collecting packets:
sudo service sniffer start
This service supports an additional parameter, a filename suffix. This will cause all pcaps collected by the service to be saved with the parameter as a file suffix. For instance, running
sudo service sniffer start filenamesuffix=test1
will yield timestamped packets with test1 as a suffix, similar to 123456789_test1.pcap.
This makes it easy to label collected pcaps. When starting a new test, simply restart the service with a new filename suffix:
sudo service sniffer restart filenamesuffix=mynewtest
Of course, if the filenamesuffix parameter is missing, no suffix is included in the filename. After completing a test, I usually restart the service without a suffix, to cleanly demarcate the completion of the test in the filesystem.
The MITM Proxy machine
In our environment, we used a machine running Ubuntu Server operating system to act as a proxy that replaces SSL certificates with those under our control. It does this by relying on the well-known mitmproxy software. Our MITM proxy machine exists outside of the Snifflab network, on the WAN. Snifflab forwards all traffic to this machine. This section describes how to configure a service that runs a transparent mitmproxy on startup.
First, install mitmproxy and mitmdump, following the guide on their website.
Now, we can set up a default configuration for mitmproxy to refer to on runtime. In our environment, we created a directory:
sudo mkdir /etc/mitmproxy sudo chmod -R 755 /etc/mitmproxy touch /etc/mitmproxy/common.conf
Edit common.conf to match the following:
--cadir=/etc/mitmproxy/cybersniff --port=4567 --noapp
I chose an arbitrary port number because I do not like using defaults. (mitmproxy defaults to 8080).
Now, create a directory to store the mitmproxy certificate authority information:
mkdir cybersniff
Next, generate your own certificate for mitmdump to use as a CA. I do this so I don’t have to trust mitmproxy’s CA. Mitmproxy has a guide on how to generate your own certificate. I would recommend adding –days 1000 to the certificate generation step, to ensure the cert doesn’t expire for some time. Make sure the certificate files are stored in the path referred to in common.conf
Now, let’s set up a service to ensure the proxy starts on boot, and can be easily started and stopped as needed:
touch /etc/init/mitm.conf
Edit your newly-created mitm.conf to include the following:
# mitm.conf start on filesystem script sudo iptables -A PREROUTING -t nat -i em1 -p tcp -m multiport --dports 80,443 -j REDIRECT --to-port 4567 SSLKEYLOGFILE=/var/log/mitmkeys.log export SSLKEYLOGFILE echo "MITM Keys being logged here: $SSLKEYLOGFILE" exec mitmdump -T --host --conf=/etc/mitmproxy/common.conf end script
This service does several things.
First of all, it starts when the filesystem is loaded. You can also start, stop, and restart it using commands like:
service mitm stop
Secondly, the service sets up iptables rules to route all incoming traffic on ports 80 and 443 to port 4567 (change this value to correspond to your mitmproxy port defined in common.conf).
Next, it specifies an SSLKEYLOGFILE environment variable. This is important, as it provides a place for mitmproxy to save session keys used to set up encrypted communications. With these keys being routinely logged, we can point wireshark to this location (discussed below), and use it to decrypt SSL traffic that uses Diffie-Hellman key exchange.
Finally, the service runs mitmdump (a non-interactive version of mitmproxy) in transparent mode, using the configuration file we previously created.
Important: One last thing to do in order to ensure that devices connect to Snifflab can access HTTPS resources without a certificate error is to copy the generated .pem file for your custom CA certificate to the web directory where redirect.html is located. Edit the link in redirect.html (included in the Github repo, and hosted on your web server) to point users to download your pem file for them to install on their device.
Of course, you can develop your own way of providing the certificate to devices on the network.
Analysing the data in Wireshark
To decrypt packets that have been encrypted using perfect forward secrecy, it’s best to have the latest version of Wireshark installed. This might mean adding the Wireshark development PPA to your Ubuntu environment.
Next, configure Wireshark’s Preferences > Protocols > SSL to utilize your MITM proxy’s CA bundled certificate file as well as the master keys list MITMproxy is logging.
Click on RSA Keys, and enter the following:
| IP address | Port | Protocol | Key File |
| 0.0.0.0 | 443 | http | /path/to/mitmproxy-ca.pem |
This tells Wireshark to attempt to decrypt all traffic (0.0.0.0 is a stand-in address for any IP) on port 443 as HTTP data, using your key file.
Back in the main SSL protocol preference pane, set up an SSL debug file to you can assess any errors that might occur.
Finally, point (Pre)-Master-Secret log filename to the path set in the $SSLKEYLOGFILE environment variable (or a place you copy that log file to).
With this, you can load any of the PCAP files collected on the SniffLab network and decrypt HTTPS connections! Happy hacking.
Acknowledgements
This work was financially supported by Canada’s Office of the Privacy Commissioner’s Contributions Program. Our larger project is looking at the privacy and security of wearable fitness tracking devices.
Special thanks go out to Jakub Dalek and the mitmproxy team.
Comments
Please share your thoughts on this article on Twitter. I’m happy to discuss issues on Github.
Notes
[1] This method does not work against applications using certificate pinning. Circumventing certificate pinning requires rooting the device and installing a custom application.
Request access to your personal data with our simple tool
