How we pwned your ICS or why you should not put your HMI on the internet - Internetwache

The team of Internetwache.org has researched the security of industrial control systems (ICS) for the past months and we have discovered more than hundred unsecured controls of waterworks, heating stations, parking lots and buildings.

Everything started last year around october when Tim discovered a privacy issue at the swiss Prime Tower (article in german). Excited by this discovery he decided to look for further ICS and shortly discovered more. The first more interesting system was a waterwork, which after a report to the german CERT (BSI), was taken offline a couple of days later. At this point Tim and Sebastian decided to do a broader scan. You can find the results in the german article on Golem.de.

This blog post will try to describe the used methods, some numbers and background information.

All beginnings are difficult

At the beginning, Tim started out with a little python script and tried to scan the internet. He succeeded in finding some interesting things, however the process itself was slow. Nevertheless, Tim was able to pinpoint a specific value in a http header which most likely identified an ICS, which used a specific control panel. All that was missing was a more efficient process.

Sebastian had the idea of using Zmap, a tool developed by the university of Michigan, in combination with the tool Zgrab to scan the IPv4 space for the specific identifier. Regarding to the project’s website, one can scan the whole IPv4 space in just five minutes when an appropriate internet connection is used. This would allow us to scan a specific port and later apply different filter criterias on the dataset to discover different systems.

The command looked like this:

sudo zmap -B 10M -p 80 --output-fields='*' | ztee results.csv | zgrab --port 80 --http="/" | gzip >  banners.gz 

Admittedly, this approach was not the best one for us, because we couldn’t find an appropriate internet connection or hoster which would allow us to do such a scan without dealing with a lot of abuse mails and so on. Additionally, a slower connection (< 1 Gb/s) would have increased the scan duration to several hours or days.

That’s why we chose another option: Sebastian remembered that the project scans.io publishes exactly the same datasets that we wanted to create. Those datasets consist of different, internet-wide scans of different ports and protocols and are usually created every other week. In this category are weekly scans of HTTP requests and their responses to port 80. We used the lastest dataset, which was compressed around 80 gigabytes in size with exactly 62276536 entries. It contains all IPs out of the IPv4 space which successfully responded to a GET request.

Every entry contains the following information as JSON:

{
    "ip": "8.8.8.8"
    "host": "8.8.8.8",
    "vhost": "8.8.8.8", 
    "port": 80, 
    "data": base64(HTTP-answer), 
}

The ‘data’-field contains the full base64-encoded http response including all headers and the body.

Performing the scan

All human machine interfaces (HMIs) could be identified by a specific value in a HTTP header (2 diffrents varients/versions). Sebastian wrote a python script which decompressed the dataset, read it line-by-line and then looked for that specific value in the base64-decoded string. This reduced the dataset to 1796 entries.

$> python2 filter_scansio.py
$> wc -l output*.ips
     142 output-type1.ips
    1654 output-type2.ips

The script’s output probably contains some unnecessary empty or duplicate lines. Extracting all ip adresses gives a better number:

$> grep -oP "\d+\.\d+\.\d+\.\d+" /tmp/output-*.ips | sort | uniq | tac | wc -l 
838

In a first run we checked all ip addresses to see if they are still online and whether they are associated with a HMI or not. We noticed that most HMIs are http basic authentication protected - but we did not only find ICS, but also some porn websites which used the same pattern in the html code. We wanted to get rid of those “false positives”.

So we used another filter. In this case we used the name of a specific javascript file which was used by the software.

$> python2 js_filter.py | sort | uniq | tac | wc -l 

The dataset became smaller again - there were only a bit more than 60 systems left which did not make use of HTTP basic authentication.

After a few contacts with the affected operators we found out that our procedure to manually verify the status of an ICS was not very effective. This was the reason why Sebastian developed a plugin for the Nmap-Script-Engine (NSE), which was used to activate the filters and to look if the patterns apply. The final command looked like this:

$> nmap -n -PN -d --script nmap-find-hmis.nse -p 80 -iL all.ips  -oX nmap-all-ips-plugin-scan.xml -T paranoid

The output of the script was in nmap’s XML-format and only had to be filtered for diffrent categories:

“Discovered, authenticated”
“Discovered, unauthenticated”
“Not Discovered”

To do so we coded another python-script:

$> python2 filter-nmap-plugin.py ./nmap-all-ips-plugin-scan.xml | sort -u | grep "Discovered, unauthenticated" | wc -l 
42
$> python2 filter-nmap-plugin.py ./nmap-all-ips-plugin-scan.xml | sort -u | grep "Discovered, authenticated" | wc -l 
40 
$> python2 filter-nmap-plugin.py ./nmap-all-ips-plugin-scan.xml | sort -u | grep "Not Discovered" | wc -l
673 

“unauthenticated” stands for ICS which did not make use of HTTP-basic-authentication and “authenticated” for basic auth-protected systems. Depending on the proxy and timeouts there were a few variations of the results - some systems had a very long response time, because they were in another part of the world or had a slow internet connection.

Results

At the end we had a list - we were able to use this list to find out a few interesting things about ICS. Out of all discovered ICS, only 50% were properly protected from unauthorized access by http basic authentication (however, still over an insecure HTTP connection). We didn’t want to use brute force or similar intrusive methods, so we focused on the accessible, unauthenticated systems. Simply typing the IP address into a browser’s address bar was enough. We identifed four major categories:

Waterworks: 4
Parking lots: ~10
Smart homes or hotels: >5
Biogas-/block or remote heating stations: 7

Geo-IP location helped us to estimate the affected countries, which are mostly from the DACH-area (Germany, Austria, Switzerland), but also USA, France, Italy, Isreal. This might be due to the fact that the vendor of the HMI sells it in the DACH area and lets reseller cover the other countries.

After analyzing our results we proceeded to contact more than ten different CERTs - most countries have one - informing them about the problem and systems which are at risk in their country. We politely asked to get in touch with the operators to secure the systems. A handful of CERTs thanked us for the information and assured us that they’ll try to contact the appropiate administrators. Sure enough more and more systems went offline or activated the password protection in the following weeks.

You can read more about this in the german article on Golem.de.

Vulnerabilities

During this project we also had a look at the webapplication’s security, after Tim had a first assumption at a specific point. Sure enough Sebastian was able to turn Tim’s assumption into a simple XSS. Furthermore he discovered a HTTP header injection. Both vulnerabilities are currently coordinated by the Cert-CC, but we’ll give more information as soon as the vendor releases a fix and CVE-IDs are assigned.

Timeline

To get an impression of how things went, we’ll share a rough timeline:

October 2015
- Looking into industrial routers / Discovery: Prime Tower web application
October 2015 - January 2016
- Multiple reports to the company => 3 months, end of responsible disclosure timeframe
22.01.2016
- Article on golem.de
22.01.2016 (6 hours after publication)
- Web application offline / Operator’s statement (see detailed description and analysis of the results)
March / April 2016
- Analysis of the saved data (HTTP-Requests, etc.), writing a scanner-script and using zmap
Beginning of April 2016
- First scan of public IP addresses (Discovered of some systems - more or less critical)
11.04.2016
- Email: Asking the vendor if they’re aware of not properly configured systems
  - Requesting the security-manual
  - First request about specific parameters (which will later become security relevant)
Mitte April 2016
- Weekend: Furthers scans / Analysis: Discovery of a waterwork (!), Tim and Sebastian discover an XSS
17.04.2016
- Email to Bund-CERT (BSI) - Informing them about the waterworks
18.04.2016
- BSI acknowledgement: “We’ll contact the operators and try to help/fix it” (quote translated)
19.04.2016
- Response from vendor - Mostly general information about products
19.04.2016
- Response to vendor
  - Information about XSS
  - Mention: Discovery of waterwork
  - Request: Phone call
End of April 2016
- Sebastian discovers HTTP-Header-Injection (Proof of Concept send to the vendor)
20.05.2016 (1 month after our first answer)
- Response from vendor - Phone call is possible
20.05.2016
- Phone call with vendor via Skype
- Content:
  - Vendor is not responsible for the client’s configuration
  - Vendor does not want to give general security information to clients, but can do so via its resellers.
  - Security-Manual: Work in progress, will be handed out in May
  - Request for a Demo system: Might be possible at the end of May
End of May
- Sebastian develops a new script => better and faster scanning possible
  - Usage of scans.io datasets
  - Development of a nmap plugin
  - Performing the broader scan (More than 120 hits, partly password protected)
  - First analysis of the results (further discoveries)
26.05.2016
- Email to vendor
  - Mention of the main scan results
  - Appeal for quick fixes
  - Appeal for distribution of security information over resellers
End of May 2016
- Informing more than ten CERTs
06.06.2016
- Response from Vendor (last received email): general information
09.06.2016
- Response to vendor: Specific questions about vulnerabilities and disclosure process.
17.06.2016
- Last request about more information
- No reaction:
  - No information about the vulnerabilities
  - No security manual (manual for a secure operation of the software) => Should had been ready by May
  - No demo system
End of June / Beginning of July 2016
- Contacting multiple CERTs/operators again
- Waiting for operators to secure their systems
Beginning of July 2016
- Reporting the vulnerabilities to CERT-CC asking for coordination/disclosure help
15.07.2016
- Publication the article on golem.de
- Publication of the tease-article at Spiegel.de
- News spread across media
16.07.2016
- Publication of the article in Print Spiegel