Certificate Transparency as a source for subdomains

We have been quite busy this year, but we would like to release a small project nonetheless: Curating a list of subdomains based on certificate transparency logs that we’ll happily share with the community and publish new results every hour.

Certificate Transparency is a project initiated by Google that tries to monitor all issued SSL/TLS certificates with the goal of identified and revoking mis-issued certificates. Certificate Authorities are requested to publish information about their issued certificates into the publicly accessible CT logs. That is why you will find all certificates for internetwache.org on crt.sh for example.

A log entry contains all information about a certificate, and therefore the fully qualified domain name (FQDN). Since the start of Let’s Encrypt, it has become really easy to obtain and use SSL-/TLS certificates free-of-cost. However, it does not support wildcard domains (yet), but it’s on the roadmap for January 2018. Until then, every subdomain has to be explicitly listed in the certificate (or multiple certificates used). This means, that subdomains whose existence was not known before (security by obscurity, e.g. xyz-asd.domain.tld…) are easy to discover now.

Undoubtedly this information can be interesting from a hacker’s perspective. Sebastian’s idea was to use the certstream python module to subscribe to Certificate Transparency log updates, parse the subdomain and save the result in a database. Each subdomain also comes with a counter to sort it by frequency. The resulting lists with the top 100/1000/10,000/100,000 subdomains is automatically exported to the “CT_Subdomains” GitHub repository on an hourly basis.

The files have the following scheme:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
# Generated 2017-12-11 11:00:01.087903
count,subdomain
8529329,*
3573653,www
779370,mail
507273,webdisk
497893,webmail
480967,cpanel
178293,autodiscover
21955,dev
19415,blog
18989,m
18518,test
16513,shop
15859,whm
10483,api
10236,admin
9341,com
7521,mdp
7484,staging
7370,cloud
7173,demo
6667,app
6275,tls
6070,eu
5882,mbox12
5223,store

Link to the GitHub “CT_subdomains” repository

One could use this list as the input for DNS reconnaissance tools like gobuster.

There has been previous research about security implications in regards to certificate transparency logs, for example by Hanno Boeck at the DEF CON 25.

The team of internetwache.org

Comments