Introduction

The Sax Benchmark project was created to test the performance, correctness and usability of the various HTML SAX parsers available.

NOTE: As with all benchmarks, these need to be taken with an appropriately large dose of salt. This suite is intended as a helpful starting point in evaluating the various libraries out there but if my experience thus far is anything to go by, your mileage will vary. Also, these benchmarks may well contain inaccuracies, bad methodology, bad configuration and general confusion, although none deliberate. If you spot anything that could be improved, please let me know.

The four SAX parsers under the knife are:

There are several different criteria that can be used when evaluating a particular HTML SAX Parser and it is important to keep this in mind when interpreting the results. These criteria could include: correctness, speed, memory or failsafety and they may well be mutually exclusive. And as with any benchmarks the application under review may be tailored to work far better in the real world than in a benchmark.

I suggest reading through the notes below before proceeding to the results.

In no particular order, my thanks to John Cowen (TagSoup), Andy Clark (CyberNeko), Ed Howland (HotSax) and Derrick Oswald (HTMLParser) for helping improve this benchmark. All inaccuracies, inconsistencies etc. are all mine, though.

SAX Benchmark Notes and Caveats

  • Number of runs - The number of times a benchmark is run is configurable. A number of runs are required to get around the System.currentTimeMillis() 16ms minimum on Windows.
  • JIT - In order to kickstart the JIT, the parser is run once without being timed.
  • Removing IO Overhead - The source is parsed and loaded into a ByteArrayInputStream to remove IO Overhead.
  • EmptyContentHandler - When measuring the parse time, a content handler with empty methods is set into the parser.
  • Memory benchmark - The memory benchmark is unreliable as it depends on a System.gc() at the start and that one doesn't happen before then end of a single parse. All of that said, the numbers are pretty consistent between runs.
  • Output - The output depends heavily on the HTML serializer that is being used. The benchmark uses a few different HTML serializers but the combinations become extreme as soon as different configurations are introduced per parser and per serializer. See the section on customizing this benchmark if you're interested in changing configuration settings for the serializers.
  • Sources - The sources for the benchmark include a simple html file to test various scenarios like balance tag support etc. and a collection of reasonably popular sites on the internet. Since only the HTML was downloaded and the links weren't rewritten, viewing the sources or output in a browser will look wrong. I recommend taking a look at the source HTML.

Configuration

TagSoup Parser

There were three configurations tested:

  • Default - The out of the box configuration
  • IgnoreBogons - The Ignore Bogons feature switched on: setFeature("http://www.ccil.org/~cowan/tagsoup/features/ignore-bogons", true)
  • NotEmptyBogons - Empty Bogons set to false: setFeature("http://www.ccil.org/~cowan/tagsoup/features/bogons-empty", false)

HTMLParser

HTMLParser was run with two configurations:

  • Default - The out of the box configuration
  • Namespaces - run with namespaces off and namespace prefixes on as this was made the some of the serializers better behaved: setFeature("http://xml.org/sax/features/namespaces", false), setFeature("http://xml.org/sax/features/namespace-prefixes", true).

CyberNeko Parser

There were two configurations tested:

  • Default - The out of the box configuration
  • NoBalancing - The out of the box configuration with tag balancing switched off: setFeature("http://cyberneko.org/html/features/balance-tags", false);

HotSax Parser

HotSax was run out of the box without any configuration set.

TagSoup XMLWriter

The TagSoup XMLWriter was run with HTML mode on (setHTMLMode(true)).

SerializerFactory

The SerializerFactory was run using the "html" option.

ToHTMLStream

ToHTMLStream was was run out of the box without any configuration set.

Parser Notes

TagSoup

Tagsoup's tagline is "Just Keep On Truckin'", a reference to it's emphasis on dealing with as wide a variety of HTML as possible. From the TagSoup site: "This is the home page of TagSoup, a SAX-compliant parser written in Java that, instead of parsing well-formed or valid XML, parses HTML as it is found in the wild: nasty and brutish, though quite often far from short. TagSoup is designed for people who have to process this stuff using some semblance of a rational application design. By providing a SAX interface, it allows standard XML tools to be applied to even the worst HTML."

CyberNeko

From CyberNeko's site: "NekoHTML is a simple HTML scanner and tag balancer that enables application programmers to parse HTML documents and access the information using standard XML interfaces. The parser can scan HTML files and "fix up" many common mistakes that human (and computer) authors make in writing HTML documents. NekoHTML adds missing parent elements; automatically closes elements with optional end tags; and can handle mismatched inline element tags."

HTMLParser

HTMLParser describes itself as "a super-fast real-time parser for real-world HTML". HTMLParser includes a rich set of classes for gathering, filtering and transforming HTML. A standard SAX parser built on the underlying HTMLParser API is included and is a relatively recent addition. I'd recommend looking at the full library, especially if your requirements are broader than just a SAX compliant XMLReader.

Here are some notes on the HTMLParser XMLReader:

  • The HTMLParser XMLReader only works with SystemIds - As the tests require an InputStream or Reader (to try to minimise IO overhead affecting the numbers), I subclassed the XMLReader to include this functionality. While I do not believe that this has had any negative effect, please bear this in mind when interpreting the results. Derrick Oswald has said that he will be including this code in a future release of HTML Parser.
  • The HTMLParser XMLReader puts in #text=" " elements for attribute whitespace - Derrick Oswald confirmed that this is to preserve whitespace in the attributes.

HotSax

HotSax describes itself as "... a small fast SAX2 parser for HTML, XHTML and XML." HotSax is at version 0.1.2c and is at Pre-Alpha. The project was last updated on May 3rd 2002.

  • HotSax doesn't work with InputStreams - HotSax will throw a NullPointerException if you try to use an InputStream but will work fine if you use an InputSource with a Reader.
  • HotSax doesn't populate the qname for elements - HotSax does populate the localName for elements but not the qname. It looks like most of the HTML serializers use the qname to output HTML. If they do use the localname, unfortunately, it looks like they use the localName for attributes in which case:
  • HotSax doesn't populate the localName for attributes - This means that serializers that expect a localName instead of a qname don't output anything.
  • HotSax throws an ArrayOutOfBoundsException - HotSax is the only parser that fails one of the tests as the result of an ArrayOutOfBoundsException.

Unfortunately, because of the localName / qName issues, it wasn't possible to get any useful HTML generated out of HotSax for comparison with the originals and the other libraries. It shouldn't be too hard to write a custom HTMLSerializer which uses the particular combination of attributes that HotSax produces.

Other HTML Parsers

This benchmark only tests HTML SAX parsers. There are a number of other libraries that parse HTML but don't have SAX implementations. A list of some of these can be found at java-source.net.

Serializers

The results of the parsers tested depend heavily on the libraries outputting the results of the parsing. Unfortunately, while parsers and serializers are tightly coupled in practice, often the best pairing of parser and serializer is not an easy task. TagSoup provides it's own HTML Serializer (XMLWriter). HTMLParser relies on it's higher level API's to do native HTML serialization, so I had to try out some of the open source alternatives to do serialization using a ContentHandler. CyberNeko comes with an HTMLWriter but it is a Xerces Filter instead of a ContentHandler.

Serializers - Xalan and Xerces

Including Xalan and Xerces in the Java 1.4 JDK has been problematic (see the FAQ). Over time, Xalan and Xerces have supported all of the following packages:

  • org.apache.xalan.serialize (deprecated)
  • org.apache.xml.serialize (deprecated)
  • org.apache.xml.serializer

All of the above packages contain a SerializerFactory, which is how Serializers should be instantiated.

Serialization classes include:

  • org.apache.xalan.serialize.SerializerToHTML (deprecated)
  • org.apache.xml.serializer.ToHTMLStream
  • org.apache.xml.serialize.HTMLSerializer (deprecated)

Tagsoup XMLWriter

TagSoup comes with it's own HTML Serializer, namely: XMLWriter. The HTMLMode was set to true for output

Using SAX Benchmark

The SAX Benchmark project was created to test HTML parsers but because it uses only the standard SAX api's it could quite easily do benchmarking of regular SAX libraries.

Running

SAX Benchmark uses Maven to build and run itself. If you have maven installed, you can run it using: 

maven -Dcount=10 saxbenchmark

Configuration

Currently, the suite is not particularly configurable, so to try out your own parsers or writers, you'll need to change the *Supplier classes (or supply your own). Specifically:

  • DefaultOutputterFactorySupplier
  • DefaultReaderSupplier

What the benchmark will do, is pick up any sources that are placed in the "./src/data" directory. So if you have any particular sites you'd like to benchmark, just put the source into that folder and run the benchmark.