Tee is for Tubes

It is a curiosity that both existing Grantlee libraries begin with the letter ‘T’. This can not have been coincidence. I thought it best that I continue the trend and keep adding libraries whose name begins with ‘T’.

So was renamed the Grantlee Pipes library to Grantlee Tubes. Grantlee Tubes is a library of classes supporting the QIODevice API in an object-orientated way. The developer can connect a series of Tubes to achieve exacting goals in much the same way that the Unix programmer connects commands with pipes (‘|’). The first class to hit the public repo is, appropriately, Grantlee::Tee.

Trial and error

I encountered the tee command when I first started using Ubuntu and came across instructions like this to add repositories to the system:

echo "deb http://security.ubuntu.com/ubuntu jaunty-security main restricted universe multiverse" | sudo tee -a /etc/apt/sources.list

I already knew about shell redirection by then so I had wondered why I couldn’t simply do this:

echo "deb http://security.ubuntu.com/ubuntu jaunty-security main restricted universe multiverse" >> /etc/apt/sources.list

Firstly, permission denied, so try Plan B

sudo echo "deb http://security.ubuntu.com/ubuntu jaunty-security main restricted universe multiverse" >> /etc/apt/sources.list

This doesn’t work because echo is run as superuser, but that permission does not cross the ‘>>’ boundary. It still attempts to write to the root-owned file as a normal user.

tee resolves that issue by reading from stdin and writing to standard out (like cat) AND to any specified targets.

Tee for Two

The name Tee comes from a plumbing term for a device which allows splitting a stream of fluid to two targets. The tee command allows duplicating a stream to N targets, allows quite a bit of versatility.

Tee: Duplicating targets

The GNU Coreutils manual is informative and illustrative in the use and use cases for tee. Primarily the advantages of its use are that it allows processing of streams which is efficient both in terms of memory use (the entire input does not need to be held in memory), and in terms of speed by enabling parallel processing.

 wget -O - http://example.com/dvd.iso \
       | tee dvd.iso | sha1sum > dvd.sha1 

Piping and Tubing

QIODevice is one of the most dominant interfaces in Qt. It is the base for most data reading and writing APIs such as QFile, QTcpSocket, QNetworkReply etc, and it is the interface used in most collaborators to data reading and writing, such as QDataStream, QTextStream, QTextDocumentWriter and more. This means for example that a QTextDocument may be written to a Tcp socket or a QProcess just as easily as it can be written to a file, from a Qt API point of view. The asynchronous nature of QIODevice adds to its versatility and suitability for many tasks around streaming data in the real world.

Just as we combine commands in a Unix shell with pipes, so too we want to be able to combine source and target QIODevices in efficient and familiar ways. It should be possible to implement something like this in Qt:

tardir=grantlee_src
     tar chof - "$tardir" \
       | tee >(gzip -9 -c > grantlee.tar.gz) \
       | bzip2 -9 -c > grantlee.tar.bz2

This is where Grantlee::Tubes comes in.

Tubular

Want to write large data to multiple files without multiple read/write cycles, holding all data in memory or file copying? Easy:

QIODevice *sourceDevice = getSourceData();
QFile *file1 = getFile("file1");
QFile *file2 = getFile("file2");
QFile *file3 = getFile("file3");

Grantlee::Tee *tee = new Grantlee::Tee(this);

tee->appendTarget(file1);
tee->appendTarget(file2);
tee->appendTarget(file3);

// Now write to Tee. readAll for demostration but is not
// memory efficient.
tee->write(sourceDevice->readAll());

Want to write to a log file while also writing to a QTcpSocket? Easy:

QIODevice *sourceDevice = getSourceData();
QTcpSocket *output = getOutput();
QFile *logFile = getLogfile();

Grantlee::Tee *tee = new Grantlee::Tee(this);

tee->appendTarget(output);
tee->appendTarget(logFile);

tee->write(sourceDevice->readAll());

An important point to grasp is that Grantlee::Tee implements QIODevice. That means that you can write to a Tee (and therefore multiple targets) just as easily as you can write to a single target.

This can be made easier (and asyncronous) using Grantlee::Pump:

tee->appendTarget(output);
tee->appendTarget(logFile);

// Replaced with Pump.
// tee->write(sourceDevice->readAll());

Grantlee::Pump *pump = new Grantlee::Pump(this);
pump->setSource(sourceDevice);
pump->setTarget(tee);

Grantlee::Pump listens to the readyRead signal and writes all available data in the source to the target until there’s no more left. That means that data can be fed into Tee in smaller chunks and it is not required to readAll data into memory up front.

Compression is equally possible. Here’s an example where a Grantlee::Template is rendered to a QIODevice which is piped through a compressor, and then a TcpSocket:

QTcpSocket *socket = getSocket();
Grantlee::Compressor *compressor = new Grantlee::Compressor(this);
compressor->setTarget(socket);

Template t = engine->newTemplate("main.html");
Context c = getContext();
// The string template is rendered to the compressor, which writes to the socket. 
t->render(compressor, c);

Down to a Tee

Several of these classes are already in various stages of being written, getting tests and documentation. For now I’ve added Tee to a volatile branch of Grantlee destined for some future release. Along with that all of the infrastructure for a third library in Grantlee has been added: documentation, CMake infrastructure, test framework etc. Grantlee::Tubes will continue growing in the coming days.

Advertisement