10 - Append-only logs

Great work getting so far in the workshop!

In the previous layer we talked very briefly about leveldb. Leveldb allows us to build modules that do persistence by accepting a leveldb instance.

Remember how we used a sequence number (the seq variable) and a random id in one of the previous exercises to make sure we wouldn't forward a message we'd already received?

We can use the same technique to persist messages with a sequence number to a database and then only forwarding stored messages to other peers if they have a higher sequence number then the latest one they have stored. We call this technique append-only logs since it's logs (or lists) that you only append to.

For example a peer could have and random id, foobar. Then when the peer stores his first message hello it would be stored with sequence number 1.

{id: 'foobar', seq: 1, message: 'hello'}

And when the peer stores a new message world it would be stored with the following sequence number 2

{id: 'foobar', seq: 1, message: 'hello'}
{id: 'foobar', seq: 2, message: 'world'}

When replicating with another peer, the other peer now simply needs to check its local database and see what its latest sequence number is for the foobar log and ask the other peer to send all messages with a sequence number larger than that. For example if another peer has {id: 'foobar', seq: 1} stored it would ask the first peer to send all foobar messages with a sequence number larger than 1 (the {id: 'foobar', seq: 2, message: 'world'} message)

Luckily there is already a module called scuttleup, https://github.com/mafintosh/scuttleup that implements this datastructure for you.

Tasks

Create two new programs, append.js and read.js.

append.js should create a new scuttleup that uses a leveldb named logs.db and append the value hello world to it. read.js should also create a new scuttleup that uses the same leveldb and reads out all the value stored in the logs.

Tips

You can create a new scuttleup by passing in a leveldb, var logs = scuttleup(level('logs.db')) and you can append to it by calling logs.append(message) and read from it by doing var stream = logs.createReadStream({valueEncoding: 'utf-8'}). To read from the stream attach a stream.on('data', function (data) { console.log(data) }) handler similar to what you did in an earlier exercise.

Testing

First run

node append.js

And then run

node read.js

The read program should now print out something similar to

{peer: 'a-log-id', seq: 1, entry: 'hello world'}

If you run append.js again and then read.js it should have appended a new value to your feed similar to

{peer: 'a-log-id', seq: 1, entry: 'hello world'}
{peer: 'a-log-id', seq: 2, entry: 'hello world'}

When you are done click here to go to the next exercise