11 - Distributing append-only logs

In the previous exercise we learned that we can use something like scuttleup and leveldb to store append-only logs in node. Now the time has come to turn this into a chat application that supports persisted and distributed chat.

We are gonna build on most of code we wrote in exercise 8 where we created a chat app that found other peers using multicast-dns and replicated new messages to other peers.

Your solution to that task might look similar to this

require('lookup-multicast-dns/global')
var topology = require('fully-connected-topology')
var jsonStream = require('duplex-json-stream')
var streamSet = require('stream-set')
var toPort = require('hash-to-port')
var register = require('register-multicast-dns')

var me = process.argv[2]
var peers = process.argv.slice(3)

var swarm = topology(toAddress(me), peers.map(toAddress))
var connections = streamSet()
var received = {}

register(me)

swarm.on('connection', function (socket, id) {
  console.log('info> direct connection to', id)

  socket = jsonStream(socket)
  socket.on('data', function (data) {
    if (data.seq <= received[data.from]) return // already received this one
    received[data.from] = data.seq
    console.log(data.username + '> ' + data.message)
    connections.forEach(function (socket) {
      socket.write(data)
    })
  })

  connections.add(socket)
})

var seq = 0
var id = Math.random()

process.stdin.on('data', function (data) {
  connections.forEach(function (socket) {
    var message = data.toString().trim()
    socket.write({from: id, seq: seq++, username: me, message: message})
  })
})

function toAddress (name) {
  return name + '.local:' + toPort(name)
}

Luckily scuttleup already has replication build in so we can reuse the swarm code to replicate our append only logs in the following way

var logs = scuttleup(level(me + '.db')) // use a database per user
swarm.on('connection', function (socket, id) {
  console.log('info> direct connection to', id)
  socket.pipe(logs.createReplicationStream({live: true})).pipe(socket)
})

To read out all messages from the logs we can do

logs.createReadStream({live: true})
  .on('data', function (data) {
    console.log(data)
  })

And to append a message to the log we can do

logs.append('hello from me')

Tasks

Adapt the above snippet (save it as persisted-chat.js) to use scuttleup to distribute change messages passed in using stdin.

Testing

Run one process

node {your-name} {another-name}

And type hello. Then afterwards open a new terminal and do

node {another-name} {your-name}

You should now see that hello message being replicated to your new terminal. This is the final exercise. If you finish this, congratulations! You're awesome. You should ping one of the organizers to get some more exercises.