How to Add Message History to your Pusher Apps

Pusher makes it really easy to add real-time data and functionality to web, mobile and IoT apps. For use cases such as notifications and signalling, that data can be transient, where the data is only relevant for a short period of time. In other cases, such as activity streams or chat, data may need to be persisted and historical data is as important as the data delivered via real-time updates.

So, in this post I’m going to cover:

• How to create a solution to retrieve a history of messages
• How to retrieve messages missed during periods of network instability

This is a lot easier than it might seem. The events the Pusher libraries expose make the process easy to follow and provide a lot of flexibility when integrating with your persistence mechanism of choice.

Getting Started

For this tutorial I’m not going to create an application from scratch. Instead we’re going to start with a chat application built using React and Python for a previous tutorial. If you’d like to start from scratch you can do so by first reading the real-time chat with React and Pusher tutorial.

Assuming you’re not starting from scratch, you can get the starting code for this tutorial by running the following on your command line:

git clone git@github.com:pusher-community/react-realtime-chat.git
cd react-realtime-chat
git checkout message-history
git reset --hard be4b1f84235e7e5303a54ad6e5dd548b0f385184
git checkout -b start

Finally, make sure you can run the application by following these steps in the README.

If you get stuck at any point you can always take a look at the source.

Message History

Being able to see a history of messages within a chat application provides context to the ongoing conversation. When you think about how you would add this functionality there are three choices that generally come to mind:

So, we’re going to fetch the historical messages from the server after we have successfully subscribed to the channel. The basic flow to achieve this is:

1. The page loads
2. Connect to Pusher
3. Subscribe to the channel
4. Bind to the pusher:subscription_succeeded event on the channel
5. Bind to the new_message event to be notified when any new chat messages arrive via Pusher
6. Upon successful subscription, retrieve the messages from the server
7. Add the messages to the UI, ensuring that those messages are sorted and no duplicates exist from any messages that may have come in via Pusher during the retrieval

If you’re a visual person the following sequence diagram may help:

Retrieve Missed Messages Upon Subscription

Within the code for the React Chat application, open up static/javascripts/ui_components/MainView.jsx and update the componentDidMount function to bind to the pusher:subscription_succeeded event, passing in a reference to a retrieveHistory handler function:

componentDidMount: function() {

    this.pusher = new Pusher(PUSHER_CHAT_APP_KEY);
    this.chatRoom = this.pusher.subscribe('messages');
    this.chatRoom.bind('pusher:subscription_succeeded',
                       this.retrieveHistory, this);

    // existing code
},

Next, define the retrieveHistory function. Within it we’ll make an AJAX call to retrieve the chat message history:

retrieveHistory: function() {
  var self = this;
  $.get('/messages').success(function(response) {
    // TODO: process response
    console.log(response);
  });
},

If you run the application now you’ll get a 405 (METHOD NOT ALLOWED) error logged to your JavaScript console due to the above request. This is because app.py only allows the /messages endpoint to respond to POST requests.

To resolve this we need to update the app.py to firstly save data to a database and then to respond to a GET request to the /messages endpoint by returning all existing chat messages.

If you started from the existing application you’ll already have a messages_repo.py that provides some of the database functionality we require. If you started from scratch you’ll need to download messages_repo.py and put it in the root of the application. Let’s include the class it defines into app.py and create a new instance:

from messages_repo import MessagesRepo

repo = MessagesRepo()

Now we can update the existing new_message function in app.py to save messages to the database.

@app.route('/messages', methods=['POST'])
def new_message():
  name, text = cgi.escape(request.form['name']),  cgi.escape(request.form['text'])

  message = repo.create(name, text)

  try:
      pusher.trigger('messages', 'new_message', message)
  except Exception as e:
      print("Error triggering the event via Pusher %s" % e)

  return jsonify(message)

The repo.create function returns a message object with all the properties we need for the event payload we send to connected clients via Pusher.

Now that the messages are being stored we can respond to the currently unhandled GET /messages request with the existing messages. Add the following function to app.py:

@app.route('/messages', methods=['GET'])
def get_messages():
    messages = repo.get_all();
    return jsonify(messages)

Note: you may need to from flask import jsonify

Running the web client should now result in a the expected log to the console.

Adding Historical Messages to the UI

Now that we’ve got the existing chat messages being returned to the client we need to show them in the UI.

Our application is now receiving chat messages in two places; from Pusher via an inline function bound to the new_message event and via the AJAX request to GET /messages. Let’s refactor the inline function so that it can be reused.

Firstly, extract the inline handler into a addMessage function:

componentDidMount: function() {

  this.chatRoom.bind('new_message', this.addMessage, this);

  // existing code...
},

addMessage: function(message){
  this.setState({messages: this.state.messages.concat(message)})

  $("#message-list").scrollTop($("#message-list")[0].scrollHeight);
},

Then update the retrieveHistory function to make use of that function:

retrieveHistory: function() {
  var self = this;
  $.get('/messages').success(function(response) {
    response.results.forEach(self.addMessage, self);
  });
},

response.results is an Array that we can loop over and make use of the new addMessage function to handle each individual historical message.

Running the front-end application will now retrieve the existing messages from the server in addition to showing real-time updates.

Handling Important Edge Cases

In addition to adding them to the UI, we need to check two things since we could receive a message from Pusher whilst the AJAX request is taking place:

1. That duplicates don’t occur
2. The ordering of messages in the UI is correct

We can easily avoid duplicates by checking the id property of each message and ensuring this value is unique within the collection of messages that we have:

messageExists: function(message) {
  var getId = function(e) { return e.id; };
  var ids = this.state.messages.map(getId);
  return ids.indexOf(message.id) !== -1;
},

addMessage: function(message) {
  if(this.messageExists(message)) {
    console.warn('Duplicate message detected');
    return;
  }

  this.setState({messages: this.state.messages.concat(message)})

  $("#message-list").scrollTop($("#message-list")[0].scrollHeight);
},

In the code above we’ve created a messageExists helper function to make sure each message is unique. If a duplicate is detected we discard it.

To ensure the messages are in order we can use Array.sort within the addMessage function:

addMessage: function(message) {
  if(this.messageExists(message)) {
    console.warn('Duplicate message detected');
    return;
  }

  var messages = this.state.messages.concat(message);
  messages.sort(function(a, b) {
    return (a.time > b.time);
  });
  this.setState({messages: messages})

  $("#message-list").scrollTop($("#message-list")[0].scrollHeight);
},

With these two changes in place we can make sure that we avoid duplicate messages and that the messages are always ordered correctly, from first to last in the UI.

Fetching Missed Messages Upon Reconnection

Sometimes a client will lose Internet connectivity. When that happens new chat messages won’t be able to be delivered to that client. So, it’s important to be able to retrieve any missed messages upon reconnection.

The Pusher JavaScript library does expose events that allow your application to monitor connectivity state. But, whilst it’s certainly valuable to use these events to provide the user with feedback about the connection state, to achieve the functionality we’re looking for we don’t actually need to use this. That’s because the Pusher library will automatically reconnect when possible and re-subscribe to the channels it was previously subscribed to. When these subscriptions succeed the pusher:subscription_succeeded event will be triggered and we’ve already got a function in place to handled that event.

The steps for this are very similar to that of retrieving a history of messages upon page load:

1. The library reconnects to Pusher
2. The library re-subscribe to the channel
3. When the pusher:subscription_succeeded event is triggered, retrieve any missed messages from the server by passing the ID of the last received message
4. Add the messages to the UI, ensuring that those messages are sorted and no duplicates exist from any messages that may have come in via Pusher during the retrieval

The full sequence diagram looks as follows:

Right now the retrieveHistory function will fetch the entire message history. If we’ve already got a number of messages it’s more efficient to only retrieve messages that have been sent since the last one that we’ve got.

Let’s update the client to call the /messages endpoint and pass the id of the last message we’ve got:

retrieveHistory: function() {
  var self = this;
  var lastMessage = this.state.messages[this.state.messages.length-1];
  var lastId = (lastMessage? lastMessage.id : 0);
  $.get('/messages', {after_id: lastId}).success(function(response) {
    response.results.forEach(self.addMessage, self);
  });
},

To do this we’ll attempt to get the last message from the this.state.messages Array. If a message is found we’ll get the .id and pass that value to the /messages endpoint as a after_id query parameter e.g. /messages?after_id=5. If no messages is found we’re defaulting to sending 0, which will fetch all messages (that’s how this endpoint works, but yours might prefer the after_id parameter not to be sent at all).

Next we need to update the server to use the after_id value.

@app.route('/messages', methods=['GET'])
def get_messages():
    after_id = request.args.get('after_id', 0)
    messages = repo.get_all(after_id);
    return jsonify(messages)

Now, if a client goes offline for any period of time it will fetch any missed messages when it reconnects again.

Conclusion

In around 50 lines of client and server code we’ve added message history to an existing real-time chat application.

By following the techniques in this tutorial I hope you can see just how easy it is to use the events that the Pusher JavaScript library (and other Pusher WebSocket libraries) expose to add historical data to an application, and also fetch any missed messages during periods of network instability.