pip install requests==2.17.3 beautifulsoup4==4.6.0 redis==2.10.5 twilio==6.3.0 Flask==0.12.2

We’ll dive into using each one of these libraries as we get further along.

Scraping the Registration System

We need to write a program that can determine whether there are seats available in a given course. To do this, we’ll use a technique called web scraping, in which we download a page from the internet and find the important bits. Two popular libraries that make this easy are Requests and BeautifulSoup. Requests makes it easy to get a web page, and BeautifulSoup can help us find the parts of that page that are important to us.

# scraper.py
import requests
from bs4 import BeautifulSoup

URL = 'http://courses.project.samueltaylor.org/'
COURSE_NUM_NDX = 0
SEATS_NDX = 1

def get_open_seats():
    r = requests.get(URL)
    soup = BeautifulSoup(r.text, 'html.parser')
    courses = {}

    for row in soup.find_all('tr'):
        cols = [e.text for e in row.find_all('td')]
        if cols:
            courses[cols[COURSE_NUM_NDX]] = int(cols[SEATS_NDX])
    return courses

The meat here is in the get_open_seats function. In this function, we use requests.get to download a page’s HTML source, then we parse it with BeautifulSoup. We use find_all('tr') to get all the rows in the table, updating the courses dictionary to indicate the number of seats available in a given course. find_all can be used more powerfully than this simple example, so check out the documentation if you’re interested in learning more. Finally, we return the courses dictionary so that our program can look up how many seats are in a given course (i.e. courses['CS 101'] is the number of seats available in CS 101).

Hooray, now we can determine whether a course has open seats. A great way to test this function out is in the Python interpreter. Save this code into a file called scraper.py, then run the script and drop into interactive mode to see what this function does:

$ python -i scraper.py
>>> get_open_seats()
{'CS 101': 1, 'CS 201': 0}

While this is great, we’re not quite to a solution; we still need some way to notify users when a seat opens up. Twilio SMS to the rescue!

Getting Updates via SMS

When building a user interface, we want simple things to be simple. In this case, users want to get notified when seats in a course open up. The simplest way for them to communicate that intent to us is sharing the course number. Let’s implement a subscription functionality by setting up and handling a webhook. I’m choosing to use Redis (a tool which provides data structures that can be accessed from multiple processes) to store subscriptions.

# sms_handler.py
from flask import Flask, request
import redis
 
twilio_account_sid = 'ACXXXXX'
redis_client = redis.StrictRedis(host='localhost', port=6379, db=0)
app = Flask(__name__)
 
@app.route('/sms', methods=['POST'])
def handle_sms():
    user = request.form['From']
    course = request.form['Body'].strip().upper()
 
    redis_client.sadd(course, user.encode('utf-8'))
 
if __name__ == '__main__':
    app.run(debug=True)

Here we use a web framework for Python called Flask to create a little service that handles SMS messages. After some initial setup, we indicate that requests to the /sms endpoint should be handled by the handle_sms function. In this function, we grab the user’s phone number and the course they were looking for, and store them in a set named after the course.

This is great as far as capturing the subscriptions, but it is a frustrating user interface because it doesn’t provide any feedback to users. We want to get back to users and tell them whether we’re able to service their request as soon as possible. To do that, we’ll provide a TwiML response. The additional lines needed for that are highlighted below.

# sms_handler.py
from flask import Flask, request
import redis
from twilio.twiml.messaging_response import MessagingResponse
 
twilio_account_sid = 'ACXXXXX'
my_number = '+1XXXXXXXXXX'
valid_courses = {'CS 101', 'CS 201'}
 
redis_client = redis.StrictRedis(host='localhost', port=6379, db=0)
app = Flask(__name__)
 
def respond(user, body):
    response = MessagingResponse()
    response.message(body=body)
    return str(response)
 
@app.route('/sms', methods=['POST'])
def handle_sms():
    user = request.form['From']
    course = request.form['Body'].strip().upper()
    if course not in valid_courses:
        return respond(user, body="Hm, that doesn't look like a valid course. Try something like 'CS 101'.")
 
    redis_client.sadd(course, user.encode('utf-8'))
        return respond(user, body=f"Sweet action. We'll let you know when there are seats available in {course}")
 
if __name__ == '__main__':
    app.run(debug=True)

We’ve made two major changes in the code above. First, we validate that the user is asking for a valid course. Second, we respond to users when they ask for updates. In the respond function, we construct a TwiML response to a given number with a given message.

Make sure to install Redis and start it up with the redis-server command. Save the above code into a file called sms_handler.py and then run python sms_handler.py.

Admittedly, the response messages here are a bit silly, but I was surprised to see how much users enjoyed them. In some contexts a personal touch can make for a better user experience.

Let’s extend our earlier scraping script to actually notify those people now that we know who wants to be notified of a course opening up.