Building a ranking system, or a leaderboard, can be useful in many types of applications: from ranking your players in a game, to showing the top selling products from your e-shop or prioritizing appointments based on multiple factors. This article will show the process of building the skeleton of such a system using Go and Redis.
You can find all the related code on GitHub and refer to it if anything is unclear about the code structure.
Prerequisites
To follow along, you will need:
· Go installed on your system.
If you have Redis installed, you can start the server by running:
redis-serverWith Docker, you can start a Redis container by running:
docker run -d -p 6379:6379 redis:latestRequirements
As mentioned in the introduction, a ranking system can be used for many types of applications. For the rest of this article, it will be assumed that you are ranking users based on a score that you compute through different means – you can assign a random score for the purposes of this tutorial, or you can use something arbitrary like the username length.
This leads to the first (and only) entity of the domain: the user. The user can be defined by a unique ID, a name, a score, and finally, a rank.
The system should, therefore, be able to store all this data for each user, and generate the rank based on the score attribute.
The system’s users should be able to perform a series of actions:
· Insert a new entry (user entity) into the system, by providing the name.
· Fetch the data for an existing user.
· Fetch the data for multiple users, using a pagination system (e.g., getting the top 10 ranked users).
Choosing the technologies
Redis is really the backbone of the system, as it is its sorted set that will be used for the actual ranking of the items. You can build the business logic using any language and framework you prefer.
Since the functionality of the system will be demonstrated using a web service, Go seems the most natural choice for the purposes of this article. The Gin web framework will also be used for building the web service.
Project architecture
Sorted sets allow the existence of two “fields” – the member that is being ordered (the user ID in the case of this system) and the score of each member. Another system or service is therefore needed for storing the other data – the username, in this case, but possibly much more information in more complex systems. In the end, there are three conceptual components in the system: the web service, the database and the ranking component. The overall architecture would look like this:
Choosing a database can depend on many factors when building an application, and since the purpose of this article is to build a ranking service, storing additional data is a bit outside the scope. Therefore, with Redis already present in the system, it will also be used for storing the user data. The simplified diagram looks like this:
What is more important in order to understand the system is the data flow:
1. A register request is sent, and a new user is created in the system. This means generating a unique ID for the user, saving it into Redis by the ID key, and adding it to the sorted set after calculating its score.
2. A fetch request is sent for an existing user, using its ID. In this case, the user data is fetched by the ID key, and then the ranking data is fetched from the sorted set. The data is merged and returned to the client.
3. A fetch request is sent for a ranking page, using an offset and a page size. In this case, the data is first fetched from the sorted set, and then the user data for each member of the set is fetched by the ID of each user. The data is then aggregated and returned to the client.
Creating the project
Navigate to an empty directory on your system and run the following command to create a new Go project:
go mod init go-redis-rankingThen create a new main.go file in the same directory. To verify that everything is running correctly up to this point, you can use the Go hello world example:
package main
import "fmt"
func main() {
fmt.Println("hello world")
}Running this should result in the “hello world” string being printed out:
go run .
hello worldNext, it’s time to set up the HTTP router for the web API. As mentioned before, the Gin framework will be used for this; the net/http package is also needed to import the constant HTTP responses. If you need more information on how Gin works, you can check their documentation.
For now, only one endpoint will be set up, which will return some static information in order to test that the API is working properly:
package main
import (
"net/http"
"github.com/gin-gonic/gin"
)
func main() {
router := gin.Default()
router.GET("/", about)
router.Run("127.0.0.1:8080")
}
func about(c *gin.Context) {
c.IndentedJSON(http.StatusOK, gin.H{"application": "Ranking System"})
}This snippet sets up a GET endpoint for the base / route, and binds that to the about function, which simply returns the name of the application as a JSON response. Then, the server is run on the 127.0.0.1 address, on port 8080.
To download Gin for your project, run:
go get .You can run the web service with the same go run . command; then, in another terminal, send a request to test that the endpoint is working:
curl -X GET http://127.0.0.1:8080
{
"application": "Ranking System"
}Registering a user
To register a new user, the client will send a POST request to the /register endpoint, containing the name of the new entity in the request’s body. The first thing needed for this is a struct to represent the request. In a new user.go file, create the following struct:
type userRequest struct {
Name string `json:"name"`
} Then, create a new function in the main.go file that represents the handler for the /register endpoint. This function will deserialize the request, save the data to Redis and return the user back to the client. For now, the second step can be omitted:
func register(c *gin.Context) {
var newUser userRequest
if err := c.BindJSON(&newUser); err != nil {
return
}
c.IndentedJSON(http.StatusCreated, newUser)
}The BindJSON call creates the newUser object based on the JSON body of the request, while the IndentedJSON creates a JSON response from the object.
Finally, register the function in the router by adding the following line to the main function:
router.POST("/register", register)After running the program again, you can send a request to the /register endpoint and expect to receive an identical response back:
curl -X POST http://localhost:8080/register -d '{"name":"test"}'
{
"name": "test"
} It’s time for the core of the project: saving the data to Redis. To find out more about how the Redis client for Go works, you can check out their repository and documentation.
In a new redis.go file, start by creating the context and client object, and importing the necessary packages:
import (
"context"
"github.com/go-redis/redis/v9"
)
var ctx = context.Background()
var rdb = redis.NewClient(&redis.Options{
Addr: "localhost:6379",
Password: "",
DB: 0,
})The client will be used for performing all Redis-related operations (e.g., adding and fetching user data). The address for Redis was set to localhost, but if you want to connect to a Redis server in another location, you can make the appropriate changes to the client object.
Next, create a addUser function that takes in the username and returns a user instance. This function will perform multiple operations:
· Generate a unique ID for the user (using google/uuid).
· Create the user object and serialize it to JSON (the length of the username will be used to calculate the score; you can use any other arbitrary data for the purposes of this tutorial).
· Save the user object to Redis on the ID key.
· Save the user score in a “rank” sorted set.
Some additional packages are needed for these operations; the final imports look like this:
import (
"context"
"encoding/json"
"github.com/google/uuid"
"github.com/go-redis/redis/v9"
)The user struct also needs to be created in the user.go file:
type user struct {
ID string `json:"id"`
Name string `json:"name"`
Rank int `json:"rank"`
Score int `json:"score"`
}Then, the addUser function would then look like this:
func addUser(name string) (user, error) {
id := uuid.NewString()
newUser := user{Name: name, ID: id, Score: len(name)}
serializedUser, err := json.Marshal(newUser)
if err != nil {
return user{}, err
}
err = rdb.Set(ctx, id, serializedUser, 0).Err()
if err != nil {
return user{}, err
}
err = rdb.ZAdd(ctx, "rank", redis.Z{Score: float64(newUser.Score), Member: id}).Err()
if err != nil {
return user{}, err
}
return newUser, nil
}As you can see, two inserts happen in Redis: first, the Set call, which sets the value for the ID key to the serialized user object, and then the ZAdd call, which sets the score for the ID on the rank sorted set.
There is an issue with the current function, however: it doesn’t return the user’s rank. To get the rank of a set member (with the data sorted in descending order), a ZRevRank call has to be made. The final function looks like this:
func addUser(name string) (user, error) {
id := uuid.NewString()
newUser := user{Name: name, ID: id, Score: len(name)}
serializedUser, err := json.Marshal(newUser)
if err != nil {
return user{}, err
}
err = rdb.Set(ctx, id, serializedUser, 0).Err()
if err != nil {
return user{}, err
}
err = rdb.ZAdd(ctx, "rank", redis.Z{Score: float64(newUser.Score), Member: id}).Err()
if err != nil {
return user{}, err
}
rank, err := rdb.ZRevRank(ctx, "rank", id).Result()
if err != nil {
return user{}, err
}
newUser.Rank = int(rank + 1)
return newUser, nil
}Finally, this function needs to be called from the register handler. If any error is returned, a 500 Internal Server Error response can be sent back to the client; otherwise, the user object will be returned; the final register function looks like this:
func register(c *gin.Context) {
var newUser userRequest
if err := c.BindJSON(&newUser); err != nil {
return
}
user, err := addUser(newUser.Name)
if err != nil {
c.IndentedJSON(http.StatusInternalServerError, gin.H{"message": "Internal Server Error"})
return
}
c.IndentedJSON(http.StatusCreated, user)
}If you have not started Redis locally, this would be the time to do so by running one of the commands in the Prerequisites section. Then, run the program again and repeat the /register request a couple of times (using different names for the user). The responses should look like this:
curl -X POST http://localhost:8080/register -d '{"name":"test"}'
{
"id": "e309ab7e-6e02-4b60-a374-52cd5c2a41dd",
"name": "user",
"rank": 6,
"score": 4
} Fetching user data
Clients might want to get an existing user’s data after registering it, as data can change based on other users’ scores. Therefore, a /fetch endpoint is needed: it will take in the user id and return the data found at that key, as well as the data from the sorted set.
Start by creating a new getUserData function in the redis.go file that takes in an id and returns a user object. This function will perform a Get call to retrieve the user data based on the ID, and then a ZRevRank call to get the user’s rank:
func getUserData(id string) (user, error) {
userData, err := rdb.Get(ctx, id).Result()
if err != nil {
return user{}, err
}
var existingUser user
err = json.Unmarshal([]byte(userData), &existingUser)
if err != nil {
return user{}, err
}
rank, err := rdb.ZRevRank(ctx, "rank", id).Result()
if err != nil {
return user{}, err
}
existingUser.Rank = int(rank + 1)
return existingUser, nil
}Since the ZRevRank call is duplicated in the addUser function, it can be extracted to a new getUserRank function:
func getUserRank(id string) (int, error) {
rank, err := rdb.ZRevRank(ctx, "rank", id).Result()
if err != nil {
return -1, err
}
return int(rank + 1), nil
}The calls from addUser and getUserData would then look like this:
rank, err := getUserRank(id)
if err != nil {
return user{}, err
}
existingUser.Rank = int(rank)Then, in the main.go file, create a new getUser handler. This will get the ID from the query string parameters, call the getUserData function and return the result of that function to the client:
func getUser(c *gin.Context) {
id := c.Query("id")
if id == "" {
c.IndentedJSON(http.StatusBadRequest, gin.H{"message": "`id` is required"})
return
}
userData, err := getUserData(id)
if err != nil {
c.IndentedJSON(http.StatusInternalServerError, gin.H{"message": "Internal Server Error"})
return
}
c.IndentedJSON(http.StatusOK, userData)
}Finally, register the handler in the router:
router.GET("/rank", getUser)Run the server again, and call the new endpoint with a previously created user’s id:
router.GET("/rank", getUser)Run the program again and send a request to the new endpoint:
curl -X GET “http://localhost:8080/rank?id=e309ab7e-6e02-4b60-a374-52cd5c2a41dd”
{
"id": "e309ab7e-6e02-4b60-a374-52cd5c2a41dd",
"name": "test",
"rank": 7,
"score": 4
} You can play around with this by adding more users with higher/lower scores and checking how the ranks change by calling the GET /rank endpoint.
Fetching a ranking page
Building a ranking system/leaderboard is not fun unless you let the users compare to the others. The final functionality that will be added in this tutorial will let clients get a subset (page) of the users ranking by providing an offset (starting position of the page), and a limit (the page size).
Start by creating a new getRedisRanks function in the redis.go file that takes in the offset and limit parameters, and returns an array of user objects. In this function, the sorted set will need to be queried first, in order to get the user ids. For this, a ZRange will be made, with the following parameters:
· Start at offset – 1 (since the sorted set is 0-based).
· Stop at offset + limit – 2 (to get the right length of the page, since sorted sets ranges are inclusive).
· Sort the data in reverse order (the highest score should be in the first place).
· Add the scores to the result of the query.
Then, looping through the result, a Get call can be made based on each user id, to gather the other user data, deserialize it and append it to the resulting array. The final function looks like this:
func getRedisRanks(offset, limit int) ([]user, error) {
cmd := rdb.ZRangeArgsWithScores(ctx, redis.ZRangeArgs{
Key: "rank",
Start: offset - 1,
Stop: offset + limit - 2,
Rev: true,
ByLex: false,
ByScore: false,
})
log.Printf("Command: %v", cmd.String())
users, err := cmd.Result()
if err != nil {
log.Printf("Error: %v", err)
return nil, err
}
log.Printf("Users: %v", users)
var result []user
for rank, userScore := range users {
id := userScore.Member.(string)
userData, err := rdb.Get(ctx, id).Result()
if err != nil {
return nil, err
}
var constructedUser user
err = json.Unmarshal([]byte(userData), &constructedUser)
if err != nil {
log.Printf("Error: %v", err)
return nil, err
}
result = append(result, user{ID: id, Name: constructedUser.Name, Score: int(userScore.Score), Rank: rank + offset})
}
return result, nil
}Then, in the main.go file, create a new getRankPage handler. This will get the limit and offset from the query string parameters, gather the data for each user on the ranking page, and return the list to the client. The query string parameters are not required and omitting them will result in the first 10 positions being returned – you can tweak this to your liking. Since the query string parameters come in as strings, they will need to be converted to integers, so the strconv package will be imported. The final imports for the main.go file look like this:
import (
"net/http"
"github.com/gin-gonic/gin"
"strconv"
)And the getRankPage function looks like this:
func getRankPage(c *gin.Context) {
offset, err := strconv.Atoi(c.DefaultQuery("offset", "1"))
if err != nil {
c.IndentedJSON(http.StatusBadRequest, gin.H{"message": "`start` must be an integer"})
return
}
limit, err := strconv.Atoi(c.DefaultQuery("limit", "10"))
if err != nil {
c.IndentedJSON(http.StatusBadRequest, gin.H{"message": "`limit` must be an integer"})
return
}
result, err := getRedisRanks(offset, limit)
if err != nil {
c.IndentedJSON(http.StatusInternalServerError, gin.H{"message": "Internal Server Error"})
return
}
c.IndentedJSON(http.StatusOK, result)
}Finally, register the handler in the router:
router.GET("/ranks", getRankPage)You can now play around with the parameters to get subsets of the ranking list. For example, to get the top 3 members of the list, you can call:
curl -X GET http://localhost:8080/ranks?limit=3
[
{
"id": "0a363580-98f9-4d4c-b57b-e30259807871",
"name": "AUserWithQuiteALongName",
"rank": 1,
"score": 23
},
{
"id": "629e846e-7d8c-4a4a-b4dc-72c98b3038cc",
"name": "AUserWithAShorterName",
"rank": 2,
"score": 21
},
{
"id": "06894035-f00b-439f-9837-8096af59de51",
"name": "AnEvenShorterName",
"rank": 3,
"score": 17
}
]Conclusion
If you have gone through all the code presented in the article, you have successfully built a minimalistic ranking system using Go and Redis. This can be furtherly enhanced based on your specific scenarios by adding more data to the user representation, or by querying the sorted set in different ways (e.g., by getting data within certain score ranges using the BYSCORE attribute).
For any issues with the code, you can always refer back to the repository link in the introduction section.
Comments