How to separate frontend and backend without losing a common ground

How to change the architecture of a monolith product to accelerate its development and how to divide a team into several, having keeping a working coherence? We decided to create a new API as a solution to these issues. Please read the details of this story and check a short overview of the stack of technologies we’ve chosen. But at first, let me make a small lyrical digression.

A few years ago I read some research article about a modern process of education. It stated that a full course of education requires more and more time, and in the near future people would spend 80 years of their lives for knowledge acquisition. It seems that in the IT industry the future already arrived.

I was lucky to start programming at the time when there was no division into a backend and frontend-programmers, when people didn’t use such words as a "prototype", "product-manager", "UX" and "QA". The world was simple, trees were high and green, and the yards were full of playing children rather than parked cars. I wish I could get back to this time, however, I should admit that this was not a plot of an evil genius, but just an evolution of society. Yes, the evolution could go a different way, but history doesn't like the subjunctive mood.

BILLmanager appeared just in those days when there was no a strict division on different directions. It had the coordinated architecture, was able to control user behavior with a huge potential of expending functionality with plug-ins. However, after a few years of developing the product we had noticed some strange things started to happen. For example, if a developer was engaged in business logic, his skills in creating intuitive and convenient forms were worsening. Sometimes adding a simple feature might take several weeks because of the architecture: all the modules had a tight connection with each other so changing one module required correcting several other.

Developers had to forget about convenience, ergonomics and global development of the product some unknown error caused BILLmanager failing. In the early years, a developer could perform different tasks to develop the product in different directions. However, the product grew bigger the requirements became higher and it became impossible. The developer had a common picture and understood that promoting the product, improving buttons, forms, tests would be useless if the product didn’t work properly. That is why he put all his current tasks away to fix the error. Such small deeds usually stayed unnoticed by users because nobody had time to promote them. In order to improve bringing new features and fixes to the clients, we divided our team into several smaller groups: frontend and backend, testing, design, support, promotion.

However, it was only a beginning. We had changed our team, but the product stayed the same. The architecture was sophisticated and the elements were highly correlated. In such circumstances, we couldn’t develop the product with the pace we wanted: even a small change in the interface might require changing the backend logic although the data structure stayed the same. We needed to do something.

It takes a lot of time and money to become a professional in everything, therefore nowadays it is a common practice to divide applied developers into two types, the first works with a frontend while the rest work on a backend.

Frontend developers are responsible for the user interface while backenders can focus on business logic, data models and other things under the hood. It’s important to notice that front-enders, back-enders, QA engineers and designers stay in the same team because they work on the same product focussing on the different parts on it. Staying in the team means having common information and, preferably, physical space; discuss features together and solve cases; coordinate the working process between the divisions.

The above-mentioned principles would be enough for some new project, but we already had a ready product and its backlog and roadmap meant that we had to have several teams. A basketball team has 5 players, a football one - 11, while we have more than 30. An ideal team for scrum has 5-9 members so we had to split the teams but keep coordination between them. We had to solve two main issues to move forward: architecture and coordination.

When a product becomes obsolete outdated, it seems logical to drop it and create a new one. This is a good decision if you can forecast the timeline that will work for everybody. However, in our case, developing a new product would take years. And besides, the more differences between the versions - the more different migration between them. Backward compatibility is very important for our clients and if we fail to provide it - many users will refuse to upgrade. This is why we decided not to create the product from a scratch, but to bring the architecture of the existing product up to date and provide the maximum level of backward compatibility.

Our application had a monolith architecture and its interface was built on the server side so front-end could only carry out its instructions. In other words, the user interface depended on back-end more than on front-end. From the architecture standpoint, both front-end and back-end worked as a single mechanism so we couldn’t divide front-end and back-end tasks. And the worst part - it was impossible to develop the user-interface without having a deep knowledge of the processes carried out on the server.

We had to divide frontend and backend to create different application. It was the only way to catch the schedule and produce the code in a needed volume. But how can we work on the two projects and change its structure if they have a high level of interdependence?

Our solution was adding an additional system — a middleware. The idea behind the middleware was very easy: it had to coordinate frontend and backend works and carry take additional expenses if needed. For example, we wanted the middleware to combine the data when we decompose the payment function on the backend while frontend shouldn’t be affected. Another example is a data aggregation in the middleware without creating an additional function on the backend for showing all the services ordered by a user on a dashboard.

Another function of the middleware is giving us more confidence in the results of calling a specific function from a server. We want to make it possible to make a call of operations without knowing how does the functions that perform it work.

Due to the strong dependence between frontend and a backend, it was impossible for us to organize a parallel working process of the two parts of the team and it slowed down both of them. Dividing the big project into several smaller ones would give us more flexibility for every project, however, it was necessary to keep coherence in work.

Some people believe that improving soft skills will allow reaching the coherence. It's true, however, it is not a panacea. Think about road traffic, it is important for drivers to be mutually polite, to be able to avoid obstacles on their way and to help each other in case of an accident. But if we don't have traffic regulation laws we will have a car crash at every corner and a high risk of reaching a destination point.

We needed the rules which would be easier to follow than break. However, bringing new rules in the working process always implies some additional costs and can slow it down and of course, we'd like to avoid it. Therefore we created a special coordination group that evolved into a separate team later. This team is responsible for creating the best conditions for successful working on different areas of our product. They configured the interfaces which allowed different teams to work as a single mechanism, giving us the rules that were easier to follow that break.

We call this team "API" although creating a new API is only a small part of their tasks. When common pieces of the code are designed to form a separate function, API team jumps in to solve the issues of different product teams. And this is the moment when our frontend and backend get connected so API team have to understand specific aspects of every team involved in the working process.

Perhaps, "API" is not the best name for this team. Maybe the name should reflect something about architecture or global vision, but we are okay with the current name as it doesn't influence their work much.

We had the interface for accessing the server functions in our initial product, but for the end user it looked chaotic. We had to bring more definiteness when we divided frontend and backend.

The tasks for the new API were generated from daily difficulties in realizing new product and design ideas. We needed:

In order to define the new solution for API, we thought what would the users want.

Any REST API is the most widespread approach. In a few recent years, different developers add descriptive models via the tools like swagger, however, it is the same REST still. Its main advantages and disadvantage at the same is a descriptive nature of the rules. Nobody can stop the developer of such API to deviate REST postulates when working on different parts of a project.

Another widespread solution is GraphQL. It is not ideal either, but unlike REST, GraphQL API is not just a descriptive model, but real rules.

I've already mentioned the system for coordinating frontend and backend works. An interlayer represents this intermediate level. We've considered several possible options of work with the server and decided to go with GraphQL as API for frontend. However, since our backend is based on C++, the realization of the GraphQL-server was an uncommon task. I won't describe here all the difficulties we faced and tricks we made to overcome them because it did not bring us real results. We decided to take a different angle of looking on the issue and found out that simplicity is our key. Therefore chose the time-proof solutions: a separate Node.js server with Express.js and Apollo Server.

The further task was defining the way of talking with the backend API. At first, we thought about using the REST API, then we tried to use C++ addons for Node.js. In the end, we found out that both solutions didn't fit our needs so we made a deeper analysis and chose gRPC-services based API for the backend.

We aggregated all our experience in C++, TypeScript, GraphQL and gRPC and created the new product architecture, giving us enough flexibility for working in parallel on the backend and frontend, allowing the teams develop the product together.

Here is the scheme where frontend interacts with the intermediate server by GraphQL requests. The GraphQL-server calls API functions of the gRPC-server in resolvers using Protobuf-schemes for communication. gRPC based API server knows the microservice he needs to reach and where to send the received inquiry. The microservices themselves are also based on gRPC that improves the speed of inquiries processing, data typification and give us a possibility of using various programming languages for developing new microservices.

This approach has also some drawbacks as well. The main is additional work on adjusting and configuring schemes and also writing back up functions. However, these costs will pay off when we have more users of API.

We chose the evolutionary way of developing our team and product. It's too early to say whether this idea was successful or not, but we can sum up some intermediate results already. That’s what we have now:

I believe that our main achievement is an opportunity to consciously develop the team and the project. We managed to create the working environment where every team member can train his skills and raise competences and stay focused on current tasks without spraying attention. Everybody is in charge of his part of work and he can work without getting distracted. It is impossible to be a professional in everything, but it is not necessary for us now.

The article turned out very general. I wanted to show the way and results of difficult research on changing the architecture to continue the product development. Another idea was revealing the organizational difficulties of dividing the team into several separate units without losing the coherence. I have also covered some aspects of a mutual working on the single product, choosing the API technology (REST vs GraphQL), connection of Node.js app with C++. All these topics deserve a separate article and we will make it if we see your interest.