May 25, 2020
There is a lot of great public content on the internet about development processes, methodologies, frameworks, best practices, a.s.o. and I will not try to describe the same things again with different words. What I’ll be doing instead will be to present some of the real-life challenges we’ve had with software building activities in one of our recent projects and the solutions the team came up with, to improve the process.
The project, in a nutshell:
I will be covering the challenges we had in each project phase:
One of the things you often hear about SCRUM is that if you get outside of the frameworks’ rules just for a bit, you’re not doing SCRUM at all. For example, “let’s skip stand-up just for today, because we have a lot to work on” or “we only have half an hour available for planning” or “we have already talked about everything, no need to retrospect”.
Things that we have struggled with:
Things were very different between pre-MVP (before the first release) and post-MVP development cycles. In pre-MVP, we had a smaller team and it took a while to get the needed discipline. We redefined at least a few times the way we were estimating. The main pain point was not being able to hold to the sprint commitments, and that was caused mainly by poor estimations.
We tried reference stories, ideal hours, but eventually agreed to drop these constraints, created some guidelines about points per complexity, stick to points and Fibonacci numbers:
e.g. 1 point is something trivial that does not need investigation, 5 points is something that is complex, but can be handled by one developer in one sprint and 8 or more points needed to break down.
Later in the project, as the team grew and had to split it, we continued the process. But since the teams were already formed and naturally assembled, with developers aggregated based on personality compatibility, timezone, etc., the project’s complexity increased and the main challenge transformed from poor estimations to stories added in the middle of the sprint (mid-sprint additions), due to various reasons: unforeseen dependencies, user feedback that was critical to the business, production bugs, etc.
So the problem remained: we were not landing sprints. However, as soon as the team committed to maintaining the SCRUM principles and ensure full transparency e.g. re-prioritizing and labeling not-committed items based on the principle “add something, then pull out something else”, we quickly started to see good results. But then, with a growing team, a new kind of problem appeared: coordination. I’ll talk about this next.
A squad is just another name for a SCRUM team. When multiple teams are working on the same project, sooner or later they will have to coordinate on some functionality. In our case the initial two teams were working on the same code base, but for different functionalities of the project. However, even though each team had their own agenda, the teams shared a common goal: the next release.
Early on, it was challenging to keep everyone in sync. So this is what we did to address the problem:
Continuing on the sprint-landing issues, even if we were “development ready” two days before the sprint ended, there was still not enough time to get the features in a “Done” state. This was because, on one hand, developers can make mistakes and code may have bugs, and on the other hand, there is often a bottleneck when features are tested by QA and need to have them accepted by Product Owners.
We focused on solving these issues by:
Things that we set up to increase code quality:
Things really started to become difficult when we transitioned from a once-in-a-while release to a continuously releasable solution. The former was a straightforward process that involved a code freeze before the release, testing, fixing for two days, and releasing. The code freeze was needed because after regression testing, all found bugs had to be validated in the lower environments first, but we didn’t want the validation to interfere with any work in progress. Since the releases were only once in a while, this flow was acceptable.
Here is a diagram showing the simple CI/CD pipeline initially used:
Developers created their feature branches from master (considered stable), tested the feature locally, merged it into dev which automatically deployed it into the Testing environment. Initial pass was done by developers (validating that it was integrated well with work from others and solving potential conflicts). The QA team would then test the feature in the Testing environment and when validated, the feature branch would be merged by developers into the master (solving potential conflicts again) which would automatically deploy to UAT. At the end of the sprint, all features that got into the master branch were considered releasable (in green), and the code freeze began. If a bug was found, it would be treated just like any other feature.
Later on, when we started to release at the end of each sprint, code freeze was not an option anymore. Imagine at least two days of code freeze every sprint and half of the developers being blocked from integrating code from other developers. So we upgraded to a new process that allowed us to continuously work on features while being able to release Done features at any moment.
Here is a simplified diagram of what it looked like:
So what happens in this diagram? The developer tests on both the local environment and Dev testing (ideally automation tests are updated as part of the local testing, which would cover much of the second testing phase), hands it over to the QA team which tests it in the QA testing environment. When everything is approved by QA, code gets pushed to UAT for acceptance by Product Owners. After being accepted, the release manager will coordinate with Product Owners and stakeholders for the set of features to be delivered on the next release, building the Release environment with those. Once the release environment has been validated (for us it was always a joint effort by QA team and Product Owners), a Release in Production is scheduled.
One of the problems we had to solve was how to push independent features.
One thing we did was to make sure that the dev branch never gets merged in the master branch. Secondly, the master branch was the base for all feature branches. The Dev branch was considered “dirty”, meaning that it contained potentially harmful code, since it was not fully tested, as integrated with other code.
The Master branch would always get deployed to the UAT environment, by simply merging the feature branches into it.
With the other environments we had to take another approach in order to be able to control the code that gets in.
For example, for the QA Testing and Release (therefore implicitly for Production as well), we did a cherry-pick based approach with auxiliary branches.
|1.||SCRUM is here to help, not get in your way:
|2.||Team alignment: have team members meet often (virtual meetings are okay) and organize public code reviews;|
|3.||Aim for code quality by enforcing constraints and have strict development flows that are known and understood by everybody;|
|4.||Automate as much as possible from QA to deployments, a.s.o.;|
|5.||More flexible (loose) release processes lead to more complex flows.|
We’d love to hear about your plans for transformation.
Use the form below to send us a message, and we will get back to you within 24 hours.
New Broad Street House, 35 New Broad Street, London, EC2M 1NH
4D Gara Herastrau Street, 2nd Floor
Building C, 020334
+40 31 425 19 08
30 Stirbei Voda Blvd,
Malmo Business Center, 200423
+40 35 142 36 80