By the time we reached Redis eviction policies, I knew we were no longer arguing about Redis eviction policies.
I am refactoring a signaling service used for calling. When a device does not acknowledge a signal, we keep that signal for 60 seconds so the device has another chance to retrieve it. Today, those signals are stored in Cassandra. I wanted to move them to Redis.
To me, Redis felt like the natural home for this kind of data. The records are temporary. They need to be written and retrieved quickly, and our real traffic is large but still within a range that I believe our managed Redis cluster can handle comfortably.
I had also designed the Redis model around the problems I wanted to avoid. Related keys would share the same cluster slot through hash tags. A Lua script could write the signal and its indexes atomically, so another request would not find half of an update. We could retrieve a signal by the identifiers we needed without maintaining several independent copies of the same record.
Cassandra can handle enormous write volumes, but our access patterns make its strengths less simple than they first appear. Fast queries by device, user, and signal may require query-specific copies of the data. Stronger consistency asks more replicas to participate. The records also expire quickly, and in Cassandra, expired TTL data becomes tombstones that remain until compaction removes them.
On paper, I could explain why I believed Redis was the cleaner and faster design.
My manager was not looking at a blank piece of paper.
The team had tried Redis for this path before. It caused problems, and the system was moved to Cassandra. I did not have evidence that Redis itself had caused the earlier failure. The problem might have been in the architecture, configuration, or way it was operated. My design was also different from the previous one.
But the old failure was real to him. Whatever its precise cause, he had already seen this road lead somewhere painful.
That history entered every part of our argument.
He raised consistency, and I explained how the keys and Lua script would keep related changes together. He said Cassandra was also fast, especially while short-lived data remained in memory. He worried about Redis eviction, and I answered that we could choose an appropriate memory policy. He worried about the write volume, and I returned to our actual scale and the capacity of the cluster.
Every time I thought I had answered the concern, another one appeared.
I felt that the decision had already been made and the technical objections were being used to defend it. The conversation seemed to move without going anywhere. I was not only frustrated that my design might be rejected. I was frustrated because I believed a better approach was being blocked by a failure that nobody had fully explained.
It seemed a pity that an old experience could make a new design carry so much extra weight.
But after the argument, I kept thinking about the difference between what each of us had brought into the room.
I had a model. I had reasoned through the data lifetime, access patterns, consistency problems, and expected load. Other teams were using similar approaches, which gave me more confidence. My manager had a memory of the system failing in production, where calling customers experienced the consequences.
I was trying to show why the new design should work. He was trying to avoid discovering, for a second time, why it might not.
Understanding that did not make me agree with his conclusion. I still believed Redis was the right choice. But it made the disagreement more human than “evidence against fear.” A past incident is not proof that every future design will fail, but it changes how much proof people require before they are willing to try again.
My manager asked me to compare 100 writes to Cassandra and Redis and show him the latency. I had not run the experiment yet. The argument had happened that same day.
Still, I felt sure I knew which one would win.
That confidence was part of the problem too. I was asking him not to let an earlier experience decide the answer, while I had already decided the answer before running the test he requested.
The test will still not settle everything. A small benchmark can compare latency, but it cannot recreate production. Load tests and lower environments are useful, but they rarely contain all the strange timing, failure, traffic, and customer behavior that appear in a real system. This is especially important when the feature is calling. If this path fails, the impact is not theoretical.
That left us with an uncomfortable question: if only production evidence can overcome production fear, how do we obtain that evidence without taking an irresponsible risk?
A phased rollout first sounded like the obvious middle ground. We could expose a small percentage of traffic, monitor it, increase it during quieter periods, and roll back immediately if the system behaved badly.
Then I started thinking about the transition itself.
A device could store its unacknowledged signals in Cassandra and later move into the Redis group before those signals expired. It might then look for them in Redis and find nothing. We could read from both databases and merge the results, but that would add latency, complexity, and another place for failures to hide.
The supposedly safe path had introduced risks of its own. For a moment, phased rollout felt almost as dangerous as replacing the system at once.
This was another useful correction to my thinking. “Release it gradually” is easy advice when the data has no memory of where it lived. In this case, even though the records survive for only 60 seconds, that short transition still matters. The rollout cannot be separated from the architecture.
We eventually agreed to move forward with a phased rollout. It is slower and more complicated than I wanted. It creates work that would not exist if everyone already trusted the Redis design. But it also gives both sides something the argument could not provide: a way for reality to participate in the decision.
The goal is no longer to make my manager borrow my confidence. It is to create enough evidence that neither of us has to depend only on confidence or fear.
Arguments like this happen all the time. Developers become attached to the clarity of a new design. Managers and long-standing team members remember the cost of systems that looked clear before they reached production. Both perspectives contain something valuable, and both can become dangerous when they stop listening.
Reaching the middle ground does not always mean choosing half of each proposal. Sometimes it means understanding why the other person is resisting, deciding what evidence could change either position, and accepting a slower path that leaves room for both outcomes.
That path is worthwhile only when the possible reward is large. Introducing production risk for a small improvement is not courage. If Redis does not produce a meaningful reduction in latency, or if the added writes put unhealthy pressure on the cluster, then the benefit does not justify the change.
I still believe Redis is the right design. The phased rollout may eventually support that belief, or it may expose something I have missed.
Either result would be more useful than winning the argument.