The Little Strangler


The Little Strangler / Le petit étrangleur

or: 6 totally CRAZY hints to have SATISFYING rollouts, the last one will Blow Your Mind!

The story so far: In the beginning, a program was created. This has made a lot of people very angry and been widely regarded as a bad move.

But time ravages all. Plus I read a blog post about the Strangler Pattern. Violent, yes, but useful.

The following is the story of how I went first from this all side-effects, no return values:

something_old()

To:

something_new()

With only a few lines of code in-between…

val (primary, secondary) = when (getEnum<Rollout>(FLAG, key)) {

    OLD_ONLY -> something_old to null

    BOTH_WITH_OLD_PRIORITY -> something_old to something_new

    BOTH_WITH_NEW_PRIORITY -> something_new to something_old

    NEW_ONLY -> something_new to null

}

secondary?.let(threadPoolExecutor::submit)

primary()

Please let me tell you a tale of the mistakes I made— err, lessons I learned along the way.

Run the new and the old

The old must make way for the new. And because I’m young at heart, I hastily wrote these two lines of code:

if (turn_on_new_code) something_new()

else something_old()

We young are rash in our confidence. We are invigorated by incidents. And apparently, my PR “needs work”

 if (turn_on_new_code) something_new()

-else something_old()

+something_old()

Lesson: run the new and the old

New Code Can Crash

… this, of course, all presumed the new code worked. It didn’t. It doesn’t. And when it crashed, it interrupted the call to the old code. This is, of course, was an easy fix:

-if (turn_on_new_code) something_new()

+try {

+    if (turn_on_new_code) something_new()

+} catch(e: Exception) {

+    e.printStackTrace()

+}

 something_old()

Clearly perfect. It is wabi-sabi. I am a genius. This goes in my hype doc.

Lesson: new code can crash

New Code Takes Time

Knock knock?

— Who’s there

TimeoutException

— TimeoutExcep— the new code ran for too long and the process ran out of time.

Back off everyone, I know threads:

-try {

-    if (turn_on_new_code) something_new()

-} catch(e: Exception) {

-    e.printStackTrace()

+if (turn_on_new_code) {

+    Thread(something_new).start()

 }

 something_old()

Goodbye ugly try / catch block! If the thread throws an exception, it’ll get logged by… something else. I don’t know and I feel better already.

Lesson: new code takes time

New Code Takes Space

Why is it when Java “failed to create a thread” it throws an OutOfMemoryError? Shouldn’t it be an OutOfThreadsError?

 if (turn_on_new_code) {

-    Thread(something_new).start()

+    threadPoolExecutor.submit(something_new)

 }

 something_old()

The parameters for the thread pool are outside the scope of thread of this story.

Lesson: new code take space

Do Staged Rollouts

Did you know we can and regularly do denial of service attack ourselves?

It was at this point, intellects vast, cool and unsympathetic regarded this code with envious eyes and slowly and surely drew their plans against it:

-if (turn_on_new_code) {

+if (getFlag(TURN_ON_NEW_CODE, key)) {

     threadPoolExecutor.submit(something_new)

 }

 something_old()

The key is the critical magic that helps getFlag decide whether or not the new code runs.

Lesson: do staged rollouts

Ship It 🚢

By jove, that if-statement was quite cumbersome; however, it worked! It worked!

And after months weeks days hours of testing the new code, we’re I was ready to turn off the old code and… oh I needed to change the code again. 🤦🏾‍♂️

Fortunately, I learned my lessons from before!

 if (getFlag(TURN_ON_NEW_CODE, key)) {

     threadPoolExecutor.submit(something_new)

 }

-something_old()

+if (!getFlag(TURN_OFF_NEW_CODE, key)) {

+    something_old()

+}

HA! Wait. No, I want problems with something new in the critical … no… err—

 if (getFlag(TURN_ON_NEW_CODE, key)) {

-    threadPoolExecutor.submit(something_new)

+    something_new()

 }

 if (!getFlag(TURN_OFF_NEW_CODE, key)) {

     something_old()

 }

No. I can’t do that either… the timeout. Umm. Umm.

 if (getFlag(TURN_ON_NEW_CODE, key)) {

-    something_new()

+    if (!getFlag(TURN_OFF_NEW_CODE, key)) {

+        something_new()

+    } else {

+        threadPoolExecutor.submit(something_new)

+    }

 }

 if (!getFlag(TURN_OFF_NEW_CODE, key)) {

     something_old()

 }

Lesson: Ship it. TGIF. I need a drink.

Next Monday

SOMETHING IS ALL WRONG. I EXPECT TO SEE SOMETHING IN THE LOGS

— “why are you yelling?”

WHO TURNED OFF BOTH MY FEATURE FLAGS

— “Oncall”

WHY

— “Because there were infinite incidents… it’s being called the incidentularity.”

— “We rolled back.”

— “Have you checked your Slack?”


ccccccknghvnevnujnlgieeuiblueekrdfikhfdigekb

Setting TURN_ON_NEW_CODE to false and TURN_OFF_NEW_CODE to true is not something I expected anyone to do. But oncall isn’t just anyone; it’s where dreams die and every panicked reality in the multiverse fractures into ours. It’s where I close off my feelings so I don't get crippled by the moral ambiguity of my actions.

I tried again. Harder, that time. I used computer science.

val turn_on_new_code = getFlag(TURN_ON_NEW_CODE, key)

val turn_off_old_code = getFlag(TURN_OFF_NEW_CODE, key)

val rollout = when {

    turn_on_new_code && turn_off_old_code ->

        NEW_ONLY

    turn_on_new_code && !turn_off_old_code ->

        BOTH_WITH_OLD_PRIORITY

    !turn_on_new_code && turn_off_old_code ->

        throw IllegalStateException("y tu, oncall?")

    !turn_on_new_code && !turn_off_old_code ->

        OLD_ONLY

    else ->

        throw java.lang.IllegalStateException("how?!")

}

if (rollout == BOTH_WITH_OLD_PRIORITY) {

    threadPoolExecutor.submit(something_new)

}

else {

    if (rollout == NEW_ONLY) something_new()

}

if (rollout == OLD_ONLY || rollout == BOTH_WITH_OLD_PRIORITY) {

    something_old()

}

I’m joking, of course. No one could write that code and still be scheduled for technical pairing interviews. Two booleans do not make a truth table. And why does Kotlin demand an else-condition? But that enum auto-completed its definition into:

enum class Rollout {

    OLD_ONLY,

    BOTH_WITH_OLD_PRIORITY,

    NEW_ONLY;

}

And it was at that moment when my seven weeks under the Bodhi Fig Tree paid off and I saw that there was a cause for my suffering:

 enum class Rollout {

     OLD_ONLY,

     BOTH_WITH_OLD_PRIORITY,

+    BOTH_WITH_NEW_PRIORITY,

     NEW_ONLY;

 }

Is this the start of something real and true and good? Maybe. But, it’s the end of our journey:

val (primary, secondary) = when (getEnum<Rollout>(FLAG, key)) {

    OLD_ONLY -> something_old to null

    BOTH_WITH_OLD_PRIORITY -> something_old to something_new

    BOTH_WITH_NEW_PRIORITY -> something_new to something_old

    NEW_ONLY -> something_new to null

}

secondary?.let(threadPoolExecutor::submit)

primary()

Martin Fowler never said anything about thread pools and enums.