Back to top
3 minute read

Flo in Wonderland

The last shiny watch was gitlab-ci. So I prepped my gitlab-ci docker image, I prepped haproxy docker images, configured it, tested it, it was all good. So I went to prod with it and went down the rabbit hole.

All doors are locked

It doesn’t work.It is too late now. All you’ve got is sad locked doors. Well, not for long as I use my golden key on the security group and allow myself in.It still doesn’t work.

Drink me

I’m now concentrating. I’m now restarting my gitlab-ci instance, it kind of looks good. I’m jostling with netcat on the host, and the logs on the container.I can see something is happening in the container.


Eat me

I’m now deconcentrating. I test DNS with dig, it’s all good (well that didn’t change since going from testing to prod).


Chapter two

Pools of tears

I have a mouse, I click it.
I hit refresh, firefox doesn’t care.
I speak to it in French.
The mouse gets annoyed, it goes away.
I’m back on my keyboard

.

Chapter three

Dry lecture

I’m now deep down into all sorts of haproxy parameter and configuration.I add some, remove some, I’m pretty sure it doesn’t help.


Cactus race

I stop and start the container.
I loop for a bit.
That doesn’t help.


Chapter four

Looking for the gloves and fan

Once again, I’m trying to think what I missed in the big picture.
DNS-> IP: check
PORT 80/443: when I listen with NC, I can clearly see that the traffic goes all the way to the front end box.


Voices

I can hear voices, it is the monitoring system.
It tells me that people will start to throw stones soon if it is not back up soon.
Will there still be cake if I fix it now?


Chapter five

Crisis time

I can see smoke in slack, maybe I’m in luck,I ask the caterpillar.
The caterpillar is not replying.
I’m not in luck.


Magic mushroom

I’m now going from the big picture to little details from bite to bite.
I use {firefox,curl,netcat,telnet,openssl s_client} from {my machine, haproxy’s host,gitlab’s host, inside gitlab’s container} to {haproxy’s host,haproxy’s conainer,gitlab’s host,gitlab’s container}.
There is no tool from within the haproxy’s container to test itself.
My head grows into a tree.
Too bad that doesn’t stop the flock of pigeon and seagulls to shut it.
Seems like they are all outside my window laughing.
I’m starting to have enough


Chapter six

The soup

I have now done everything a gazillion times like if I was caught into a sneezing frenzy.
It clearly looks like what doesn’t work is https, haproxy wants none of it.
The fact that http works, that doesn’t really help me because it redirects to https.
As soon as get the baby, it transform into a pig… happy days.


Cheshire Cat

I’m starting to think that I looked at the problem for so long that there is only me and my laptop floating in space.
Then I realise that I often seen a cat without a grin but never a grin without a cat.
If there is no connection possible, maybe it is because there is no port open.
If there is no solution, there is no problem


Chapter seven

Tea time

It’s now tea time, both the mad hatter and the march hare are at the table.I try to explain what is going on, but it is not worth it, I escape.


Chapter eight

Croquet

I’m being fed up of not being able to do anything from within the haproxy container.I decide that even if I don’t have a mallet or a ball, I will play.


Off with their heads

I use “echo” and “/dev/tcp/localhost/{80,443}” to get the ball rolling.
They don’t have the same outcome, namely I’m refused to echo to 443.
Now I’m clearly getting there, I know that even from within the container something is wrong.


Chapter eleven

ENOTARTS

It’s trial time for haproxy.cfg
Then shock and horror: the tart never existed as haproxy was the listenting to 433 all along.


Chapter twelve

Nothing is sacred

I can be heard shouting at myself “All persons not capable of typing 443 must copy and paste”.Next time, I’ll stop for tea to think.I should have known that what really matters is not the answer, but what questions to ask.


Fin

I’m somehow relieved that those “potentially impossible” {configuration,network} issues were simply my brain trying to make sense of the situation.
I can now put fat-finger on linkedin as I clearly demonstrated my qualifications here.

Ric Harvey

Ric leads engineering and technical architecture for Ngineered. He has a vast amount of experience in cloud computing, having been responsible for the delivery of large-scale cloud migration projects at companies like Ticketmaster and Channel 4.

Discussion