Episode 14

Jérémy Nancel & John Renne - Optimizations Ideal vs Reality

Published on: 25th October, 2022

Episode 14 Jérémy Nancel & John Renne - Optimizations Ideal vs Reality

Rightsizing an under-utilized resource is an easy way to save on cloud costs, right? Or is it? In a conversation taken from the FinOps Foundation Slack channels, Jérémy Nancel & John Renne debate the pros and cons of reducing cloud spend by rightsizing resources vs incurring tech debt via code drift.

Transcript
Jérémy:

Hi, I'm Jérémy Nancel

John:

Hi, I'm John Renne and this is FinOpsPod.

Stacy:

All right, so let's hold on.

Stacy:

Let's do our thing.

Joe:

Okay.

Joe:

Hi, I'm Joe Daly.

Stacy:

And I'm Stacey Case.

Stacy:

Wait, who says it?

Stacy:

don't.

Stacy:

I usually start

Joe:

this is FinOpsPod.

Stacy:

Wait I start and then you say your name and then you

Stacy:

say, This is FinOpsPod, right?

Joe:

It could be either way.

Joe:

I don't think we have a standardized way of doing it.

Stacy:

Hi, I'm Stacy Case.

Joe:

And I'm Joe Daly.

Stacy:

This is FinOpsPod.

Stacy:

Um, okay, so I'm not gonna lie, I didn't get a chance to listen

Stacy:

to this one yet because you, you were able to get this through a

Stacy:

little bit faster than I thought.

Joe:

but it was made available, was it not?

Stacy:

Yes, Dear listener, Joe made this available to me yesterday, but

Stacy:

I was not able to pre-listen to it, so I will have to listen to it when

Stacy:

it is released with everybody else.

Joe:

And we are in a little bit, You know what, In real life, real

Joe:

world - podcasts don't grow on trees.

Joe:

You gotta make 'em.

Stacy:

What?

Joe:

and there's, you know, just like anything else, we got schedules.

Joe:

We just did our October summit, which was really good.

Joe:

And then next week we're gonna be in Detroit for KubeCon North America.

Stacy:

Yeah I know that, I wanna make this podcast so people can listen

Stacy:

to it later and be like, Why stop talking about what's happening today?

Stacy:

But I am very excited to go KubeCon and see everybody and see the

Stacy:

folks and meet new people, but,

Joe:

it'll be fun.

Joe:

, Here's why it's special, Stacy, because we're gonna, we're gonna meet people and

Joe:

then they're gonna join the foundation, and then I'll be like, We have a podcast.

Joe:

And then they're gonna learn, listen to the podcast, and they're gonna pick

Joe:

this one first, because one, it'll be released, and they'll also be like, Whoa,

Joe:

this topic is really interesting to me.

Joe:

And then they'll be like, Oh my gosh, I so relate.

Joe:

Because they were talking about going to KubeCon and that is where I met them.

Stacy:

It's a full circle moment, folks.

Stacy:

We bring it full circle.

Joe:

And for all of you who aren't at KubeCon Forget you,

Stacy:

No, we still like you.

Stacy:

That is our director of community, everybody.

Stacy:

Director of community

Joe:

All right.

Stacy:

All right.

Stacy:

So talk, talk to me about this one.

Stacy:

Cause I know that the topic for this one came out of a really interesting place.

Joe:

Mm-hmm.

Joe:

Yeah.

Joe:

So.

Joe:

Jérémy Nancel was posting a question in Slack and it got my attention

Joe:

immediately because it was, it's just a real world question.

Joe:

he basically asked the question, Hey, if you're rightsizing resource,

Stacy:

Mm-hmm.

Joe:

At what point do you start worrying about code drift?

Joe:

And if you listen to the FinOoopsPod episode, you know, someone rightsized

Joe:

a resource under my direction and cause the rolling production outage.

Joe:

code drift is when basically the resources don't match the scripts

Joe:

that launch the solution anymore.

Joe:

and it's a great question because that starts to develop a problem in and of

Joe:

itself, which John Ren responded with.

Joe:

In the slack thread as tech debt, and it's true.

Joe:

the more that the solution doesn't recognize the source of truth of the code,

Joe:

the more tech debt you are incurring.

Joe:

But Jérémy makes a great point, is like, yeah, tech debt is not as

Joe:

immediately impactful to my bottom line as this oversized resource is.

Joe:

There is the ideal state, which is you go through and you fix

Joe:

the code and you fix the problem.

Joe:

but there's often a lot more that goes into it.

Joe:

Thus, our favorite answer of whenever anyone asks any FinOps question,

Stacy:

It depends.

Joe:

It depends.

Joe:

FinOps does not happen in a vacuum.

Joe:

That's a t-shirt.

Stacy:

There's a shirt for you,

Joe:

it happens in the real world with lots of different pressures.

Joe:

they had a really great slack thread

Stacy:

So is that why you pulled John into it as well?

Stacy:

John and Jérémy, they don't work together, do they?

Joe:

No, they had never actually talked to each other before.

Joe:

Other than the Slack thread.

Joe:

I just really enjoyed the conversation because this is the debate that if

Joe:

FinOps practitioners aren't having this debate, they need to consider it.

Joe:

because they're both right.

Joe:

Neither of them are wrong in the debate.

Joe:

And they both agree with each other and they both understand each

Joe:

other's point, but they both have to do what's best for their company.

Joe:

Jérémy is a cloud architect at EuroNext in France.

Joe:

And John is a DevOps engineer in the Netherlands working

Joe:

for a company called Bux.

Stacy:

Wait, let me get this right, Joe, John and Jérémy were in a

Stacy:

conversation together talking about this, having never spoken about it

Stacy:

before, and, that's what y'all captured.

Joe:

Yeah, and they really hit it off.

Joe:

And there was actually 10 minutes I removed because they

Joe:

started getting really technical about potential solutions.

Joe:

And I'll be honest, they went way over my head and at the end I

Joe:

think they saw me staring at them and they were like, Perhaps this

Joe:

is too technical for a podcast.

Joe:

I was like, Yeah, I'm gonna remove that part.

Stacy:

So basically reach out to them if you wanna get the technical bits.

Stacy:

You know what is awesome though about this though, is how

Stacy:

this all came about, right?

Stacy:

again, here's folks that are in the community.

Stacy:

It's not someone that you know, that we've had specifically talk anywhere before,

Stacy:

but it's literally just the, this is what you, as our community is going through.

Stacy:

This is the questions that you have, these are the responses that you provide, and

Stacy:

I think that's the whole of this podcast as an extension of the community, is

Stacy:

we're able to amplify those conversations so other folks can learn from 'em.

Stacy:

Right?

Stacy:

Like, you may have missed that slack thread, but hopefully

Stacy:

now listening to this, you're gonna be able to be like, Aha.

Stacy:

Yeah.

Stacy:

Okay.

Stacy:

I got it.

Stacy:

I can relate.

Stacy:

Or, or not, maybe this is not a topic for you, but I think that's just one of

Stacy:

the really awesome things about having this as an extension of the community

, Joe:

I mean, yeah, it's, it was, I mean, I feel very strong.

, Joe:

I joke that, I joke that I tease the community, but I do love them and I

, Joe:

feel strongly about it, and these are the types of conversations that I feel.

, Joe:

I feel don't get the visibility,

Stacy:

Mm

Joe:

from the industry because so much of it, I mean,

Joe:

prescriptive advice is amazing.

Joe:

It's great and it's useful, like I said, you got real world competing priorities.

Joe:

John represents the, here's the ideal way to do it, and Jérémy represents

Joe:

the competing priorities side and how do they work together and mesh.

Joe:

And it's a good conversation and I think we can just let them have it.

Stacy:

Awesome.

Stacy:

Well, take it away gentlemen.

Joe:

I really enjoyed the conversation from Friday, September 9th., Jérémy,

Joe:

you asked, you asked a question.

Joe:

Do you wanna say what question you posted into the slack?

Jérémy:

Yeah.

Jérémy:

So the question was about how to make, work together code drift and automatic

Jérémy:

FinOps actions because we are actually having that issue at the moment.

Jérémy:

So we have resources in Dev and so on that are not used anymore,

Jérémy:

but we want to delete them.

Jérémy:

But if we delete them using some kind of basic automated action

Jérémy:

then we end up with code drift and afterwards end up in technical debt.

Jérémy:

But the thing is sometimes when you do that deletion, it saves you so much

Jérémy:

money that maybe, maybe it's worse to have a little technical depth over that.

Jérémy:

And afterward, John.

Jérémy:

John answered a very, very, interesting response.

Joe:

Yeah.

Joe:

John, I see your responses like, it's so important because it's the ideal.

Joe:

Can you explain your perspective of how, what the answer is?

John:

Yeah.

John:

, I see what technical debt, I see how it, how it occurs.

John:

I see how it exists and, I've been in both sides of the story.

John:

I've a product owner doing a team that did some infrastructure engineering,

John:

and I've been an engineer and what I would consider the most important thing

John:

is, there is a single point of truth.

John:

If for any reason, you have a drift between your infrastructure

John:

and your infrastructure as code.

John:

You might as well not do it.

John:

Cause basically, , people rely on the infrastructure code to do their work and

John:

if there's drift people do stuff that either doesn't work cause it isn't as

John:

expected or even worsens the situation.

John:

It's a big anti-pattern.

John:

Both as an engineer and as a product owner in which it's always been, well,

John:

the goal for me to limit the amount of technical depth and even to, to make sure

John:

it doesn't automatically get created.

John:

Cause that's basically what happens, I think, if you do automatic actions.

Joe:

Right.

Joe:

And John, I have personal experience.

Joe:

I've shared on previous episodes of the podcast where there

Joe:

was infrastructures code.

Joe:

We used a third party automation tool that changed what the

Joe:

infrastructure looked like and it had catastrophic implications

Joe:

because of how the code worked.

Joe:

I always kind of hesitate, I, I, at least I advise like, Hey, before

Joe:

you use automation, check first, don't be so fast with automation.

John:

Oh, I'm, I'm more in favor of automation.

John:

You have to have automation, but make automation do the right stuff if your

John:

infrastructure is completely defined by infrastructure as code in ideal situation

John:

have your automation edit the code.

John:

So basically you edit the source of truth technical debt doesn't get created.

John:

You just have influence on

Jérémy:

I want to say the worst thing is that I definitely agree with you.

Jérémy:

I hate technical debt on going over and over again on the same stuff because it

Jérémy:

gets updated with an out of process way, and I could not agree more on that,

Jérémy:

but sometimes, sometimes it's good.

Jérémy:

It's, it's ideal.

Jérémy:

It's, let's say, I don't know how to say, but when you arrive on an

Jérémy:

infrastructure that has a proper state, you know, proper processes

Jérémy:

and and so on, I would say fixing technical debt is definitely a must.

Jérémy:

But at some points I'm sure there will be a lot of listeners and

Jérémy:

people as well that do work, that do arrive in a place where there's

Jérémy:

already a kind of big technical depth.

Jérémy:

And actually we are talking about FinOps here.

Jérémy:

So if you make some saves by adding some technical depth, it can help

Jérémy:

the managers free some time for you and your colleagues to afterwards

Jérémy:

fix the remaining, technical debt.

Jérémy:

Because at some point, if you spend that much in cloud resources, Then well,

Jérémy:

your margin for doing stuff outside of new features and fixes is limited.

Jérémy:

But , if you show your manager, , that was actually my situation.

Jérémy:

I wanted to show that I was, , giving them something, giving them some,

Jérémy:

, savings on resources that were unused or oversized or things like that.

Jérémy:

And by doing that, I was hoping that it would free the team some time to

Jérémy:

get into a better state, you know, , small step by small step, and so on.

John:

Yeah, but the question is, do you free the team?

John:

Cause basically what you do is you create technical debt.

John:

Which is an extra burden for the team, which, it makes it more difficult

John:

to do troubleshooting cause people actually have to watch what the

John:

truth is instead of relying on the truth that is in their control.

John:

So, yeah, I can imagine on one side you want to do savings, . We should

John:

do that, but, by doing so , on the same time, creating technical debt

John:

and creating more work for the engineers to solve the technical debt.

John:

you really save stuff?

John:

Do do you have savings or do you pay less for infrastructure

John:

and more for human resources?

John:

Cause basically the engineer is often the most expensive part of the infrastructure.

John:

That's not the two or three or tens.

Joe:

I find this so, so, so interesting because it's real life, right?

Joe:

There is a Yeah, of course.

Joe:

Fix the code.

Joe:

But Jérémy, I empathize with you so much because , I understand like,

Joe:

hey, there's multiple pressures, influencing that decision.

Joe:

I think one of the problems I made was, Oh, let's right size and then walk away.

Joe:

Whereas at least you're saying, let's right size or shut it down or delete

Joe:

it and then focus on fixing it.

Joe:

And then John, your point is, No, just fix it because you're

Joe:

creating additional burden outside of your cloud bill, , on that.

Joe:

But there, there's so many different pressures that go

Joe:

into those sort of decisions.

Joe:

It's a debate that I think every FinOps practitioner in every company

Joe:

IT department needs to take when focusing on the FinOps value of their.

Joe:

Perspective.

Joe:

Jérémy, what was the team's reaction, the application team's reaction to this?

Jérémy:

Well, the thing is, the application team, how can I say?

Jérémy:

When we have to do things like that, we first have to we have someone

Jérémy:

system that says, Okay, that resources underutilized or not utilized at all.

Jérémy:

So, one part of the thing that is problematic is sometimes tagging is not

Jérémy:

as good as expected and we have to look into it and find across the organization

Jérémy:

the one guy if he's still there that is responsible for, for the application.

Jérémy:

So, from what I could see when we find some people and say, Okay, you

Jérémy:

have that result that is not good.

Jérémy:

it's okay for them.

Jérémy:

I mean, it's normal.

Jérémy:

It's just that they would find normal to avoid that cost.

Jérémy:

But, we needed to set up the processes, in place in order to fix those.

Jérémy:

So this part is quite easy , when you manage to find the owner, the owner says

Jérémy:

yes, okay, it's maybe time consuming, but then you stay in line with Infr

Jérémy:

code because you do the update yourself, and then you remove the stuff and it's

Jérémy:

automatically directed through your deployment system and everything is fine.

Jérémy:

The complicated stuff is mostly when you don't find any owner.

Jérémy:

And I work in the financial world where regulation is quite heavy and it's very

Jérémy:

sensitive to delete anything, basically in any environment because we have some

Jérémy:

regulation that says that some stuff have to been kept for 10 years and so on.

Jérémy:

So , it's a little bit complicated.

Jérémy:

We have to snapshot stuff and maybe to destroy or to just dispose.

Jérémy:

We have different decision.

Jérémy:

There's no one pattern to solve everything.

Jérémy:

So at some points we need to make some choices and sometimes we say,

Jérémy:

Okay, we're going to take more time and, and solve it through us through

Jérémy:

proper processes, and sometimes we can save so much money by one action.

Jérémy:

Then you say, Okay, doing that now and fixing later.

Jérémy:

Okay.

Jérémy:

It introduces some technical debt, but a month or two, we will not

Jérémy:

have to pay that we would otherwise.

Jérémy:

But I understand John's view and I agree that in general,

Jérémy:

time is is actually hidden.

Jérémy:

Very expensive cost.

Jérémy:

So as I said, noisy choice.

John:

But, I see two things that trigger me.

John:

first one is you modify resources you don't have the owner from yet mean

John:

that, that sounds a bit like, Well, the whole FinOps practice , Was introduced

John:

as a crawl, walk and run stage.

John:

And one of the first stages you always go to is the inform.

John:

So make sure you have the right information and looks

John:

like you are trying to run.

John:

So you're trying to do the automation, which is basically about the last stage

John:

you can get in your maturity while the Inform phase isn't complete yet.

John:

Cause you dunno who the owner is, but you try to take automated

John:

responses based on that resource that, that, that would seriously

John:

scare hell out me and the second is,

John:

When I started my FinOps journey I thought it was the next thing

John:

after DevOps, , You could integrate developers and operations people.

John:

You could use the same tool, use the same speak, just started I

John:

hoped it would be the same thing.

John:

So engineering actually talking to finance, doing the more, and I

John:

should more and more FinOps teams, just as more the central team

John:

without debt engineering engage.

John:

How do you manage that?

John:

Cause basically, uh, yeah.

John:

How do you distinguish the FinOps team from a team like security, quality, etc.

John:

All those team that try to influence the backlog for the infrastructure engineers?

Joe:

Yeah, John, Both points are really good cuz the first point is,

Joe:

You know, like that informed phase if you don't have really good tagging,

Joe:

that is a form of technical debt.

Joe:

and I'm trying not to laugh out loud.

Joe:

As you're describing, taking automation, which is a very

Joe:

mature run state sort of thing.

Joe:

And then, but using, really bad tagging, to drive it, that is kind of dangerous.

Jérémy:

Yeah, indeed, indeed.

Jérémy:

This is dangerous.

Jérémy:

And indeed, we missed out, what would have been the proper processes.

Jérémy:

But the thing is, getting the informed phase can be challenging

Jérémy:

with huge technical debt.

Jérémy:

So waiting for the end of the informed phase before moving to the other ones,

Jérémy:

it's the proper process indeed, but, , management will not see it that way.,

Jérémy:

if all the time you spent if the inform phase is used to reduce your,

Jérémy:

your cloud spent at the end of the month, at the end of the year, they

Jérémy:

are still not seeing improvements, but only by you adding tagging.

Jérémy:

So we also have we also have some initiatives to, fix tagging,

Jérémy:

all over our infrastructure.

Jérémy:

But in the meantime unfortunately, we have some budgets, that we

Jérémy:

were on the point of exceeding.

Jérémy:

And at some point when you have that kind of situation, you cannot help

Jérémy:

but look into your overall costs and see, okay, is there anything I can do

Jérémy:

that can be qualified as a quick win?

Joe:

This is why I absolutely love this conversation because John, you're,

Joe:

you're right, but Jérémy, you're also right, is just like, I don't

Joe:

always have the time because , you know, the boss, the management, the

Joe:

business is like, You know, it's, it's kind of an impossible situation,

Joe:

but they're like, deliver right now.

Joe:

I wanna go back to what you said, John, and you posted it in the Slack thread

Joe:

and I liked how you said it, that, you were saying in two points earlier.

Joe:

The second point was that, you know, you were excited about FinOps cuz

Joe:

you thought it was going to bring the financial impact and engineering

Joe:

engagement together closer.

Joe:

And that's, you often are seeing teams that add just another layer in

Joe:

between finance and engineering and that exists for a reason for a lot

Joe:

of companies and they do a good job translating between the two.

Joe:

However, I think it's really interesting that, if you have a centralized FinOps

Joe:

team, you really do need engineering skillset to jump into these conversations.

Joe:

and I think that's what you're trying to say, John, is

Joe:

there needs to be engineering solutions to problems like this.

John:

Yeah, that's least personal belief is team should have a senior

John:

engineer to just do an impact anlaysis.

John:

to just think in code.

John:

Just like team should have a well educated finance specialist

John:

that really knows the processes.

John:

Because, I remember I studied for the practitioner exam and I

John:

came into terms like amortization.

John:

I was like, Okay, this has been since high school economics.

John:

Think this, please.

John:

I don't anymore.

John:

It's been 20 years or something.

John:

So, no, but every team needs the different disciplines, and I'm not

John:

opposed to a centralized FinOps team.

John:

FinOps teams, require some sort of, helicopter view over the teams,

John:

but it should really have an engineer in them just like it should really

John:

have a finance professional in them.

John:

And preferably a software engineer too.

John:

Not just the engineer but the software engineer.

John:

Now they need a software engineer.

John:

Cause basically the software engineer can also add the

John:

impacts cause what will they do?

John:

Will have serverless function?

John:

Software architect easily make good judgment on whether to use serverless

John:

frameworks to use, for example, he can get the impact on the application.

John:

And FinOps doesn't end with infrastructure.

John:

It's one of my long term thoughts is the, the separation between software

John:

and infrastructure will be gone.

John:

it'll end.

John:

Basically, infrastructure is always about four, five years

John:

behind those software development.

Joe:

I love it.

Joe:

It's so true because , you're talking about infrastructure

Joe:

as code and deployment as code, pipelines , and all that.

Joe:

But there are still plenty of applications that are launching through the console.

Joe:

, but they won't always be and they need that modernization path forward.

Joe:

And software engineers probably really good path of insights there

Joe:

I love the conversation as much as I love the slack thread, this is real

Joe:

life FinOps debates and actions.

Joe:

I think it's so valuable to give attention to.

Joe:

So I wanna thank you both for being here and having this conversation further.

Joe:

me.

John:

Yeah.

John:

But it's really nice to have to talk with peers and to know, to get more

John:

insight in, in, in what the other people.

Jérémy:

Indeed, indeed.

Jérémy:

Even, even the slack is inspiring about, everyone's different use cases and

Jérémy:

being able to talk is actually great at another level, but it's is actually

Jérémy:

great to be able to exchange with everyone, with you guys, but with anyone.

Jérémy:

Best practices, how to, how to do what, what, who we should

Jérémy:

include in teams and so on.

Jérémy:

It's is very inspiring especially for me who's, let's say

Jérémy:

young in, in the FinOps world.

Jérémy:

So really, really, I'm really glad, we had that, that conversation together.

Joe:

Hmm, Hmm, hmm.

Joe:

Things that make you go.

Joe:

Hmm.

Joe:

Now, look folks like we said, podcasts don't grow on trees

Joe:

and neither do music rights.

Joe:

And if I had the money to get the music rights to C&C Music Factory's "Things That

Joe:

Make You Go Hmm", I'd be playing it right now because that conversation makes me go

Joe:

'Hmm, what would I do in that scenario?'

Joe:

How would I handle that?

Joe:

And honestly, rather than me buy the music rights and play it under

Joe:

this right now, I recommend you go and look it up on YouTube.

Joe:

It's a good music video, 1990.

Joe:

The glory days of music videos.

Joe:

But back to the conversation at hand, this is a real debate, the type of

Joe:

thing that when I was a practitioner, you had to consider what was the

Joe:

best way to go about it, you know?

Joe:

Unfortunately, not everything is in code, and the only

Joe:

thing you have is the console.

Joe:

Launching things to the console.

Joe:

you would hope you get more mature and start launching the solution through

Joe:

code, and then you start launching the solution through pipelines.

Joe:

And this conversation gets a lot easier as your application stack matures.

Joe:

but not everyone is at that same maturity.

Joe:

So these are the considerations, and debates that you have to have.

Joe:

Hey, we can right size this now and stop the expense, but we are creating

Joe:

more expense later if we leave this tech debt and code drift behind in our wake.

Joe:

Great conversation, great insights from Jeremy and John.

Joe:

If you have any questions, reach out to Jeremy and John

Joe:

on our FinOps Foundation Slack.

Joe:

And if you don't have access to our FinOps Foundation Slack, then head over to FinOps

Joe:

dot org and click join the community.

Joe:

We'd love to get you involved

Joe:

. Big thanks.

Joe:

Go out to Jérémy Nancel and John Renn.

Joe:

Thanks for being willing to have the conversation, and letting me record

Joe:

it, and share this with the community.

Joe:

Thank you to Stacy Case.

Joe:

Always bringing the energy.

Joe:

Can't wait to do KubeCon with you and Suha and J.R..

Joe:

Stop by the booth and say hello to us if you hear this and

Joe:

you happen to be in Detroit.

Joe:

If not, hey, there'll be other conferences.

Joe:

You could say hello, then we're friendly.

Joe:

Come say hi.

Joe:

And that is all for this episode of FinOpsPod.

Next Episode All Episodes Previous Episode

Listen for free

Show artwork for FinOpsPod

About the Podcast

FinOpsPod
Advancing FinOps Practitioners in Podcast Form
FinOpsPod connects practitioners with the rest of the FinOps Foundation community. Real world practitioners will share their experiences and the Foundation team will share the latest news about the community. Learn more about the FinOps Foundation at www.finops.org.