[linux-lvm] Reserve space for specific thin logical volumes

[Xen]

Zdenek Kabelac schreef op 14-09-2017 21:05:

> Basically user-land tool takes a runtime snapshot of kernel metadata
> (so gets you information from some frozen point in time) then it
> processes the input data (up to 16GiB!) and outputs some number - like
> what is the
> real unique blocks allocated in thinLV.

That is immensely expensive indeed.

> Typically snapshot may share
> some blocks - or could have already be provisioning all blocks  in
> case shared blocks were already modified.

I understand and it's good technology.

>> Yes I mean my own 'system' I generally of course know how much data is 
>> on it and there is no automatic data generation.
> 
> However lvm2 is not 'Xen oriented' tool only.
> We need to provide universal tool - everyone can adapt to their needs.

I said that to indicate that prediction problems are not current 
important for me as much but they definitely would be important in other 
scenarios or for other people.

You twist my words around to imply that I am trying to make myself 
special, while I was making myself unspecial: I was just being modest 
there.

> Since your needs are different from others needs.

Yes and we were talking about the problems of prediction, thank you.

>> But if I do create snapshots (which I do every day) when the root and 
>> boot snapshots fill up (they are on regular lvm) they get dropped 
>> which is nice,
> 
> old snapshot are different technology for different purpose.

Again, what I was saying was to support the notion that having snapshots 
that may grow a lot can be a problem.

I am not sure the purpose of non-thin vs. thin snapshots is all that 
different though.

They are both copy-on-write in a certain sense.

I think it is the same tool with different characteristics.

> With 'plain'  lvs output is - it's just an orientational number.
> Basically highest referenced chunk for a thin given volume.
> This is great approximation of size for a single thinLV.
> But somewhat 'misleading' for thin devices being created as 
> snapshots...
> (having shared blocks)

I understand. The above number for "snapshots" were just the missing 
numbers from this summing up the volumes.

So I had no way to know snapshot usage.

I just calculated all used extents per volume.

The missing extents I put in snapshots.

So I think it is a very good approximation.

> So you have no precise idea how many blocks are shared or uniquely
> owned by a device.

Okay. But all the numbers were attributed to the correct volume 
probably.

I did not count the usage of the snapshot volumes.

Whether they are shared or unique is irrelevant from the point of view 
of wanting to know the total consumption of the "base" volume.

In the above 6 extents were not accounted for (24 MB) so I just assumed 
that would be sitting in snapshots ;-).

> Removal of snapshot might mean you release  NOTHING from your
> thin-pool if all snapshot blocks where shared with some other thin
> volumes....

Yes, but that was not indicated in above figure either. It was just 24 
MB that would be freed ;-).

Snapshots can only become a culprit if you start overwriting a lot of 
data, I guess.

>> If you say that any additional allocation checks would be infeasible 
>> because it would take too much time per request (which still seems odd 
>> because the checks wouldn't be that computation intensive and even for 
>> 100 gigabyte you'd only have 25.000 checks at default extent size) -- 
>> of course you asynchronously collect the data.
> 
> Processing of mapping of upto 16GiB of metadata will not happen in
> miliseconds.... and consumes memory and CPU...

I get that. If that is the case.

That's just the sort of thing that in the past I have been keeping track 
of continuously (in unrelated stuff) such that every mutation also 
updated the metadata without having to recalculate it...

I am meaning to say that if indeed this is the case and indeed it is 
this expensive, then clearly what I want is not possible with that 
scheme.

I mean to say that I cannot argue about this design. You are the 
experts.

I would have to go in learning first to be able to say anything about it 
;-).

So I can only defer to your expertise. Of course.

But the purpose of what you're saying is that the number of uniquely 
owned blocks by any snapshot is not known at any one point in time.

And needs to be derived from the entire map. Okay.

Thus reducing allocation would hardly be possible, you say.

Because the information is not known anyway.


Well pardon me for digging this deeply. It just seemed so alien that 
this thing wouldn't be possible.

I mean it seems so alien that you cannot keep track of those numbers 
runtime without having to calculate them using aggregate measures.

It seems information you want the system to have at all times.

I am just still incredulous that this isn't being done...

But I am not well versed in kernel concurrency measures so I am hardly 
qualified to comment on any of that.

In any case, thank you for your time in explaining. Of course this is 
what you said in the beginning as well, I am just still flabbergasted 
that there is no accounting being done...

Regards.


>> I think they are some of the most pleasant command line tools 
>> anyway...
> 
> We try really hard....

You're welcome.

>> On the other hand if all you can do is intervene in userland, then all 
>> LVM team can do is provide basic skeleton for execution of some 
>> standard scripts.
> 
> Yes - we give all the power to suit thin-p for individual needs to the 
> user.

Which is of course pleasant.

>>> So all you need to do is to use the tool in user-space for this task.
>> 
>> So maybe we can have an assortment of some 5 interventionalist 
>> policies like:
>> 
>> a) Govern max snapshot size and drop snapshots when they exceed this
>> b) Freeze non-critical volumes when thin space drops below aggegrate 
>> values appropriate for the critical volumes
>> c) Drop snapshots when thin space <5% starting with the biggest one
>> d) Also freeze relevant snapshots in case (b)
>> e) Drop snapshots when exceeding max configured size in case of 
>> threshold reach.
> 
> But you are aware you can run such task even with cronjob.

Sure the point is not that it can't be done, but that it seems an unfair 
burden on the system maintainer to do this in isolation of all other 
system maintainers who might be doing the exact same thing.

There is some power in numbers and it is just rather facilitating if a 
common scenario is somewhat provided by a central party.

I understand that every professional outlet dealing in terabytes upon 
terabytes of data will have the manpower to do all of this and do it 
well.

But for everyone else, it is a landscape you cannot navigate because you 
first have to deploy that manpower before you can start using the 
system!!!

It becomes a rather big enterprise to install thinp for anyone!!!

Because to get it running takes no time at all!!! But to get it running 
well then implies huge investment.

I just wouldn't mind if this gap was smaller.

Many of the things you'd need to do are pretty standard. Running more 
and more cronjobs... well I am already doing that. But it is not just 
the maintenance of the cron job (installation etc.) but also the script 
itself that you have to first write.

That means for me and for others that may not be doing it professionally 
or in a larger organisation, the benefit of spending all that time may 
not weigh up to the cost it has and the result is then that you keep 
stuck with a deeply suboptimal situation in which there is little or no 
reporting or fixing, all because the initial investment is too high.

Commonly provided scripts just hugely reduce that initial investment.

For example the bigger vs. smaller system I imagined. Yes I am eager to 
make it. But I got other stuff to do as well :p.

And then, when I've made it, chances are high no one will ever use it 
for years to come.

No one else I mean.


>> So for example you configure max size for snapshot. When snapshots 
>> exceeds size gets flagged for removal. But removal only happens when 
>> other condition is met (threshold reach).
> 
> We are blamed already for having way too much configurable knobs....

Yes but I think it is better to script these things anyway.

Any official mechanism is only going to be inflexible when it goes that 
far.

Like I personally don't like SystemD services compared to cronjobs. 
Systemd services take longer to set up, have to agree to a descriptive 
language, and so on.

Then you need to find out exactly what are the extents of the 
possibilities of that descriptive language, maybe there is a feature you 
do not know about yet, but you can probably also code it using knowledge 
you already have and for which you do not need to read any man pages.

So I do create those services.... for the boot sequence... but anything 
I want to run regularly I still do with a cron job...

It's a bit archaic to install but... it's simple, clean, and you have 
everything in one screen.

>> So you would have 5 different interventions you could use that could 
>> be considered somewhat standard and the admit can just pick and choose 
>> or customize.
>> 
> 
> And we have way longer list of actions we want to do ;) We have not
> yet come to any single conclusion how to make such thing manageable
> for a user...

Hmm.. Well I cannot ... claim to have the superior idea here.

But Idk... I think you can focus on the model right.

Maintaining max snapshot consumption is one model.

Freezing bigger volumes to protect space for smaller volumes is another 
model.

Doing so based on a "critical" flag is another model... (not myself such 
a fan of that)... (more to configure).

Reserving max, set or configured space for a specific volume is another 
model.

(That would be actually equivalent to a 'critical' flag since only those 
volumes that have reserved space would become 'critical' and their space 
reservation is going to be the threshold to decide when to deny other 
volumes more space).

So you can simply call the 'critical flag' idea the same as the 'space 
reservation' idea.

The basic idea is that all space reservations get added together and 
become a threshold.

So that's just one model and I think it is the most important one.

"Reserve space for certain volumes" (but not all of them or it won't 
work). ;-).

This is what Gionatan refered to with the ZFS ehm... shit :p.

And the topic of this email thread.




So you might as well focus on that one alone as per mr. Jonathan's 
reply.

(Pardon for my language there).




While personally I also like the bigger versus smaller idea because you 
don't have to configure it.


The only configuration you need to do is to ensure that the more 
important volumes are a bit smaller.

Which I like.

Then there is automatic space reservation using fsfreezing.

Because the free space required for bigger volumes is always going to be 
bigger than that of smaller volumes.



>> But how expensive is it to do it say every 5 seconds?
> 
> If you have big metadata - you would keep you Intel Core busy all the 
> time ;)
> 
> That's why we have those thresholds.
> 
> Script is called at  50% fullness, then when it crosses 55%, 60%, ...
> 95%, 100%. When it drops bellow threshold - you are called again once
> the boundary is crossed...

How do you know when it is at 50% fullness?

> If you are proud sponsor of your electricity provider and you like the
> extra heating in your house - you can run this in loop of course...

> Threshold are based on  mapped size for whole thin-pool.
> 
> Thin-pool surely knows all the time how many blocks are allocated and 
> free for
> its data and metadata devices.

But didn't you just say you needed to process up to 16GiB to know this 
information?

I am confused?

This means the in-kernel policy can easily be implemented.

You may not know the size and attribution of each device but you do know 
the overall size and availability?

>> In any case the only policy you could have in-kernel would be either 
>> what Gionatan proposed (fixed reserved space for certain volumes) 
>> (easy calculation right) or potentially allocation freeze at threshold 
>> for non-critical volumes,
> 
> 
> In the single thin-pool  all thins ARE equal.

But you could make them unequal ;-).

> Low number of 'data' block may cause tremendous amount of provisioning.
> 
> With specifically written data pattern you can (in 1 second!) cause
> provisioning of large portion of your thin-pool (if not the whole one
> in case you have small one in range of gigabytes....)

Because you only have to write a byte to every extent, yes.

> And that's the main issue - what we solve in  lvm2/dm  - we want to be
> sure that when thin-pool is FULL  -  written & committed data are
> secure and safe.
> Reboot is mostly unavoidable if you RUN from a device which is 
> out-of-space -
> we cannot continue to use such device - unless you add MORE space to
> it within 60second window.

That last part is utterly acceptable.

> All other proposals solve only very localized solution and problems
> which are different for every user.
> 
> I.e. you could have a misbehaving daemon filling your system device
> very fast with logs...
> 
> In practice - you would need some system analysis and detect which
> application causes highest pressure on provisioning  - but that's well
> beyond range lvm2 team ATM with the amount of developers can
> provide....

And any space reservation would probably not do much; if it is not 
filled 100% now, it will be so in a few seconds, in that sense.

The goal was more to protect the other volumes, supposing that log 
writing happened on another one, for that other log volume not to impact 
the other main volumes.

So you have thin global reservation of say 10GB.

Your log volume is overprovisioned and starts eating up the 20GB you 
have available and then runs into the condition that only 10GB remains.

The 10GB is a reservation maybe for your root volume. The system 
(scripts) (or whatever) recognises that less than 10GB remains, that you 
have claimed it for the root volume, and that the log volume is 
intruding upon that.

It then decides to freeze the log volume.

But it is hard to decide what volume to freeze because it would need 
that run-time analysis of what's going on. So instead you just freeze 
all non-reserved volumes.

So all non-critical volumes in Gionatan and Brassow's parlance.


>> I just still don't see how one check per 4MB would be that expensive 
>> provided you do data collection in background.
>> 
>> You say size can be as low as 64kB... well.... in that case...
> 
> Default chunk size if 64k for the best 'snapshot' sharing - the bigger
> the pool chunk is the less like you could 'share' it between
> snapshots...

Okay.. I understand. I guess I was deluded a bit by non-thin snapshot 
behaviour (filled up really fast without me understanding why, and 
concluding that it was doing 4MB copies).

As well as of course that extents were calculated in whole numbers in 
overviews... apologies.

But attribution of an extent to a snapshot will still be done in 
extent-sizes right?

So I was just talking about allocation, nothing else.

BUT if allocator operates on 64kB requests, then yes...

> (As pointed in other thread - ideal chunk for best snapshot sharing
> would be 4K - but that's not affordable for other reasons....)

Okay.

>>        2) I would freeze non-critical volumes ( I do not write to 
>> snapshots so that is no issue ) when critical volumes reached safety 
>> threshold in free space ( I would do this in-kernel if I could ) ( But 
>> Freezing In User-Space is almost the same ).
> 
> There are lots of troubles when you have freezed filesystems present
> in your machine fs tree... -  if you know all connections and
> restrictions - it can be 'possibly' useful - but I can't imagine this
> being useful in generic case...

Well, yeah. Linux.

(I mean, just a single broken NFS or CIFS connection can break so 
much....).



> And more for your thinking -
> 
> If you have pressure on provisioning caused by disk-load on one of
> your 'critical' volumes this FS 'freezeing' scripting will 'buy' you
> only couple seconds

Oh yeah of course, this is correct.

> (depends how fast drives you have and how big
> thresholds you will use) and you are in the 'exact' same situation -
> expect now you have  system in bigger troubles - and you already might
> have freezed other systems apps by having them accessing your
> 'low-prio' volumes....

Well I guess you would reduce non-critical volumes to single-purpose 
things.

Ie. only used by one application.

> And how you will be solving 'unfreezing' in cases thin-pool usage
> drops down is also pretty interesting topic on its own...

I guess that would be manual?

> I need to wish good luck when you will be testing and developing all
> this machinery.

Well as you say it has to be an anomaly in the first place -- an error 
or problem situation.

It is not standard operation.

So I don't think the problems of freezing are bigger than the problems 
of rebooting.

The whole idea is that you attribute non-critical volumes to single apps 
or single purposes so that when they run amock, or in any case, that if 
anything runs amock on them...

Yes it won't protect the critical volumes from being written to.

But that's okay.

You don't need to automatically unfreeze.

You need to send an email and say stuff has happened ;-).

"System is still running but some applications may have crashed. You 
will need to unfreeze and restart in order to solve it, or reboot if 
necessary. But you can still log into SSH, so maybe you can do it 
remotely without a console ;-)".

I don't see any issues with this.

One could say: use filesystem quotas.

Then that involves setting up users etc.

Setting up a quota for a specific user on a specific volume...

All more configuration.

And you're talking mostly about services of course.

The benefit (and danger) of LVM is that it is so easy to create more 
volumes.

(The danger being that you now also need to back up all these volumes).

(Independently).

>>> Default is to auto-extend thin-data & thin-metadata when needed if 
>>> you
>>> set threshold bellow 100%.
>> 
>> Q: In a 100% filled up pool, are snapshots still going to be valid?
>> 
>> Could it be useful to have a default policy of dropping snapshots at 
>> high consumption? (ie. 99%). But it doesn't have to be default if you 
>> can easily configure it and the scripts are available.
> 
> All snapshots/thins with 'fsynced' data are always secure.
> Thin-pool is protecting all user-data on disk.
> 
> The only lost data are those flying in your memory (unwritten on disk).
> And depends on you 'page-cache' setup how much that can be...

That seemes pretty secure. Thank you.

So there is no issue with snapshots behaving differently. It's all the 
same and all committed data will be safe prior to the fillup and not 
change afterward.

I guess.