[Libguestfs] libnbd flamegraph Odd benchmark results

[Richard W.M. Jones]

On Sun, May 30, 2021 at 09:27:18PM +0530, Abhay Raj Singh wrote:
> 
> 
>     The question (for you!) is how we can choose better settings, and
>     whether we should choose
> 
>  
>  In my opinion configuration resolution will be best if resolved in the
> following order of increasing priority/weightage.
> 1. Auto-detect depending on the system parameters.
> Auto-detection of best params is a must IMO as there's
> enough diversity in server hardware that one size won't fit
> all can be implemented without much effort(excluding benchmarking).

What are the inputs to this formula?  I think the only ones we can
portably get are: number of cores, total memory.  Free memory
possibly, although it's a slippery concept (is that "free" before or
after the page cache is freed?)  Other inputs that could make a
difference, such as number of NUMA nodes, size of L2/L3 cache, core
clock rate, ... are difficult to get reliably and portably.

Still, if the formula is clear and understandable to users it could be
fine.

> 2. Configuration file: A yaml, toml, or any other simple format
> - Load a different config file with *nbdcopy --file=custom-config.yaml*
> - Maybe generate the default config file having values from the auto-detect
> phase.
> nbdcopy --generate-config
> 
> ~/.config/nbdcopy/config.yaml (used to get values)
> ~/.config/nbdcopy/example.yaml (generated for reference from auto)

I think this part is a horrible idea for several reasons:

- Adds hidden state: As an end user you need to know the configuration
  file exists.  And if end users have problems we need them to send us
  the configuration file, and my experience with virt-v2v tells me
  that follow-up questions about bugs are rarely answered.

- Scripts will behave differently depending on where they are running.

- Any script or program consuming nbdcopy must disable the config file
  processing.  It will be completely unwanted for them to depend on
  the random contents of a config file that could greatly affect (or
  even break) the program.

- Adds much complexity to the implementation of the tool.

- Yaml etc are awful formats for configuration files in general.

Rich.

> 3. CLI params
> A quick way to override configuration file params
> ex: functionally same
> `nbdcopy --requests=4 --config=custom-config.yaml`
> `nbdcopy --config=custom-config.yaml --requests=4`
> 
> The flow of initialization:
> Read config file ----override ---> CLI params --- missing params -->
> auto-detect fill
> or
> easier implementation:
> auto-detect fill --- override ---> config file --- override ---> CLI params
> 
> 
> The rationale for choosing configuration file method:
> - Easier to use avoiding long commands (less typing, scripts easier to read)
> - Easy switch between multiple configurations and override them.
> - Documentation and configuration in the same place
> So easier for someone else to understand why certain parameters were chosen.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
libguestfs lets you edit virtual machines.  Supports shell scripting,
bindings from many languages.  http://libguestfs.org