[et-mgmt-tools] Re: Question about virt-top csv...

[Richard W.M. Jones]

On Thu, Mar 20, 2008 at 06:58:42AM -0600, Spencer Parker wrote:
> I am using Python to parse the file and then dumps the info into a MySQL
> database to then be displayed and aggregated for a webpage.  O know I just
> need to parse every 25 places in the index...then just do the next 28 and so
> on.  I'm pretty beginner at programming in general...so its nice having
> something like this to help teach me Python.

[Please CC replies to et-mgmt-list]

Attached is an example CSV file from the latest version of virt-top so
we have something in common to talk about.  Load this into a
spreadsheet program like OpenOffice Calc.

The basic problem with CSV is that it's a 2D file format, and if you
have more dimensions than two in your data, as in this case, you have
to squeeze them into two somehow.  This is what I'm doing here.

The data format is:

     [common columns]   [domain columns]
time                    [dom1]  [dom2]  [dom3] ....
 |
 |
 V

Looking at the spreadsheet, columns A-Q are the common columns.

Column R is the first domain column.  Note the title "Domain ID".
This title always marks the first domain column.

Column Y is the last domain column for the first domain. Note that it
is the last column that has a title, and this is how you detect it.

Column Z is the first domain column for the second domain.  It
contains the domain ID of the second domain.  The second domain goes
through columns Z - AG.

If there was a third domain it would start at column AH (which would
be its domain ID), and so on.

You can work out the width of each domain as Y-R (8 columns in this
instance, but you have to calculate it because it might be different
in future versions of virt-top, or even in current versions of
virt-top if you use the --no-csv-* command line flags).

BTW, which Python library are you using to parse CSV?  You should
always use a library.  You can't get away with just splitting at comma
characters.  If you have the ocaml-csv module installed from Fedora
then there's a command-line tool called csvtool included which can
also do some manipulations of CSV files correctly.

Rich.

-- 
Richard Jones, Emerging Technologies, Red Hat  http://et.redhat.com/~rjones
virt-p2v converts physical machines to virtual machines.  Boot with a
live CD or over the network (PXE) and turn machines into Xen guests.
http://et.redhat.com/~rjones/virt-p2v
-------------- next part --------------
Hostname,Time,Arch,Physical CPUs,Count,Running,Blocked,Paused,Shutdown,Shutoff,Crashed,Active,Inactive,%CPU,Total memory (KB),Total guest memory (KB),Total CPU time (ns),Domain ID,Domain name,CPU (ns),%CPU,Block RDRQ,Block WRRQ,Net RXBY,Net TXBY
amd.home.annexia.org,13:03:50,x86_64,4,2,2,0,0,0,0,0,2,0,0.0,1024000,1024000,0,1,rhel51x32kvm,0.,0.,,,,,2,rhel51x64kvm,0.,0.,,,,
amd.home.annexia.org,13:03:53,x86_64,4,2,2,0,0,0,0,0,2,0,6.6,1024000,1024000,790000000,1,rhel51x32kvm,430000000.,3.58034253276,,,104,0,2,rhel51x64kvm,360000000.,2.99749607394,,,104,0
amd.home.annexia.org,13:03:56,x86_64,4,2,2,0,0,0,0,0,2,0,6.5,1024000,1024000,780000000,1,rhel51x32kvm,440000000.,3.66482943344,,,52,342,2,rhel51x64kvm,340000000.,2.83191365311,,,394,0
amd.home.annexia.org,13:03:59,x86_64,4,2,2,0,0,0,0,0,2,0,6.5,1024000,1024000,780000000,1,rhel51x32kvm,440000000.,3.66471706899,,,446,0,2,rhel51x64kvm,340000000.,2.83182682603,,,104,342
amd.home.annexia.org,13:04:02,x86_64,4,2,2,0,0,0,0,0,2,0,6.3,1024000,1024000,760000000,1,rhel51x32kvm,440000000.,3.66482943344,,,52,0,2,rhel51x64kvm,320000000.,2.66533049704,,,52,0
amd.home.annexia.org,13:04:05,x86_64,4,2,2,0,0,0,0,0,2,0,6.8,1024000,1024000,820000000,1,rhel51x32kvm,440000000.,3.66482797789,,,104,0,2,rhel51x64kvm,380000000.,3.16507870818,,,104,0
amd.home.annexia.org,13:04:08,x86_64,4,2,2,0,0,0,0,0,2,0,6.2,1024000,1024000,750000000,1,rhel51x32kvm,430000000.,3.57913635914,,,52,0,2,rhel51x64kvm,320000000.,2.66354333703,,,52,0
amd.home.annexia.org,13:04:11,x86_64,4,2,2,0,0,0,0,0,2,0,6.6,1024000,1024000,790000000,1,rhel51x32kvm,450000000.,3.74807337613,,,446,0,2,rhel51x64kvm,340000000.,2.83187766197,,,104,342
amd.home.annexia.org,13:04:14,x86_64,4,2,2,0,0,0,0,0,2,0,6.6,1024000,1024000,790000000,1,rhel51x32kvm,440000000.,3.6622060958,,,52,342,2,rhel51x64kvm,350000000.,2.9131184853,,,394,0
amd.home.annexia.org,13:04:17,x86_64,4,2,2,0,0,0,0,0,2,0,6.6,1024000,1024000,790000000,1,rhel51x32kvm,440000000.,3.66491473022,,,104,0,2,rhel51x64kvm,350000000.,2.91527308086,,,104,0
-------------- next part --------------
csvtool - Copyright (C) 2005-2006 Richard W.M. Jones, Merjis Ltd.

csvtool is a tool for performing manipulations on CSV files from shell scripts.

Summary:
  csvtool [-options] command [command-args] input.csv [input2.csv [...]]

Commands:
  col <column-spec>
    Return one or more columns from the CSV file.

    For <column-spec>, see below.

      Example: csvtool col 1-3,6 input.csv > output.csv

  namedcol <names>
    Assuming the first row of the CSV file is a list of column headings,
    this returned the column(s) with the named headings.

    <names> is a comma-separated list of names.

      Example: csvtool namedcol Account,Cost input.csv > output.csv

  width
    Print the maximum width of the CSV file (number of columns in the
    widest row).

  height
    Print the number of rows in the CSV file.

    For most CSV files this is equivalent to 'wc -l', but note that
    some CSV files can contain a row which breaks over two (or more)
    lines.

  setcolumns cols
    Set the number of columns to cols (this also makes the CSV file
    square).  Any short rows are padding with blank cells.  Any
    long rows are truncated.

  setrows rows
    'setrows n' sets the number of rows to 'n'.  If there are fewer
    than 'n' rows in the CSV files, then empty blank lines are added.

  head rows
  take rows
    'head n' and 'take n' (which are synonyms) take the first 'n'
    rows.  If there are fewer than 'n' rows, padding is not added.

  drop rows
    Drop the first 'rows' rows and return the rest (if any).

      Example:
        To remove the headings from a CSV file with headings:
          csvtool drop 1 input.csv > output.csv

        To extract rows 11 through 20 from a file:
          csvtool drop 10 input.csv | csvtool take 10 - > output.csv

  cat
    This concatenates the input files together and writes them to
    the output.  You can use this to change the separator character.

      Example: csvtool -t TAB -u COMMA cat input.tsv > output.csv

  join <column-spec1> <column-spec2>
    Join (collate) multiple CSV files together.

    <column-spec1> controls which columns are compared.

    <column-spec2> controls which columns are copied into the new file.

      Example:
        csvtool join 1 2 coll1.csv coll2.csv > output.csv

        In the above example, if coll1.csv contains:
          Computers,$40
          Software,$100
        and coll2.csv contains:
          Computers,$50
        then the output will be:
          Computers,$40,$50
          Software,$100,

  square
    Make the CSV square, so all rows have the same length.

      Example: csvtool square input.csv > input-square.csv

  sub r c rows cols
    Take a square subset of the CSV, top left at row r, column c, which
    is rows deep and cols wide.  'r' and 'c' count from 1, or
    from 0 if -z option is given.

  replace <column-spec> update.csv original.csv
    Replace rows in original.csv with rows from update.csv.  The columns
    in <column-spec> only are used to compare rows in input.csv and
    update.csv to see if they are candidates for replacement.

      Example:
        csvtool replace 3 updates.csv original.csv > new.csv
        mv new.csv original.csv

  call command
    This calls the external command (or shell function) 'command'
    followed by a parameter for each column in the CSV file.  The
    external command is called once for each row in the CSV file.
    If any command returns a non-zero exit code then the whole
    program terminates.

      Tip:
        Use the shell command 'export -f funcname' to export
        a shell function for use as a command.  Within the
        function, use the positional parameters $1, $2, ...
        to refer to the columns.

      Example (with a shell function):
        function test {
          echo Column 1: $1
          echo Column 2: $2
        }
        export -f test
        csvtool call test my.csv

        In the above example, if my.csv contains:
          how,now
          brown,cow
        then the output is:
          Column 1: how
          Column 2: now
          Column 1: brown
          Column 2: cow

  readable
    Print the input CSV in a readable format.

Column specs:
  A <column-spec> is a comma-separated list of column numbers
  or column ranges.

    Examples:
      1                       Column 1 (the first, leftmost column)
      2,5,7                   Columns 2, 5 and 7
      1-3,5                   Columns 1, 2, 3 and 5
      1,5-                    Columns 1, 5 and up.

  Columns are numbered starting from 1 unless the -z option is given.

Input files:
  csvtool takes a list of input file(s) from the command line.

  If an input filename is '-' then take input from stdin.

Output file:
  Normally the output is written to stdout.  Use the -o option
  to override this.

Separators:
  The default separator character is , (comma).  To change this
  on input or output see the -t and -u options respectively.

  Use -t TAB or -u TAB (literally T-A-B!) to specify tab-separated
  files.

Options:
  -t Input separator char.  Use -t TAB for tab separated input.
  -u Output separator char.  Use -u TAB for tab separated output.
  -o Write output to file (instead of stdout)
  -z Number columns from 0 instead of 1
  - 
  -help  Display this list of options
  --help  Display this list of options