[Pulp-dev] PulpExport performance anecdote

[Grant Gainey]

A bored period this weekend led me to some performance-numbers for the
PulpExport work we're doing right now.

   - Imported CentOS7 and exported it as 2GB chunks.
   - Memory use for the pulp-worker never exceeded *125MB*
   - Disk usage went up only by the size of the resulting set of .tar.gz
   - Time for full-export was *8m50s*

Found one bug around persisting the chunked-file-checksums (note to self:
don't call a routine that generates a string with "now to the minute" in
it, when the operation takes more than a minute), and how to get Content
and ContentArtifact exported without relying on pulp_ids...

G

Raw data:
(pulp) [vagrant at pulp3-source-fedora31 import_export]$ http
:/pulp/api/v3/tasks/fcea3cc6-f02d-4b2a-a552-3e407f0bb338/
{
    "child_tasks": [],
    "created_resources": [

"/pulp/api/v3/exporters/core/pulp/1cfb0006-5909-433c-b931-ef8dd1146794/exports/63960fdd-d01b-4af9-8527-0a5e788ae430/"
    ],
    "error": null,
    "finished_at": "2020-05-24T21:26:50.367799Z",
    "name": "pulpcore.app.tasks.export.pulp_export",
    "parent_task": null,
    "progress_reports": [],
    "pulp_created": "2020-05-24T21:18:00.663723Z",
    "pulp_href": "/pulp/api/v3/tasks/fcea3cc6-f02d-4b2a-a552-3e407f0bb338/",
    "reserved_resources_record": [

"/pulp/api/v3/exporters/core/pulp/1cfb0006-5909-433c-b931-ef8dd1146794/"
    ],
    "started_at": "2020-05-24T21:18:00.795450Z",
    "state": "completed",
    "task_group": null,
    "worker": "/pulp/api/v3/workers/648c157b-ebe6-4816-aa93-2b88dc2c19ea/"
}
(pulp) [vagrant at pulp3-source-fedora31 import_export]$ http
:/pulp/api/v3/exporters/core/pulp/1cfb0006-5909-433c-b931-ef8dd1146794/exports/63960fdd-d01b-4af9-8527-0a5e788ae430/
{
    "exported_resources": [

"/pulp/api/v3/repositories/rpm/rpm/6c6e3e91-65e8-40d9-afce-18d720bdcaea/versions/1/"
    ],
    "output_file_info": {

"/tmp/exports/export-63960fdd-d01b-4af9-8527-0a5e788ae430-20200524_2118.tar.gz.0000":
"6883aa4af0f264d6fc4904e92bd979c696528fd6820e83dd5d062c64b1551f2c",

"/tmp/exports/export-63960fdd-d01b-4af9-8527-0a5e788ae430-20200524_2118.tar.gz.0001":
"15ef9fd397a9b6ecd08d140f81d6d75621e3eb38716320ba98db6789f121da10",

"/tmp/exports/export-63960fdd-d01b-4af9-8527-0a5e788ae430-20200524_2118.tar.gz.0002":
"2e46f4bef01ef07443473c97577600b5f5269aa6b0319fa5f7021f40972af584",

"/tmp/exports/export-63960fdd-d01b-4af9-8527-0a5e788ae430-20200524_2118.tar.gz.0003":
"bd0f7eb61be2b6cb852c12f6d726cd9a5d440446041da3352ec31d717bd74628",

"/tmp/exports/export-63960fdd-d01b-4af9-8527-0a5e788ae430-20200524_2118.tar.gz.0004":
"64446d4d3530fbb37ce074969e1bc02027aa2957f7ce4443d6ac6d916fea1790"
    },
    "params": {
        "chunk_size": "2GB"
    },
    "pulp_created": "2020-05-24T21:18:00.632656Z",
    "pulp_href":
"/pulp/api/v3/exporters/core/pulp/1cfb0006-5909-433c-b931-ef8dd1146794/exports/63960fdd-d01b-4af9-8527-0a5e788ae430/",
    "task": "/pulp/api/v3/tasks/fcea3cc6-f02d-4b2a-a552-3e407f0bb338/"
}
(pulp) [vagrant at pulp3-source-fedora31 import_export]$ ls -ltrh /tmp/exports/
total 9.7G
...2.0G May 24 21:19
export-63960fdd-d01b-4af9-8527-0a5e788ae430-20200524_2118.tar.gz.0000
...2.0G May 24 21:20
export-63960fdd-d01b-4af9-8527-0a5e788ae430-20200524_2118.tar.gz.0001
...2.0G May 24 21:22
export-63960fdd-d01b-4af9-8527-0a5e788ae430-20200524_2118.tar.gz.0002
...2.0G May 24 21:23
export-63960fdd-d01b-4af9-8527-0a5e788ae430-20200524_2118.tar.gz.0003
...1.7G May 24 21:25
export-63960fdd-d01b-4af9-8527-0a5e788ae430-20200524_2118.tar.gz.0004
(pulp) [vagrant at pulp3-source-fedora31 import_export]$
-- 
Grant Gainey
Principal Software Engineer, Red Hat System Management Engineering
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-dev/attachments/20200524/6a030a32/attachment.htm>