<div dir="ltr">A bored period this weekend led me to some performance-numbers for the PulpExport work we're doing right now.<div><ul><li>Imported CentOS7 and exported it as 2GB chunks.</li><li>Memory use for the pulp-worker never exceeded <b>125MB</b></li><li>Disk usage went up only by the size of the resulting set of .tar.gz</li><li>Time for full-export was <b>8m50s</b></li></ul></div><div>Found one bug around persisting the chunked-file-checksums (note to self: don't call a routine that generates a string with "now to the minute" in it, when the operation takes more than a minute), and how to get Content and ContentArtifact exported without relying on pulp_ids...</div><div><br></div><div>G</div><div><br></div><div>Raw data:</div><div><font face="monospace" size="1">(pulp) [vagrant@pulp3-source-fedora31 import_export]$ http :/pulp/api/v3/tasks/fcea3cc6-f02d-4b2a-a552-3e407f0bb338/<br>{<br>    "child_tasks": [],<br>    "created_resources": [<br>        "/pulp/api/v3/exporters/core/pulp/1cfb0006-5909-433c-b931-ef8dd1146794/exports/63960fdd-d01b-4af9-8527-0a5e788ae430/"<br>    ],<br>    "error": null,<br>    "finished_at": "2020-05-24T21:26:50.367799Z",<br>    "name": "pulpcore.app.tasks.export.pulp_export",<br>    "parent_task": null,<br>    "progress_reports": [],<br>    "pulp_created": "2020-05-24T21:18:00.663723Z",<br>    "pulp_href": "/pulp/api/v3/tasks/fcea3cc6-f02d-4b2a-a552-3e407f0bb338/",<br>    "reserved_resources_record": [<br>        "/pulp/api/v3/exporters/core/pulp/1cfb0006-5909-433c-b931-ef8dd1146794/"<br>    ],<br>    "started_at": "2020-05-24T21:18:00.795450Z",<br>    "state": "completed",<br>    "task_group": null,<br>    "worker": "/pulp/api/v3/workers/648c157b-ebe6-4816-aa93-2b88dc2c19ea/"<br>}</font><br></div><div><font face="monospace" size="1">(pulp) [vagrant@pulp3-source-fedora31 import_export]$ http :/pulp/api/v3/exporters/core/pulp/1cfb0006-5909-433c-b931-ef8dd1146794/exports/63960fdd-d01b-4af9-8527-0a5e788ae430/<br>{<br>    "exported_resources": [<br>        "/pulp/api/v3/repositories/rpm/rpm/6c6e3e91-65e8-40d9-afce-18d720bdcaea/versions/1/"<br>    ],<br>    "output_file_info": {<br>        "/tmp/exports/export-63960fdd-d01b-4af9-8527-0a5e788ae430-20200524_2118.tar.gz.0000": "6883aa4af0f264d6fc4904e92bd979c696528fd6820e83dd5d062c64b1551f2c",<br>        "/tmp/exports/export-63960fdd-d01b-4af9-8527-0a5e788ae430-20200524_2118.tar.gz.0001": "15ef9fd397a9b6ecd08d140f81d6d75621e3eb38716320ba98db6789f121da10",<br>        "/tmp/exports/export-63960fdd-d01b-4af9-8527-0a5e788ae430-20200524_2118.tar.gz.0002": "2e46f4bef01ef07443473c97577600b5f5269aa6b0319fa5f7021f40972af584",<br>        "/tmp/exports/export-63960fdd-d01b-4af9-8527-0a5e788ae430-20200524_2118.tar.gz.0003": "bd0f7eb61be2b6cb852c12f6d726cd9a5d440446041da3352ec31d717bd74628",<br>        "/tmp/exports/export-63960fdd-d01b-4af9-8527-0a5e788ae430-20200524_2118.tar.gz.0004": "64446d4d3530fbb37ce074969e1bc02027aa2957f7ce4443d6ac6d916fea1790"<br>    },<br>    "params": {<br>        "chunk_size": "2GB"<br>    },<br>    "pulp_created": "2020-05-24T21:18:00.632656Z",<br>    "pulp_href": "/pulp/api/v3/exporters/core/pulp/1cfb0006-5909-433c-b931-ef8dd1146794/exports/63960fdd-d01b-4af9-8527-0a5e788ae430/",<br>    "task": "/pulp/api/v3/tasks/fcea3cc6-f02d-4b2a-a552-3e407f0bb338/"<br>}</font></div><div><font face="monospace" size="1">(pulp) [vagrant@pulp3-source-fedora31 import_export]$ ls -ltrh /tmp/exports/<br>total 9.7G<br>...2.0G May 24 21:19 export-63960fdd-d01b-4af9-8527-0a5e788ae430-20200524_2118.tar.gz.0000<br>...2.0G May 24 21:20 export-63960fdd-d01b-4af9-8527-0a5e788ae430-20200524_2118.tar.gz.0001<br>...2.0G May 24 21:22 export-63960fdd-d01b-4af9-8527-0a5e788ae430-20200524_2118.tar.gz.0002<br>...2.0G May 24 21:23 export-63960fdd-d01b-4af9-8527-0a5e788ae430-20200524_2118.tar.gz.0003<br>...1.7G May 24 21:25 export-63960fdd-d01b-4af9-8527-0a5e788ae430-20200524_2118.tar.gz.0004<br>(pulp) [vagrant@pulp3-source-fedora31 import_export]$</font><br></div><div>-- <br></div><div dir="ltr" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div>Grant Gainey</div><div>Principal Software Engineer, Red Hat System Management Engineering</div></div></div></div></div></div>