[Pulp-list] What do you want from an API documentation solution?

[Jay Dobies]

I'd warn you guys this is a long read, but if you haven't realized by 
now that I'm annoyingly long-winded, you haven't been paying attention.

I have a sprint story related to an API docs solution and have been 
trying to find something to get in place before everyone runs around 
doing their tasks to document the currently undocumented APIs. I'm 
running into problems defining what it is we're trying to solve and 
wanted to think/talk it out before throwing in another attempt at a 
solution. I also hope you'll suffer through reading it all and we can 
discuss at a deep dive this week; bring your ideas.

= Problem =
The way I see it, there are three problems with our API docs:
* Missing
* Incomplete
* Incorrect

== Missing ==
Not much to say to describe this. This also isn't a blame thing; we knew 
they'd be a priority one day and did what we could. Now, they are a much 
bigger priority and we need to get them done and keep them maintained.

== Incomplete ==
By this I mean inconsistencies between API docs. Some have examples, 
some don't. Very few appropriately describe error codes. Things like that.

== Incorrect ==
This isn't so much meant to refer to those APIs that are done wrong 
themselves, more it's meant to address the idea that things change and 
the docs need to be updated to reflect those changes.

= Solutions =

== Automated ==
We attempted to do some sort of generation of API docs from docstrings 
and there's been another increased interest in this area with the 
proposal of sphinx. I'm not sure either are the answer.

The perfect solution would be to write code and have the API docs 
generated automagically from the code itself. That's never going to 
happen since we're talking about REST APIs, not simply python APIs. The 
REST APIs need to cover things like HTTP status codes and example URLs 
and response bodies. Those things just aren't possible by code 
introspection.

The last attempt was to use a specific format in the docstrings of the 
methods and have the docs generated from there. I'd argue it failed for 
a few reasons:

1. No clearly defined template (could be that I just never knew where to 
look, but there is clearly a format I had to follow and I never knew how 
to get started).

2. Manual process of generate .wiki file and then manually copy/paste 
into the wiki. It's actually _more_ work than editing the wiki directly.

Assuming we fix it so that there's no intermediate manual copy/paste 
step into the wiki, I'm still not convinced it's going to work for a few 
reasons:

1. Most developers don't bother reviewing docstrings when they change 
code. It's very unfortunate, but also very true. In Pulp alone there are 
a number of places where the docstrings reflect an older version of the 
method (PyCharm can highlight this for you, but I digress).

2. It's a new syntax. That means you still have to dig up the template 
to drop in the docstring and figure out how to use it.

So given those two issues, I don't think the question is simply if we 
edit them in the wiki or in code. Either way you still have to hunt 
around for the template. You still have to remember to actually do it. 
You still have to remember to review it after you've changed something.

I'll talk more about technical solutions later. I'm curious what people 
think about this point though. I haven't dug too much into sphinx yet, 
but I have no reason to think it won't be subject to all of the above 
issues that wouldn't render the wiki (almost*) as easy.

* The biggest benefit to sphinx is co-locating the docs in the codebase 
which arguably isn't necessarily a benefit; you can put a browser 
editing a wiki side-by-side with code and not have to scroll up and down 
to hop between impl and docstring. Plus you have more flexibility on a 
wiki; sphinx calls itself WYSIWYG but it's exactly the opposite of that, 
whereas with a wiki it's much simpler to hit "preview".

== Discipline ==
Since we won't be able to derive the API docs purely from code, at the 
end of the day we all just need to be better about stopping for a second 
to think about if we need to edit them (I'd like to see us use a 
boyscout rule* mentality). In the past, it was easier to blow these off 
since people weren't using them. Now, we have both a community and 
Katello, so we need to be aware of the increased priority (and if you 
still don't buy in, hop to the Katello team for a sprint and use an area 
of Pulp you didn't write; you'll hate us).

* http://programmer.97things.oreilly.com/wiki/index.php/The_Boy_Scout_Rule

== Peer Review ==
Let me get this out of the way now: I'm not suggesting everyone have to 
take part in API documentation review.

John has the skill and patience for debugging really obtuse issues. I 
hate that shit, so I never volunteer for it. I leave it for John (who I 
_think_ enjoys those sorts of things).

Same goes for reviewing docs, both user guide and API docs. Some people 
have no interest in it, so there's no reason to force them to have to 
review other people's docs. Personally, I get an odd OCD satisfaction 
knowing we have really clean, readable API docs. So I'm happy to 
volunteer to do a quick proof read as a fresh pair of eyes who doesn't 
know the code itself and who is compulsive enough to make sure the whole 
docs package fits together.

Different skills from different people. Let's take advantage of that and 
request people who are willing to to take a quick once over when you 
finish docs.

== Template ==
Sayli sent out a proposed template a few weeks ago. I don't know where 
it is on the wiki, but I should. Bookmark the template so it feels like 
less of a chore to dig it up when you have to add new APIs.

I can also do another quick review on what pages on the wiki are slurped 
into the look & feel process when the userguide is generated.

We should also add in a dirt simple way of getting the CLI to dump URLs, 
requests, and responses. I already know where it'd go, I just haven't 
done it yet. Then we all have a consistent way of getting that example 
data for the docs.

== Technical Approach ==
It's probably clear by now that I'm a fan of just using the wiki like we 
did in the beginning. I'm not convinced an in-code solution gets us 
anything that simple discipline wouldn't; arguably, it's yet another 
syntax we have to learn and frankly, I have a hard enough time just 
remembering one wiki format from another.

I'm still willing to look into sphinx, but my early impressions are that 
we're going to have to invest a reasonably large amount of work getting 
it to look at the places in code we want it to look and format it for 
the web site. Ultimately, even if we do, you still have to find the 
template for what our API docs should look like and remember to 
write/update them, so I'm not convinced it's any better than the wiki -> 
user guide process we already have in place.

= Conclusion =
Actually, there is no conclusion, I just needed an ending. Please give 
it some thought though. Why are we in this position in the first place? 
Is an in-code solution really going to make that much of a difference to 
you when you edit something or write a new API?

Let me know in some way how you want to proceed. If you don't want to 
get into a giant reply-to-list war on this thread and think it'd be 
worth a deep dive, ping me separately and I'll set one up. If you don't 
really care and just want to be told "here's the new process", that's 
fine too, just let me know. Otherwise, I'm at a bit of a pause on how to 
proceed with this story, so I'm waiting to see what others think.

-- 
Jay Dobies
Freenode: jdob @ #pulp
http://pulpproject.org | http://blog.pulpproject.org