The Idea
I had a brilliant idea yesterday, and couldn’t wait to get home so I could start coding it. Things were moving along great. I had my database created, I’d gotten Code Igniter configured and running with my standard set of libraries, helpers, etc. and I was hammering away at some code.
Damnation
And then it happened. I got to the heart and soul of the application: the part that interacted with the NewsGator API. Almost instantly, my entire world crashed in around me.
What’s that? You’ve never dealt with the NewsGator API? Consider yourself lucky. Only murderers and rapists should be doomed to such a fate. The NewsGator API is by far, and without a doubt, the worst API I have ever dealt with - and I’ve dealt with quite a few in the past.
The Spectrum
Think of the overall web API spectrum as a scale.
On the one end, you’ve got Yahoo!. Yahoo! offers what could be considered the Utopia of web APIs. They do your “standard” XML and SOAP interfaces, but they also offer a plethora of other options for many of their services: various REST APIs, JSON, and even plain old serialized PHP are all readily available for use. Whatever the language, whatever the platform, you can get data out of Yahoo! with a minimum of pain.
On the far, extreme, totally opposite end of this vast, gigantic, absolutely humongous spectrum of available APIs is NewsGator. You name it, they’ve probably done it wrong with their API. First problem? SOAP.
Ok, SOAP sounds great, and I have no doubt that it has its purposes, but for the love of god people… why? There are so many other alternatives available that could greatly simplify the process we really care about here: getting bits back and forth. I won’t say I have to end-all of answers as to the “perfect” format, but I can readily recommend any number of alternatives to SOAP.
REST? When? I’m working around bugs 24×7!
What? NewsGator has a REST API?
Well, you’d be partially right, at least. You POST your request and get back the same gobbly-gook XML data that you’d have gotten from the SOAP interface. This seems a bit backwards to me, since SOAP is XML-based and a SOAP interpreter would therefore also interpret the XML you’re receiving back (well, most of it). Tacking an XML parser on top of your REST API just seems perverted.
Oh yeah, there’s also no documentation for the REST API… None… Notta. The only thing you’ll find of help is a poorly-formatted PDF document that seems to kinda, maybe, sorta, give you a general idea of what’s going on… To top it off, apparently there’s no equivalent REST-based API for the SOAP-based interface called “PostItem”, according to forum posts.
Sample, Shmample!
Finally, the only code samples you’ll get from NewsGator are in .NET (C# to be precise). Talk about a long way migrating that over to PHP or, god save you, something like Ruby… Yahoo!, on the other hand, provides examples for most of their APIs in .NET, Java, Perl, PHP (4 and 5), Python, etc…
The Adventure Begins…
In my journeys over the past two days, I’ve hit more bumps in the road than I do on my way to work… In the mountains… In the middle of the ghetto… Through a huge swamp… Infested with mutant turtle speed bumps. It’s like a huge real-life version of Frogger, all wrapped in disgusting XML nastiness.
First, and not the least of my problems, has been a bug in PHP’s SOAP implementation that apparently has existed since September of 2004 and which was supposedly fixed in CVS in March of 2005. Here it is July of 2007, and I’m still hitting a problem parsing out a WSDL file that contains a ref attribute in it.
Alright, I know, we’re here to talk about NewsGator’s crappy API, not PHP’s crappy SOAP implementation1.
NameThatTune
As for NewsGator, let’s start small, shall we?
Ever noticed a difference in naming styles between .NET developers and those developers more commonly found on *nix platforms (Perl, Python, etc.)? Of course I have no empirical data to back me up here, but it seems like .NET developers commonly use functions that FollowACamelCasedNamingPattern, while *nix developers_prefer_to_space_things_out_with_underscores.
I’m sure it’s just personal preference, but I prefer the underscored version, which just seems more easily read to me. That said, NewsGator has a serious case of .NET-itis2.
Now it’s time for some real complaints. Let’s dive in.
You’re Lucky Number 17!
One of the first things you’ll obviously want to do when dealing with RSS feeds is get a list of the user’s subscriptions, right? For that you’d probably want the GetSubscriptionsList call… Or, as it’s called in the WSDL declaration, just GetSubscriptions (remember that…). What does the documentation say this call returns?
The result of this call is an OPML document with NewsGator extensions.
Yes, the only link about these extensions points to the raw XSD definition. No further documentation on what exactly each of the returned fields means is forthcoming. To top it off, we’ve taken SOAP and XML to another level entirely. Stick that OPML in your PHP pipe and try to smoke it, fool… You’ll end up using the built in XML parser and looping through looking for tags of the type OUTLINE, if that’s any hint.
So what does the call actually spit back? Well, it’s a huge list of feeds (which I parsed out into an array). Each feed has a number of detail items (16, in fact - and remember, none of them are documented). Some of these items are self-explanatory, such as “TITLE”, “NG:ID”, and “XMLURL”. Others aren’t quite so helpful: “DESCRIPTION” (My blog’s tagline, for instance, is stored in “TEXT”), “NG:SYNCXMLURL”, “NG:UNSEEN”, and “NG:USEDEFAULTCREDENTIALS” among them.
Alright, great. Now I’ve got a list of subscriptions for the user. How many unread items are in each? Well, out of those 16 fields I just got back for each feed in GetSubscriptions, one of them is “NG:UNREAD”. It’s a boolean value.
Apparently we’re meant to use the call GetSubscriptionCounts for this purpose. It returns a similar OPML document as its previous counterpart, but only contains 9 fields for each post this time. Out of those 9 fields, 8 of them are exactly identical to those found in GetSubscriptions. The 9th? “NG:UNREADCOUNT”. And why couldn’t we just make it the 17th field for GetSubscriptions and used a single call? Better yet, how about eliminating “NG:UNREAD” and simply using “NG:UNREADCOUNT” for the same purpose? Guess what, if it’s 0, there aren’t any unread items!
Now the fun really starts!
Spin the Wheel and Place Your Bets! Your Parameter could be the lucky winner!
They say a picture’s worth a thousand words, so we’ll let them do some of the talking here.
Remember our good friend GetSubscriptions? Take a look at its parameters, as per the online documentation:
Now that we’ve got a list of subscriptions, let’s pull in all the items for each feed. For that, we’d use GetNews. Check out its documented parameters:
Wait, what? Is it “token”, or “syncToken”? Make up your minds! With all this SOAP-based nonsense, I’m copy-and-paste’ing my brains out, and suddenly you throw a wrench into the clockwork!
Far from a unique incident, however, we can’t let GetSubscriptions and GetNews take all the blame… GetNews appears to be a bad influence, since it doesn’t play well with other calls either. Another GetNews highlight:
As you can see, our redundant friend GetSubscriptionCounts doesn’t quite match either:
Think you’ve seen enough now? Well, we’re not quite done yet… Just in case format blunders, missing documentation, and parameter SNAFUs weren’t your style, how about “undocumented features”3?
Presto! Watch Me Pull a Parameter out of My Hat!
Not to leave well enough alone, let’s make one more last jab at good old GetNews. I realized my parameter naming problem and corrected it, only to be met with a remarkably similar error telling me that I had an invalid value for the “stripMarkup” parameter, that it couldn’t be left blank.
The stripMarkup parameter? Am I the only one who missed that? Did you see that in any of the screenshots I just posted of GetNews parameters? Let’s look again at the extent of the documentation for this API call:
No, apparently it’s not just me. If you click on through to the SOAP Endpoint, NewsGator provides a helpful test mechanism for each of their API calls. Along with SOAP 1.1, 1.2 and REST mockup requests, they provide a simple HTML form that does an HTTP POST to the REST API and spits back the raw XML you’d get as a response.
Loading up this test form, I see where my error comes from:
Wait a minute! Where did all those extra fields come from? Those aren’t mentioned anywhere else! The only way you could find those is by looking at the raw XML of the WSDL file4. Whereas at least some of the other non-mentioned parameters we’ve seen have been slightly user-intuitive, some of these are absolutely meaningless to me. What the hell is the difference between “stripMarkup” and “useScrubber” anyway?
Try… Catch… Finally…
So that’s it. Everything I hate about the NewsGator API, all condensed down into one single (albeit lengthy) rant. It’s all laid out there on the line for anyone to see. Do what you will with it. With any luck, NewsGator will track down my post and (hopefully) fix some of these problems. Even if you just want to use these points as examples of what not to do when building your own web API, at least some good will have come from it all.
And to be perfectly clear, I want to make sure no one walks away from this thinking the API is a total loss. To be fair, there are a lot of good points about it. I think their sync token concept has some really strong points (even if it is confusing at first), and really strengthens their process of syncing so much data back and forth without totally bogging down everyone’s bandwidth in between.
I also don’t want to be thought of as making a personal attack here. As I recall, Gordon at NewsGator was their head API guy a while back, and he was always willing to jump in and help. The first few times I had trouble with the API, oh so long ago, he found my blog and did everything he could to help me through my problems, even though he knew nothing about PHP. He was a great guy, and for all we know this entire API was inherited in some bastardized form from a previous developer.
I also got a comment from Greg Reinacker on one of my other blog posts on the subject, and he too was very eager to help. It’s clear people aren’t the problem here (well, at least on the intentions and motivational sides of the equation), because NewsGator obviously has some amazing talent and passion working for them.
The one thing I can’t figure out through all of this, though, is how the hell NewsGator has managed to build so many products on top of this API. I’m one person and I can’t figure this out - how have they managed to coordinate an entire team of developers, geographically dispersed, and get them all on the same page with this, across so many different programming languages (.NET to Cocoa to Java) and platforms (Windows to OS X to Mobile)? It boggles the mind…
- For the record, I also attempted to use NuSOAP, as I’ve tried in the past. Even after renaming all the “SoapClient” class declarations to “NuSoapClient” to avoid conflicts with the PHP5 native implementation, there are several hurdles to overcome. Eventually, I gave up and went with the PHP-native version, expecting it to have better support than NuSOAP, which appears to have been abandoned for several years. [back]
- Which really makes sense, considering it’s coded in ASP… [back]
- The “politically correct” phrase for a “bug” these days… [back]
- Which I suspect is used to auto-generate these forms, as a matter of fact. [back]
Well, you’re right… someone from NewsGator would find this!
Let me introduce myself. I’m Nick Harris, the lead developer for NewsGator Inbox, Desktop, ScreenSaver and Toolbar.
All of my applications rely heavily on the NewsGator API – both SOAP and REST – not to mention NewsGator Online and various versions of NewsGator Enterprise Server.
I also spend a lot of time coordinating between Nick Bradbury (FeedDemon), Brent Simmons (NetNewsWire) and Kevin Cawley (NewsGator Go!) and our various platform developers and testers to make sure all of our applications know how to use our API for both NewsGator Online and NewsGator Enterprise.
Our API is always expanding to incorporate the new features for all of our client products… and the documentation for those changes doesn’t always keep up with our rate of innovation. I apologize for that.
I’m more than willing to help you out or find the person in-house that can answer your questions. You should have my email from this comment.
We’re also always open to comments – negative or positive – about our API and how we can improve it.
An email or a blog post… we’ll see it and take it all into consideration.
Please get in touch!
The reason that I don’t jump in and help anymore is that I’m no longer a NewsGator employee. I could write a long comment on why things are the way they are, but I’ll save that for my own blog.
The REST samples would be in the link marked “sample code” next to the link to the poorly formatted PDF.
http://www.newsgator.com/ngs/api/RestAPISamples.zip
It’s a Python wxWindows app, I had a Ruby app at one time as well. The problem with Ruby was that the wxWindows port for Ruby was plain awful. Also, I have a feeling that doing HTTP Digest in Ruby was a big problem (meaning you had to write your own). Digest in Python was not straightforward to implement, but doable.
NuSoap is typical of the dynamic language SOAP stacks - it solves the RPC/Encoded problem but stinks at Doc/Literal, and support for things like SOAP headers is weak. At one time, I had a blog post on the gyrations you had to do to get NuSoap to work with the NG SOAP API, and I know that Shelley Powers had some success with that as well, so it is doable, just not easily.
I think the biggest problems you mention here are documentation. There are some inconsistencies (as you point out), which I’d like to see fixed, but in any case with adequate documentation this all probably would have been a much smaller problem.
You’ll be happy to know that we have an updated REST API doc on our intranet, and Nick Harris (who commented above) and a few other folks are going through it in some detail to check for accuracy before it’s published on the web site. It should just be a day or two, and we’ll have something up.
Your post also prompted some internal chatter this morning about how best to present the API documentation, and keep it up to date. We’re considering setting up a wiki with all of the developer documentation, and allowing folks to actually come in and edit or annotate the docs if they find something that wasn’t clear…and at the same time, this will make it much easier for our API developers and testers to keep the docs up to date. Hopefully we can move on this quickly.
As for some of the oddities (i.e., two similar but different calls to do similar things) - there is typically a story behind all of these, usually stemming from the large set of client applications we have using the API’s and the huge existing deployments of these applications. We also have some calls that are designed to be faster or more efficient in terms of bandwidth and such. Not that this is a good excuse to have a confusing API…just explaining how it got there.
As for how we manage to build so many products using it (and have so many syndication services customers using it) - I think it boils down to this. Once you “get” the idea of how the API works in general (ex. sync tokens and their purpose, etc), it’s generally fairly easy (at a high level) to design your application to use them. Then it comes down to the actual implementation, and some of the quirks, and it sounds like that’s where you’re hitting most of your frustration.
But in any case…thanks for your candid post - feedback like this helps us make things better! If you follow up with Nick or myself, we can make sure to get the updated docs in your hands. And I’m going to make sure every person on our API team reads this post.
Wow, I have to say I’m a bit surprised at the response I’ve gotten - although considering the experiences I’ve had with NewsGator in the past, I suppose I shouldn’t be.
First off, thanks to all three of you for taking the time to comment and letting me know that I’m actually effecting change (scary thought…). Like I said, despite the gripes I may have about the API implementation, NG has some great people working for it (and, well, formerly working for it). I really can’t tell you how much this kind of response means to me.
Nick:
Thanks for so quickly stopping by and offering to help out. I meant to get back to you last night, but I’m afraid I just didn’t get around to it. I’m sure everyone knows about FeedDemon and NetNewsWire, but I particularly enjoy getting to know the people behind lesser-known products, so it’s nice to meet you.
I’m all too familiar with the ways in which documentation can so rapidly become dated, particularly during heavy development. I suspected this was actually the case behind the missing parameters, since I didn’t recall having seen them in my previous brief experiences with the API. Thanks for confirming my suspicion, and especially for the apology - it really means a lot when you get something like that from someone you’ve just brutally criticized.
Should I encounter any further problems that I can’t figure out in a timely manner, I’ll be sure to drop you an email. As Greg noted, the majority of my problems have been documentation related. Even if I’m still working off old documentation, at least now I’m familiar with some of the potential pitfalls I can encounter, so I hope to be able to work through everything else I need on my own. Again, thanks for the offer!
By the way, a quick way to disable NewsGator Desktop (like from the context menu of the notification icon) would be convenient for those times when you’re heavily involved in something. Basically a faster way to get to the ‘Disable / Enable’ button in Notifier Options. With FeedDemon and NetNewsWire, Desktop is a bit redundant (and, you know, has ads stuck in from time to time), but I love the overall functionality (quickly hammering through a list of feeds is really nice) and it’s a great freebie for those who may want more than the web interface, but aren’t ready to go up to a full app yet. Screensaver also gets bonus points for being utterly unique… I look forward to more cool tools in the future!
Gordon:
Hello again! Sorry to hear you’re not with NewsGator anymore, but thanks for stopping by anyway.
I checked out the REST examples, but I have to admit I’m not the best at reading Python (in fact, probably about as far as you can get from ‘best’), so unfortunately they weren’t of the most help. That and the real lack of documentation is what lead me to return to the SOAP API (since mixing and matching the two really seemed like redundant work from the basic setup perspective).
I agree that NuSoap (and the PHP5 Soap client as well) is lacking in some of the finer points of the process. I couldn’t find the post you mentioned on your blog (at least not the one linked in your comment), and there’s no search (the Google search didn’t find anything), but I’d be interested to read some more of your thoughts on the topic, if you’ve got a minute to track it down. I’ve also read Shelley’s experience with it, which did help with some of the basics, but unfortunately there was never any more detailed analysis of the process…
Greg:
You’re right, my problems were mostly centered around the lack of documentation. The inconsistencies are annoying, but are easy to spot once you realize they do crop up from time to time. Fixing those in current versions of each call would be a major pain, but I agree that it’d be nice to see them disappear in future versions.
Fresh documentation would be great (thanks again, Nick)! I’ll be eagerly awaiting the updates.
A wiki sounds like a good idea, as long as it doesn’t get out of control over time. Difficult to navigate and consume documentation can often times be more annoying than no documentation at all. I’m not sure what the best format for such a wiki would be, but public assistance in updating it and contributing insight / examples / recommendations would certainly be a welcome addition. I’ll be looking forward to seeing this in the near future as well.
Things like the intended use behind two similar calls and the intended performance benefits of different calls would also be great subject matter for the wiki (although a time-consuming process to compile). Understanding why a certain call was added in the first place and why one might want to use it over another call would certainly help anyone developing on top of the API create better applications (better both for them and for NG).
Thanks again to all three of you for the incredibly wonderful positive response I’ve gotten after thoroughly bashing several points of your API, it really is great to see. I’ll be watching for new documentation and try to follow up on the progress of the wiki idea in the future.
Thanks!
A new version of the REST API documentation is up:
http://www.newsgator.com/ngs/api/NewsGatorRESTAPI.pdf
It took a little longer then expected but I think you’ll agree that it’s much, much better.