BC API Direction - Clean discussion

[Liam Dilley]

With the other thread being a bit of Liam bashing and off target a bit I wanted to do a proper thread on this.

Summary:

API V3 implementation and some decisions on the use with module_data as well as other API such as the web app API and SKD are quite strange and looking at the scope will cause more requests and more load rather then less as well being quite cumbersome to use in real world situations.

The Good:

Firstly I just wanted to say module_data as a whole is bloody awesome and I am massively thankful for BC to offer this, this and liquid - OH YES! BC has needed this bad. The implementations and progress with the filters (WHERE) has been great.

The issues:

Web apps:

Web apps BC, you have mentioned will go to V3 but the current Implementations along with most things just seem really odd.
If you want to get a web apps items, compile them into a collection as it were to do something in a scope of all the items or even a large batch of them over the 500 limit there is the very odd method.

Firstly you can not really do this with the API methods and have to turn to the SDK... With this you have to a number of .fetch batches with a strange small limit of 100 (it would not fetch any more of this for me at a time) and then from this you have webappitems object that... well its very strange and you only get the core data of a web app item. If you want the description and custom fields you have to do another backbone fetch on top of that.

Trying to batch do that is doable but so verbose its crazy.

If you then need to get the categories as well you then have to make even more requests per item, pretty crazy stuff.

module_data:

You can use sub resource method for order with products but then you then have a limited set of order data coupled with the products and its only one order. I have tried to add more attributes for the order to such a module_data method based on the API discovery tool as reference and trying to expanding on it but these fail to render and not aligned with the documentation.

So to have product information you then also have less order data.

The other issue with this of course is its for one order, if you want to do a listing of orders and show the products you would have to do another module_data request per order in a loop which seems over the top.

The 500 limit in terms of loop and scoping as well unless you do the crazy fetching, but with no means to combine the sets... Not much you can do.

API V3:

Things become a bit worse here where you have to fetch an order then looping through making further API calls to fetch the products and then trying to combine them. The total number of requests you end up making is heaps more then if the products were just there as part of the order. It is also more hassle to collect the data and the process it considering your using javascript based technology.

Cases where to get a customer and associated orders or that and something else you will end up doing double the requests.

Let's say you have 10k products and to get all of them and do something with them in a scope of data you want you could end up having to do 30k requests to get them. In a scope of 7k products I have had to do 14k of requests over a batch process with wait times and small process of 50 lots! which took ages through javascript just to get the data out. This took ages, ran through few other develoeprs and .net developer as well to optimise and... well not much.. With browser, javascript and the BC limits... Crazy stuff.

IF we take this when you can use the API through 3rd party so requests coming from outside and through oAuth and people having to do more requests then just how SOAP works... This seems worse to me not better.

This is just one example of it.. getting products and then having to get their prices in separate calls seems really odd.

In all the APi the requests and code you have to form to work in large data scopes seems just overly difficult and as I mentioned overly verbose... If you have the scope of 10 items, 1 item, 20 items things work and it is ok, but when you do a larger scope, especially over 500 items everything seems... Well bad.. Sorry guys but it does.... Things seemed to be formed in the scope of using it for 1 item, or a few item or a bare minimum set of data.

All the samples, docs etc out there are all this - Small stuff and not practical.

If you guys can form examples of things with large data, open platform examples fetching batches of data to form into a single array of data and deferred methodology so you can fire completion stuff with the data has been fetched that is quick... Ok I will pay attention... But that leads to the other issues...

API Performance:

I am also concerned with this, I make a request for 200 items and not even all the data and watch it hang for 30/40/50 seconds. I am using other system API scripting and requisition JSON based data of more then 500 items and its heaps easier and heaps faster.

I have sent heaps of data on this, screenshots, examples and spoken to other developers and partners, shown them, got them to try and make use cases and....

- People agree

- Its obvious in some cases

- Some developers have failed to even get the data on a large scale let alone useable to do what they want...

Why has open platform not delivered like better scoping apps? is a question to raise for example. Even from high end developers? Our new .NET guy has gone over things with me and he has been in total agreement with fresh eyes to all the BC API's as he has not used it before. SOAP is a pain but in terms of scope of getting the data his opinion is that is bad but better then the new stuff.

This is a really odd thing to have been said, but kind of true.

You guys are doing a great job but this stuff really needs to be flagged, things need to be reviewed and sorted and implemented right ASAP... I am sure you can do it but please, listen to me!

[Liam Dilley]

Just an updated example in terms of the web app API.

If you was to do a sync or just getting all web app information for what ever reason...

- Total web app items: 888

- To get web app items outside of basic name, date created etc another request is required

- To get categories for an item another API request is required

888 * 3 = 2664 API requests.

- BC limit of 500 requests is different on web apps as it seems to max out at 100 items

- Scripting Batch Processing to run effectively has limit of 40/50 at any given time with a safe wait time that does not cause issues of 8 seconds between batches.
- Some process requests for one web app item to get description and custom fields for no reason spike to up to 30 seconds.

Average process time to fetch and complete the 888 items - 15 minutes.

Notes:

- Yep sometimes it can be done in 5-10 minutes but with the spikes it averages around 15 minutes

- SDK requests sill slower then direct API calls but offers skip and limit features that seem to not work correctly in direct API

- You have to consider the requests and processing multiple items and ajax functionality and what you have coded in terms of the .fetch in backbone js. You need to consider people will need to run deferred processing to know when everything has been completed, this becomes increasingly more messy when you need to do multiple requests just to get the data as currently.

- Lack of ability to combine collection in liquid means that even if you had module_data you can not have a complete list at any time.

I know with categories - For what ever the reasons are for them not to be in list views and have separate API requests or lack of in other cases... Something here is annoying your end... OK, but with a lot of the API request splits You are doubling and in some cases tripple'ing the number of requests needed to get data.

[Liam Dilley]

Comments / Thoughts

- JSON files for every module in settings that can be turned on

- JSON files with limited load requests per day which are more on higher plans and not a feature on the very lower plans. Option in partner portal to increase requests and pay for more request tokens.

- Reduce the requests needed for V3 API's combining more information into the initial requests

- Force requests to require field specification data. Most people only need some fields at any given time and if you do need more you request those. Most requests should be a lot less and in tests API requests with field specifications are a lot faster.

- WHERE use and good WHERE options will allow reduced result set of data

- Limit and offset use of course but good methods in javascript with ajax consideration and waiting for completion. With less inside multiple API requests this will be better anyway.

I think if you open up OAUTH and considering most 3rd party applications will be syncing data with other systems and having to fire off all these what I say are undeeded requests... If people get on to this and BC increases use again as we all hope your just going to be hammered.
And considering some of the talk out of BC already, if you guys just try to fix the front facing problem and not the actual problems all your going to do is whack in lots of API request limits, make people pay etc... That is going to be silly in my view and upset people.

Sure, introduce some extra charge for more requests for more enterprise/large scale sites, but fix the root issues where you need to make heaps more requests then you actually need to.

[bcalpha]

I think you are spot on here, i would like it if the API's where not so verbose and more "general".

Instead I'd like to added filters just as with module_data, this would immensely shorten the amount of requests and time waste on development.

And yeah, the performance spikes are noticeable...

...one of our partners that works a lot with Amazon AWS says that it's quite a common issue - but it shouldn't be the case since BC has their own datacenters that is based on AWS?

[Liam Dilley]

You know BC moved onto AWS a while ago now, Big change and transition. BC are not on their dedicated DC's any more. All the big spam issues were because they were then on AWS IP and had to do heaps of work to stop all that and cut out the new noise.

[bcalpha]

Yes, i know, but i got the impression that the transition was still on their dedicated datacenter just that it was a different datacenter owned and operated by Amazon.

[Liam Dilley]

You can read on what 2ndWatch did for it here..

http://2ndwatch.com/resources/customers/adobe-business-catalyst/

[bcalpha]

I read through it and i guess it makes a lot of sense from a lot of perspectives and they don't really have many other alternatives to go on, perhaps OpenStack providers like Rackspace - but then again, the price is double that of amazon for the same performance (Based on serverbear.com‌ benchmarks)

But i wonder why they didn't go with Microsoft Azure? Would make more sense since BC is built on .net.

The enormous scope involved here should not be left to speculations though - I just hope they can fix the performance spikes.

[Liam Dilley]

Probably the cost as you scale up the rates are more on Azure

[bcalpha]

Maybe and probably also about meeting deadlines, anyway, this is quite Off Topic so lets leave it here.