JSON - A lot of work

[Liam Dilley]

Hey guys,

This is continuing to frustate me and I really wish JSON everywhere was a feature of BC to get excited about but unfortunatly its not useable. I really would like to see some work on migration along with the final liquid stuff to be a core focus to get finished and right please.

The premise of this is awesome, but it has no practical use:

BC.Next - Access All BC Data: JSON Output Everywhere, More REST APIs

Dynamic .json pages

Great idea this one and I have gone to use it but I have only managed to have a viable use.

- The first issue is that it is only reading modules on a page and forming a json output for them. If you run a collection or some liquid code to form data - it does not read those. I was hoping to form custom data output and then have a JSON version of that to do some ajax etc - Not the case.

- The next issue is the 500 limit. JSON data should be all the data of an items but BC has the limit of 500. The concept of paginated Data which you cant know the total number or get all the json in one set is pretty unhead of. So you can not really use this to read a module on a page in JSON. So this is also pretty limiting and not viable for a lot of applications.

module_json

Another great concept but really has no viable use in the real world unfortunately.

The ONLY use case is really if a json file is FTP'ed onto the site or an APP (Which can only run when loaded) Creates it. This is not a lot.

- The module_json can not consume BC's own JSON page output option. module_json can only handle .json files

-  module_json can not read external json data. This is the major, the engineers have given some very loose reasons why they not done this but this is a must. Pretty much most use cases people will look to use this for will be to consume 3rd party data.

Vimeo - You want to get a video out and get the pictures from it :API Playground on Vimeo Developer API, You may want to get your wall feed from facebook, you may want to pull out the videos from a playlist.... And more.

Yes - Having the step of accessing 3rd party json data via authentication I would not expect this to have as a feature any time soon (although a nice to have) but to consume the open available data is kind of a must.

- This is a combo one. Since the module can only read .json files it can not consume BC or 3rd party json files that are not .json so a double whammy limitation really.

- JSONP Since 3rd party can often be JSONP BC does not have the ability consume this type of json output.

Summary

These are the core issues and I think other options need to be considered as well.

If you really do have JSON everywhere then the following needs to happen:

1. Loose authentication:

If you want to get the full JSOn output of a web app/ events, news... And so on through a 3rd party / your own javascript then you need to authenticate.

Basic use would be something like eventbrite - http://developer.eventbrite.com/doc/authentication/

All you need is an interface in the admin to say what URL's will be needing the JSON data and generate an authentication token. The script makes the request with the access_token. You can look to implement this when you do the oAUTH stuff you are going to do guys.

Basic request but stops the servers being hammered with requests. You probably want to always support worldsecuresystems sub domain of a site and development domain. OR just say the request always has to be made to the worldsecuresystems domain version and you have to authenticate the live domain or or others.

2. JSON for everything - Properly

As mentioned with point 1 - Everything from web apps, events, news, gallery has to return a JSON set of all data on request. Authentication as above to cover proper valid requests.

3. module_json reading 3rd party and JSONP and non .json JSON

Pretty straight forward to remove its big limitations - It needs to read BC own json pages, it needs to read 3rd party JSON.

You can have a white listing admin use to validate this.

So in the admin you have a list of URL's that the module will read from JSON. The developer/partner has to put these in for validation, OR you guys at least support a number of sites out the box. If Support needs to approve a URL - ok.

But the module_json should check these valid URLS and let you make requests to them to consume 3rd party data.

4. JSON paramater in URLS

If we have JSON requests on modules you can leave this as is, otherwise a module on a page when JSON parameter passed has to return all items, not the limit of 500.

If you have some liquid code to form a collection of data etc - This needs to be part of the JSON data so you an

This would be pretty much cover everything you would be expecting this stuff to do.

[Robert_K__Bell]

Re module_json and external data - does it really make sense to block the page rendering and delivery on an external request that might timeout? The browser won't even get the first byte of the page until that external resource request is fully downloaded and parsed.

[__Alex__]

Agreed! This is THE TO-DO-LIST. vote up folks!

[bcalpha]

I would like to able to parse JSON from a Webapps item custom-field.

It would make it a TON easier to make Apps, since we could get a dataset of JSON easily via the REST API's and then easily render it on the template with liquid!

Right now we have to do that with split and it's dirt, ugly and prone for errors - Plus we have to store it with an extra space between each value to split since this happens:

{% assign testString = "#;Tadaa#;#;" | split: "#; %}

Gives: A single string - "Tadaa" where value 1 and 3 can't be tested against since testString[0] becomes value "Tadaa".

{% assign testString = " #;Tadaa#; #;" | split: "#; %}

Gives: An array of three values that can be used as testString[0] becomes " " and be tested.

Being able to parse JSON from a custom-fields would make these cases much easier and powerful.

[Liam Dilley]

Thomas,

When the APi Web apps comes out, just as the other API it will work the same and you can have that JSON data. So nothing needs to change there.

You can read any page as JSON already so a web app list on pages with custom fields will return as JSON as well.

In terms of Liquid, what we have covered is if you had a page with module_Data doing some WHERE stuff to get a set of data and then reading it with liquid which, when web apps come will be again readable in liquid.

If you make a custom field which is a string and you do a comma separator value data and you want it to be an array, you do as any other coding platform - Create the array, that is what Split would be doing.

So not sure what your asking?

[bcalpha]

I know that you can read and output custom-fields as JSON, what i mean is nested JSON - Like you have a multi-line field that contains JSON and you can parse that as liquid as well, just like with module_json but for webapp item fields.

Like: {module_json json="{{this.myCustomField}}" template="/the-template-to-use"}

Since we now have a limit of 20 custom-fields per webapp item, this would enable more data into a single field.

Besides, this is related to this:

With that, we would be able to construct our own JSON data structures.

[bcalpha]

And split is NOT working like in other languages, because in other languages you get the empty arrays too - on BC you don't.

[Liam Dilley]

Yes you can do empty arrays, do not know why you would want to. You do not have limit of 20 custom fields in a normal web app use case, again this is only the proposed limit for the API. I am not sure if that spans to module_data as well. Nothing been confirmed but normal web app use will be fine.

In terms of normal webapp output and liquid what you seem to want to achieve you can do right now.

[bcalpha]

How would i do empty arrays? Is there another thing that's not documented?

[Liam Dilley]

First question, like I would ask with any language is why would you want one?

Declaration first for memory in some languages and others advising trying to avoid empty arrays. In terms of BC I can not think of a use case on why you would want it?

[Robert_K__Bell]

There's a thread specifically about empty / sparse arrays: Re: Idea: Handling Empty Arrays

[bcalpha]

Thanks, good discussion! So another vote for split_parse!

Liam: Sometimes you need more values but don't want/need more fields - doing a split join in JS and split_parse in liquid solves that.

[jnowland]

module_json

Another great concept but really has no viable use in the real world unfortunately.

The ONLY use case is really if a json file is FTP'ed onto the site or an APP (Which can only run when loaded) Creates it. This is not a lot.

"No viable use in the real world"…

We use this feature on every site for the past two years (in the real world).

[Liam Dilley]

Good for you.

But I stand by my comments.

A manual process is not viable for me.

Like I said, only a 3rd party connecting to FTP dropping and updating a file OR via API something creating the JSON file dynamically is a real JSON use in this case.

A file manually made and updated or updated via manual process in an app is not really viable.

[jnowland]

I've always viewed module_json as a configuration rather than a "dynamic" thing.

As Rob mentioned there would be major performance issues trying to handle this.

While interacting (and depending on) third party APIs in almost every case it's best that it's done via the front-end i.e javascript.

There is no way a server can do this in an asynchronous fashion which is why this approach is not limited to just BC but general "best practice" regardless of the CMS.

As for the "Vimeo" example if you are just selecting one video in the admin it would likely be best to make an app that pulls in the api via javascript that saves it once into multiple BC fields i.e Video URL, Video Thumb etc…

---

It kind of feels like the answer to a few of these problems is more "webhooks" + module_json so you can avoid the FTP workflow.

[Liam Dilley]

I am glad discussions with BC gone differently and correctly. It is just resources and priorities right now and of course a lot of upvotes so lots of others agreeing.

[mbc9]

Personally, I think that while it might be the only thing BC will consider, adding more 'features' to Liquid smacks of adding more hacks to an already ugly solution.

What BC perhaps needs in to implement NodeJS as an alternative to Liquid and support for consuming REST apis - both BC and external - at the server.

[Liam Dilley]

I think your missing the boat on a lot of stuff there.

Node.js and Liquid are two very different things.

Liquid is a templating language and modules is a pre existing BC feature.

We already have REST API's - Events | Business Catalyst Support

While not complete they are there.

You can also consume any of the REST endpoints using module_data and liquid already.

In terms of the backend you already have open platform.

So while Node.js has come a long way and is great.. Your asking BC to replace something that will be harder to use then liquid that will do the same thing as what is already there and spend the time investing in all that for little to no difference other then more support.

Does not really make any sense.

node.js space as part of every Platinum plan to consume the REST & SOAP API for small solutions rather then having to have an external host at cost where you can do some scheduling and handle requests from purchases and contacts - Then that could be useful for developers.

But BC could also just expand on open platform further to offer similar features so apps can run scheduled tasks as well as handle incoming site requests rather then just being something that runs when you run the app.

[mbc9]

I think you may have misinterpreted 'alternative' for 'replacement'.

Liquid is an extremely clunky 'language' with more bugs and limitations than you can point a stick at, but obviously has an existing install base that can't be dismissed.

However, to not consider progressing because of that, effectively consigns BC to oblivion, as the rest of the world moves on.

In the same way that Wordpress is working on implementing a NodeJS alternative to PHP, or MS implemented MVC alternative to WebForms ASP.Net, there is no reason that BC could not implement a NodeJS alternative to co-exist with the Liquid install-base, and allow BC to have a future with modern technology.

[Robert_K__Bell]

One of the primary points of Liquid Markup is "You want to allow your users to edit the appearance of your application but don't want them to run insecure code on   your server."

The pace of new features is already slow enough without Adobe having to deal with all the issues that running   untrusted code would expose.

Also, both of the examples you give are code-only; they're not CMS services.

[Stephen_Blignault1]

I agree with Robert,

mcbc9, you're talking about replacing PHP and/or ASP with NodeJS in your examples. But this is not an apples with apples comparison. BC is not PHP or ASP or Ruby etc.

As much as I personally would love NodeJS, as I'm a developer at heart, this just isn't the BC model.

BC needs to flesh out Liquid, which is a "standard" pioneered by Shopify for non scripting based CMS's.

[mbc9]

Again, those aims and technologies don't need to be mutually exclusive - I am NOT talking of REPLACING in either this case or the examples given, but providing an ALTERNATIVE.

I'm not sure I see the significance of the 'CMS' to the argument - it's simply a set of APIs and UIs. WordPress is a CMS, and any number of CMS exist with developer environments supporting.

Restricting the ability of the developer as a means to control the CMS admin, is self defeating for the future of the platform.

[Liam Dilley]

And as in my other post. Having YET ANOHER thing to both support, maintain, have people doing differently... And all that - It is a bad idea.

[Liam Dilley]

Outside the bugs BC should be fixing for sure I actually find Liquid to be one of the better template languages out there.

I do find peoples comments as the opposite and talking to them and their commented limitations nearly every time relate to lack of knowledge of the language and the features. I get a lot of "Does not do this" but it does.

What your suggesting is not really a good idea in my view. It needs to be either or. If you try and introduce a platform with too many methods for the same thing you end up with a mess. Drupal while not in terms of language IS A MESS in a lot of ways for the same reason. Other solutions are in the same boat.

Even BC now with tags and liquid solution is touching that same problem and people like me have already suggested to BC that new sites etc everything is all Liquid based values, the default sites, BC templates, new sites and any new stuff is only liquid output and tags become very much by the way side and no legacy support made on them. That needs to be done for the same reasons as discussed here.

I can tell you use node.js and that is what you know well, I love it for a lot of reasons too - but for front end BC it's not the platform or scope for that to exist here.
As I mentioned, as a space to consume API as a step to not have to go full on external $$$ service just to consume some basic orders and apply some updates based on them - Then there is discussion there, but as I said already.. A discussion on this could lead to just updates and features to Open Platform to achieve the same things people want there.

Thought and discussion, always important but it is important to get a good big scope.

[jnowland]

BC primary positioning is not having to write backend code. Node JS is not only an entire different language and would be a huge time consuming task.

The community would be much better getting more API's opened/improved so we become self sufficient in writing BC Apps for the admin console.