[Bioc-devel] BrowserViz and sub-protocols

Kitware seems to have had good experiences with WAMP, which is a good
sign, given the complexity of their typical visualization projects
(stemming from old HPC projects in fluid dynamics, astrophysics, realtime
anatomical reconstruction, etc). Thus WAMP seems like a great way to
leverage the experiences of an open source focused, medium sized shop known
for doing highly technical work with some similar aims to Paul's pkg.

Avro's IDL becomes onerous for large projects rather fast.  The lack of
inheritance can be excruciating when it is pushed beyond its typical scope.
Best not to poke that sleeping dog ;-)

--t

On Apr 6, 2015, at 9:35 AM, Michael Lawrence <lawrence.michael at gene.com>

wrote:

Sounds like a good plan. It does seem like there is a trend away from

type

formalism in application development. Everything is essentially JSON.

Have

you checked out WAMP? It seems very similar to your solution; it might

pay

off to stick with standards, even if it takes more up-front investment.
It's JSON-based, so it doesn't solve the high-level type problem. Very

few

serialization formats do. Avro has a nice IDL, but that's out of the

hadoop

world and would be foreign to the web. There is someting to be said for
simplicity, but the incurred technical debt can become burdensome.

On Mon, Apr 6, 2015 at 8:39 AM, Paul Shannon <pshannon at fredhutch.org>

wrote:

Hi Michael,

Thanks for the link to the subprotocol negotiaion discussion in "High
Performance Browser Networking".  There's a number of things in that
discussion of which I was not aware.  Very helpful.

There may be real merit -- that is, a good trade-off between some added
complexity and extra clarity -- to adding a mandatory content-type

subfield

to the payload or to the message.  Or adding full sub-protocol

negotiation

as the reference you supplied discusses.

For the first release of BrowserViz (and its demo subclass,
BrowserVizDemo, along with RCyjs) I would like to keep things as simple

as

possible.  By which I mean:  provide (as we do now) the mechanism by

which

content-type can be specified, without promulgating the policy that

every

message must have that payload subfield.

In practice I have found it quick and easy in both Javascript and R to
examine the incoming 4-field message (cmd, status, callback, payload):

1) Is the cmd field something I can respond to?  (Do I have a handler
registered for that cmd?)
2) Does the status indicate trouble? A request?
3) If 1 & 2 suggest proceeding, then  names(msg$payload) can help me
figure out if the incoming data is structured as I expect.

One could reasonably object that this is all "programming by (ad hoc)
agreement", rather than by transparent, self-describing, clearly
negotiatied data and exchange standards.

In order to arrive at such standards, to figure out how formal we should
go, let's experiment for a release cycle.  This could be accomplished
without any changes to BrowserViz by using the status field
(status="summarizedExperimentRequest"), or by always using two fields in
every message payload (payload$contentType, payload$content).  If a few
people collaborated on (let's say) a SummarizedExperiment browser viz
webapp, those people could agree on how to do this, try it out, perhaps
discover a few sub-varieties of content-type are actually needed.  Then,
when things have settled down, we could explore adding some formality to
the process.

How does that strike you?

Thanks for the discussion!

- Paul



On Apr 3, 2015, at 3:11 PM, Michael Lawrence <lawrence.michael at gene.com

wrote:

The high-level type issue is sort of discussed here:

http://chimera.labs.oreilly.com/books/1230000000545/ch17.html#_subprotocol_negotiation

What about extending your protocol so that the payload consists of two

fields:

content-type and content, where the content-type adheres to the media

type specification? This is analogous to how every S4 object has a class
attribute.

Thanks for the discussion!

On Fri, Apr 3, 2015 at 2:57 PM, Paul Shannon <

paul.thurmond.shannon at gmail.com> wrote:

Hi Michael,

Thanks for the clarification.   You make a good point about caching:

nobody wants to have to reinvent and re-engineer that!   If too many
features like that become important, then the simplicity of websockets

will

have an attendant cost.

With regard to more formalism around the payload:  I can imagine that

there will be circumstances in which that is essential.  A
SummarizedExperiment browser would be very useful, and would clearly
benefit from a standard payload structure.

I figure, however, that that's above my pay-grade ;).   I leave that up

to people like you.  And I stand ready to add any features such

formalized

structures might require.  I'd like to think (I may be naive) that the
architects of SummarizedExperiment and heavy users of it could devise

and

negotiate some standard JSON representation of the class, translators to
and from, which all could then be used by BrowserViz without BrowserViz
needing to know it's there.

The "binary JSON" format may be useful in some circumstances:

http://bjson.org

- Paul

On Apr 3, 2015, at 2:43 PM, Michael Lawrence <

lawrence.michael at gene.com>

wrote:

On Fri, Apr 3, 2015 at 2:00 PM, Paul Shannon <

paul.thurmond.shannon at gmail.com> wrote:

Hi Michael,

Great to get your response, comments and questions.  Answers attempted

below.

I think our overriding difference lies in our contrasting experience

of

complexity.  I have come to see websockets as minimal, simple, fast and
flexible, whereas you see them quite differently.  I would like to
understand your views on this; I could be overlooking somce important

and

maybe costly complicating features, perhaps because of my fondness for
other simplifying features.

I was just referring to potential complexity. If one attempted to

reimplement the useful features of HTTP (like caching), complexity

would be

introduced into application code, while you really want that complexity

in

the protocol implementation. HTTP seems most appropriate for when the
server is acting as the data model. Outside of that, I see the benefits

of

web sockets.

I think you missed my question about RPC. Also, any thoughts as to

more

formalism around the payload? We obviously have complex data structures,
and it would be nice to communicate semantics somehow to the web

browser.

For example, could there be some convention for representing a
SummarizedExperiment? Could a payload contain the equivalent of a media
type that an R/JS library could understand to marshal objects? Could

there

be some way to query for the types of payload a command supports?

I've seen stuff like WAMP, but they seem to lack the ability to

declare

high-level types. Maybe that's just out of style?

More below...

I look forward to hearing back from you.

- Paul

On Apr 3, 2015, at 1:45 PM, Michael Lawrence <

lawrence.michael at gene.com>

wrote:

Thanks to Val's excellent newsletter, I've had my first glance at
BrowserViz. I'm glad to see something that is more flexible and

low-level

than e.g. shiny.

I'm curious about the motivation behind web sockets. I guess any
application with an R-driven web UI actually has two UIs: the R

console and

the browser. But what if the R session is headless, or if there is no

need

for commands in R to affect the browser? Then the web socket layer

brings

mostly unneeded complexity.

I see websockets (like TCP sockets) as musch simpler than HTTP.  No

headers,

no explicit server and explicit client (once the connection is open).
Could you explain the complexity you see?

An interesting comparison to BrowserViz is not
shiny but OpenCPU. It's purely HTTP-based and still manages to

maintain

state (not sure how efficiently). I guess one advantage of web

sockets

is

that one can program imperatively instead of declaratively on the

server,

i.e., the server can send a command to show a popup in response to

some

event, instead of returning a "declaration" that the popup should be

shown.

Exactly!

So essentially web sockets are more natural for implementing

server-side

controllers (think MVC), instead of just the data model, but man,

it's

a

shame to lose the features of HTTP.

I'll confess:  I set out to -shed- the features of HTTP.  Isn't it a

protocol designed for serving up web pages on demand?  Not for fast,
lighweight peer-to-peer communications?

Ultimately, I think we want web apps that are easy to develop and

maintain,

and run equally well from either a useR's session or a remote client
communicating to a dedicated, headless server. Is the generality of
websockets worth the complexity?

As an aside, it would seem relatively straight-forward to implement a
simple bi-directional RPC mechanism between R and JS using the

standard

protocol (i.e., hide details like the callback). Does that sound

reasonable?

I was also a bit surprised about the need to copy/paste the JS

boilerplate.

Certainly there must be javascript frameworks with a more elegant

solution

to extensibility.

You are correct on this, and I only need to correct the vignette,

written many weeks ago.

I now provide "BrowserViz.jx", a simple Javascript module.  It is used

by both

BrowserVizDemo and RCyjs.  Sorry to have sent out an obsolete

vignette.

    [[alternative HTML version deleted]]

   [[alternative HTML version deleted]]