multimedia authoring wars

| 4 Comments

In his musings on his PhD work, Baldur said something that made me laugh, since it's been an area of conflict/discussion/debate in our department for some time. Our "multimedia" courses are split into two areas--web development (where I teach) and multimedia development (where I don't). The latter is based solidly on Macromedia Director. My frustrations with that revolve around the problems inherent in basing an academic curriculum on a vendor-specific platform. So when I read Baldur's comment:

Most of my colleages here come from the multimedia industry where you use a single monolithic tool to produce a single monolithic file format which can only be used on the exact platforms you feel like implementing and testing on.

Well...it resonated.

I forwarded it to Andy Phelps, my favorite "multimedia" colleague. He can program circles around most people I know (and know of) in everything from Javascript to Lingo to VRML to Java. He's developing our new and very cool game development curriculum (follow the games edu link on his site for more info), and is always willing to play in a debate about proprietary technologies.

He doesn't have a blog (yet...we're working on that), so I asked him for his permission to repost these comments, which I thought were worth sharing. (Baldur's original comments are in italics...)

Andy Responds: (first, I think it's particularly neat-o)

And since my bouts with insanity don't extend into that area, I've been doing a lot of preparatory research work. My idea is to avoid the historical trend in interactive media and hypertext of sticking with proprietry unified approaches.

Ahhh. unfortunately Andy's bouts of lunacy tend in the direction of "I'd like to access that piece of hardware in your machine that draws graphics". There is no standard for doing that. There is OpenGL, but its not very "open" its a spec that is produced by a conglomeration of 4 major video card manufacturers since SGI is in the toilet, and these days any one of them feels obliged to make its own 'extensions' to the spec - which happily seem to void any other hardware but their own. The spec itself is so old as to qualify as a historical landmark of the computing age, having been updated for the last time in 1997. Thus proprietary giants like Microsoft and Sony produce standards like DirectX and Playstation2 (respectively). The proprietary, unified approach is all that exists to effectively combine music, sound, and advanced graphics capabilities using modern hardware and gleaning real-time performance. Yes, there is SMIL, yes there is X3D. SMIL has not gotten a quality foothold because there is not yet a truly capable authoring packages, nor is it capable of accessing hardware accelerated features directly, even for movie display (you need the Quicktime layer or WMP to find the DVD Codec garbage in your machine) - X3D is an absolute dismal failure, still stuck in the era of XML = DTD and completely ignoring the changes to the rendering pipeline that fundamentally altered graphics research in 1999, and again in 2001, namely per vertex and then per pixel hardware accessibility.

Most of my colleages here come from the multimedia industry where you use a single monolithic tool to produce a single monolithic file format which can only be used on the exact platforms you feel like implementing and testing on.

Classic examples are Macromedia's Director, Hypercard, and to a lesser extent Flash. The problem with the .swf format (published by Flash) is that it is not really as open a standard as you'd think. The published specification tends to lag considerably behind what is actually implemented in the Flash Player and Macromedia's authoring tool.

All very true. To view 'Multimedia' in the whole as defined by the large monolithic authoring environment is bordering on certifiable. However, at the same time, there does *not* exists a compelling cross-platform non-proprietary method of delivery. The browser? please. Browsers have *distinct* problems with one rather fundamental aspect of multimedia - the timing loop. Flash / Director / Hypercard / etc do one thing and one thing very well - provide a space to display media / graphics / etc along the axis of time. It is possible to effectively synch media to display at the same time, or on the same rate across a range of machines. SMIL is bordering on capable in this regard, and is the first standard-compliant way of doing so. MPEG4+ was supposedly going to give us this capability, and boy don't we all wish it really had. But it didn't. The 3D support is laughable, using good ol' VRML from 1996. (note that VRML still relies on a nice proprietary plug in none of which are completely standards compliant). The synching is still off in several spots, and authoring is an absolute
nightmare.

Javascript timing is downright laughable. In my own tests the setTimeout and such methods seem to be the worst implemented pieces of garbage across the standard set of browsers ever. You can not create a quality timing loop that is capable of driving even quality 2D animation.

Scripting / page display is great for the asynchronous display of information. It can even be 'real-time' in the sense that you want to occasionally push content to the client within a reasonable amount of time. If you study animation you begin to look at it as a very poor medium of display however. Should your website depend on animation? absolutely not. But 'multimedia' - broadly defined - does take into account timing, animation, synchronous display, etc. Unfortunately there is no standard for these things. There is only the proprietary gorilla.

For some reason people feel the need to use the gorilla for things it was not intended - like 'Let's make an interface out of Flash'. That's just not using the brain you were born with. There are lots of tools to build interfaces, that *are* standards compliant, that work everywhere, and with a little effort can look just as good (if not better - see my excursions into Javascript land). Markup and scripting makes great interfaces. Unfortunately it makes piss-poor animation. What gets you into trouble is when people want an 'animated interface' (QUESTION: do you really *need* an animated interface?) because there is not yet a solid way of doing that. There are basic animation principles like 'bounce and stretch' (or 'bounce and squish' depending on your point of view) that are almost impossible to recreate on the web. Sound support is pathetic.

Opposed to this approach you have a way of doing things which is a combination of markup and scripting. This approach has resulted in the only really successful genre of interactive media, namely the web

Now we're going to go an disagree in a rather big and powerful way. There is a history of computer games stretching back to the 1960's that I think disproves this statement - many of which are 'networked' and not on what I would guess you mean by 'the web' - most of these have used the mother of all proprietary gorillas - the custom built C++ playback engine - for several of the reasons above.

I would caution that the web is a great tool, but is not the sum of multimedia. The proprietary crud is important - but not the sum of multimedia. It is interesting that one side is obsessed (finally) with standards, and on the other side there realy is no standard, other than the best-selling toolset. To my great chagrin, none of the stuff you generally see is fully standards compliant, however. Don't be confused by idiot's running around screaming "Multimedia == FlashMX" - anyone who studies anything even tangentially related to media theory knows this isn't true - however the proprietary engine has a long a glorious history for a reason - namely no single entity has ever even attempted a standard in 'multimedia' - only the little subset that trickles into our limited imagination that's confined in a browser.

Caveat: the one exciting standard is a standard to define custom things: XML. Someday, perhaps the line will blur between proprietary and non when everything is (FINALLY) reduced to basic data and separated from display. Display is what everyone is arguing about. Lets argue about data instead for a little while.


4 Comments

Please take these comments with a grain of salt, I've just returned from the pub y'see

::::There is a history of computer games stretching back to the 1960's that I think disproves this statement - many of which are 'networked' and not on what I would guess you mean by 'the web' - most of these have used the mother of all proprietary gorillas - the custom built C++ playback engine - for several of the reasons above.:::

True, the computer games industry is the only commercially successful multimedia industry so far.

But there is a reason why I'd put it in second place after the web.

Namely, it's hell.

It's hell to work in (it's standard practice to fire everybody of a project upon completion, job security is nonexistant).

It's hell on creativity (90% of the projects started don't get finished, go bancrupt before they are completed).

It's hell on motivation (most of the work is repetitive, you write a 3d engine in one job, that one tanks, and you then have to do the exact same thing for the next job).

It's hell on interest (most of the work published is crap).

Some of these problems apply to the web (html + cheap, opensource scripting engines) as well, but to a much lesser degree. Mainly because the web is cheaper, more standard, and allows for much more use and reuse of free tools.

It is not the sum of multimedia, actually it is a horrifyingly small fraction of multimedia.

But the rest have doomed themselves to irrelevance. In a few years, most of our current games will only be playable in specialised emulators (read: MAME) while I'd say that we'd have a good chance of having native support for most of today's XML standards.

From a long-term cultural perspective the proprietry crud is more damaging than anything else.

Less than 3 per cent of the books published in the 20th century are in print. With proprietry crud, today's multimedia has even less of a chance.

But I agree on the rest of the points.

I think this is a chicken and egg problem. We don't have these multimedia tools because for the most part most lecturers and researchers have taken the industry's word that they need to use proprietry crap.

Universities as a whole have huge developmental and training resources. If we don't have these tools, based on open standards, then it is our own fault for not asking for them.

A finnish university hammers out a java-based SMIL/XFORMS player as a small scale research project.

http://www.x-smiles.org/

It is our own fault for not pushing this and other standards-based authoring projects further.

All, IMHO, of course.

More later, once I have recovered from the University's Christmas festivities ;-)

Ahhhh.

Interesting debate, although we are probably largely in agreement. Interesting Liz chose to title this 'multimedia authoring wars'.

I apologize for not having my own blog happening just yet, I am (once again) a lil behind the times. Sad. This is what teaching can do to people when we're not careful...

Anyway, You're perfectly correct in pointing out that the games industry is "hell" - I would agree with your assessment. Although, at the same time, it has produced some absolutely wonderful things - but 90% of it is drivel. (The same could be said of Geocities? but at least there anyone who wants to can write what they feel and not need a publisher and roughly $5M)

I liked x-smiles. CMU wrote Alice as a system for doing 3D for a while, and that was interesting, based on Python.

I am currently at a point where I have to say to myself 'OK - this isn't going to be standards compliant in terms of drawing (especially in 3D), so lets make the part that draws non-standard but then make the way we store data and manipulate data standards-compliant. Keep XML and the like in mind so that I can store and retrieve data in a powerful way, and only break down when I have to put it on a screen"

It's sad, really. What it really is (it seems to me) is that the standards stop at that point. There are standards for what functions are implemented in the langauge, what tags are available and what they do, how that is supposed to work, how that is supposed to operate etc. But no entity is willing to take it all the way to the metal and say 'a standards compliant browser will have a tool bar of width X - a standards compliant browser will draw its first line of text N pixels from the top edge of the window - text will be N pixels high if it is defined as Y point size' - we're getting there (slowly) with the DOM / CSS etc. - but there are those last things like 'well, how it actually draws images is left up to the browser' or 'well, the width of the toolbar is beyond your control'. Blech.

There are standards at the lowest level - meaning things like synch rates for monitors and supported pixel resolutions and color depths. There are a finite set, and if you are in a color depth, you are in it - there is no half-assed 'well, I am *mostly* in the color depth'

There are standards at the high end like markup, or language definitions. I want something to bridge the two, to force browser and OS vendors to adhere to *something* that I can realisticly use as a promise to develop on top of. If that were there, there would be one hell of a lot of people that stopped using proprietary engines tomorrow, or more accurately a lot of people that started developing their own low-cost engines that were standards compliant so they could play other people's content. A little proprietary engine is an Oasis from that problem, you can sit inside Flash and say 'inside here, it's Macromedia's rules, and if I learn them well enough, this will come out looking like what I authored'. The reason for that is that Macromedia (or another vendor) has taken care of drilling all the way down to the 'pixels get stuck on the screen' level so you don't have to. If there was any sort of standard way of doing that, then I'd be willing to wager academia would leap all over it.

It is all of our faults for not pushing standards-based authoring further - but at the same time you eventually push it to a point where its simply 'I am going to write the software for controlling this machine' and not a lot of people want to go there. Once upon a time there were standards for OSes that people could layer on top of that were guarenteed to work a certain way, but while the web world has been gratefully creating standards and moving towards them, the OS standards are moving (a) farther apart and (b) into obscurity, and so there are still massive differences between Netscape on a PC and a Mac, even in 7, not because of the DOM implementation per se, but because of the layers it's all built on top of.

So in some sense I agree - there is way to much proprietary crud and I would love to teach students a more open-ended and appropriate set of technologies. On the other hand - well - an Oasis is, by definition, a shelter, if you take my meaning.

There was a fundamental moment in graphics class not to long ago where someone asked (to paraphrase) "What are we responsible for, and what can we leave up to the OS / GDI / Graphics Card / etc?"

My response: you are fundamentally responsible for every pixel of every screen. That includes anything you make out of those pixels like text, pictures, spaces, backgrounds, everything. (note: that doesn't get to sound, but hey, it was graphics class). That is designing in a system that gives you the absolute ability to create "multimedia" (combined with a standard way to get from one screen to the next, see my notes on timing). No one has of yet gone that far. I would imagine the solution is probably a merging of standards or some such, but even there, eventually, somehow the standards have to go deeper and all the way to the core or we are left with what we have now: standards that work across and on top of pieces of proprietary technologies, and differences between those technologies are forcing time effort and energy away from the use of fully standard compliant authoring.

All this, of course, is just me rambling. I welcome any comments and if nothing else, discourse may spark innovation. That, to me, is the beauty of the web beyond all things: discourse.

I gave it that title not because I thought you and Baldur were warring factions, but because of my history of being an agent provocateur in the "authoring wars" here, especially when I first arrived in the department.

::::But no entity is willing to take it all the way to the metal and say 'a standards compliant browser will have a tool bar of width X - a standards compliant browser will draw its first line of text N pixels from the top edge of the window - text will be N pixels high if it is defined as Y point size' - we're getting there (slowly) with the DOM / CSS etc. - but there are those last things like 'well, how it actually draws images is left up to the browser' or 'well, the width of the toolbar is beyond your control'. Blech.::::

I personally find this lossy approach quite acceptable. Giving up this control means that the content is more adaptable. If you define things too strictly then you have to account for each and every special situation were your original specs don't apply. One design for a 72dpi 800x600 display. Another one for a 180dpi 1600X1200 display.

Accounting for everything, ad infinitum.

This is like quantum physics, the more you know/define about one thing, the less can be found/defined about the other.

Giving up control in exchange for flexibility is for the most part a good thing in light of how varied computers are, and that's not even mentioning alternative devices.

But then again I'm not into the 3d problem at all. 2d has always been flexible (look at how many formats a novel or even a graphic novel can appear in). I suspect that the way 3d works means the flexible approach results in unacceptable losses.

You're spot on when you say that the real power comes from storing and manipulating data in a standards compliant way.

The above-mentioned flexiibility is also part of the future-proofing issue. If you define things too strictly and bind it to today's hardware, what happens when you want to review the project in a decade's time and the hardware you wrote it for doesn't exist anymore?

And people will want to view previous art in ten years time. Sadly, they won't be able to view 99% of it.

I'd also argue (and his is probably a bit of a controversial point that I need to elaborate on, some day) that Free Software (GPL) serves the same effective function as open standards.

So that if you can't use an open independent standard, using GPLed tools will give you effectively the same benefits.

For example, because id software GPLed the Doom engine I can still demo the game if I want to reference it as a part of a teaching session. It runs on practically anything from the lecture room's laptop to one of the student's colour mobile phone.

I came upon another example yesterday when I was supposed to demo Riven for some of the students (the main lecturer thinks it is an example of 'effective multimedia').

The postgrad lab is OS X only for a variety of reasons.

And Riven just wouldn't work on any of the machines, Classic simply wouldn't have it.

Give it five years and Riven won't work on any of the then currently maintained Macintoshes, while I'd be willing to bet that you could play quake and quakeII on all of them.

It's a problem that I don't really know the proper solution for. The best I can to is to try my best to use Free Software and open standards on my projects, although I am fully aware of the fact that that approach only works in some, limited situations.

About this Entry

This page contains a single entry published on December 11, 2002 12:30 PM.

typing tests was the previous entry in this blog.

ideas at blog speed is the next entry in this blog.

Find recent content on the main index or look in the archives to find all content.

Archives

Category Archives