pheloniusfriar: (Default)
More on this later, but I'm about to launch a Kickstarter (with Jeff Green) to "port" the groundbreaking (and award-winning) CD-ROM title "Midnight Stranger" from its 1995 technological roots (Macromedia Director running on Windows 95... it won't run on anything newer than Windows/XP without an emulator) onto modern systems, specifically anything that can render HTML5 canvases and video. This is the first semi-public (my blog is an Internet backwater that nobody seems to notice, heh) unveiling of the proof-of-concept I wrote to prove it could be done. At the time Midnight Stranger was released, DVDs didn't exist and with the technology at the time you could only fit around 6 minutes of crappy video onto a CD-ROM... hardly enough for an immersive or engaging experience. The solution is a bit cheesy, but the limitations are soon forgotten (from my experience and watching others): instead of full-frame video, a background image is used and little videos overtop that background image are used for people's heads or other movements. It is quite funny looking now, but over an hour of video could then be put on a single CD-ROM and allow for the telling of complex stories with the very limited computer capabilities at the time. While we're way past this now, it it still engaging on the desktop, and the format works for mobile devices because the bandwidth required is so, so much less than doing full-frame video. Where it really shines is in the use of a novel interface called the Mood Bar, where you respond emotionally rather than analytically.

I think this recollection of Jeff's sums it up best: When this production was first premiered at Macworld in San Fransisco, despite appearing there in a 4-foot booth with a single 18-inch poster and presented on the smallest desktop Macintosh of the time with discount bin headphones, and situated only forty feet from a million-dollar Sony booth with 30-foot screens and live actors, Midnight Stranger consistently had large crowds and a 30-minute wait-time for just a few minutes of interaction. It became clear that for those for whom this form was well-suited, it gave the opportunity to achieve a significant sense of virtuality; the combination of eye contact, immersive sound, and the onscreen person‘s apparent response to their 'input' being sufficient engagement to facilitate periods of true suspension of disbelief — the holy grail of media.

As it stands, my demo is only two levels deep and it just repeats (and yes, it's a partial scene pulled from Midnight Stranger). It starts with the background image and you need to make a Mood Bar selection. That launches a video. If you choose wisely, you will be presented with another Mood Bar choice (if you don't choose wisely, you wind up back at the picture and can try again). The second mood bar has 3 choices, a movie runs, and then you wind up back at the picture and can do it all again. The menu and help buttons "work" now (they just report being pressed for now... any functionality can be added later). For me, it worked on Firefox, on Pale Moon (a fork of Firefox), and on Chrome. It did not work on my old LG mobile phone or the default browser for my Ubislate 3G7 (ultra cheap) tablet, but the video did work on the Pale Moon for Android browser on said-same Ubislate (I'm not sure if the audio was working, so that might be a problem). If you get a chance to try it, please let me know how it works out for you (any information on what operating system and/or device and/or browser you were using, and how it worked or didn't, would be really helpful). More work will continue to be done to get it working on more devices, but this was enough for us to be confident we could do it. Fyi, my previous post was me digging into ways of making it run on more systems... the initial proof-of-concept was written months ago when we were first wondering whether it could be done. At the time, it would only run on Chrome, but this latest (as stated above) runs on at least two browser families ;).

Click on the image to open the demo in a new tab:

Here's the Kickstarter info should you be so inclined to take a sneak-peek at what's coming up. I've been told the bio-video is particularly entertaining (somewhat at my expense, but... it's an interesting way to present some of my resume, heh):

Obeing Kickstarter Information
pheloniusfriar: (Default)
"The nice thing about standards is that you have so many to choose from."
— Andrew S. Tanenbaum, Computer Networks, 2nd ed., p. 254

So... I'm learning to write full distributed applications in HTML5, ECMAScript 5.1 (aka JavaScript), and CSS3 (I've done this sort of stuff in system programming languages, but not in this programming environment). One would think that the adoption of global standards would lead to some facility of use or at least the availability of well-structured documentation (yes, I can hear anyone who has ever done any serious programming laughing right now, I know I am laughing at myself for that silly statement). My albatross-du-jour is I'm struggling with HTML5 video. I finally got ffmpeg compiled with all the bits I needed for all the different major formats (MP4, WebM, Ogg)... and again, anyone who has ever even brushed up against this stuff is probably at least smirking at me (grimacing perhaps with the memory). Alas, all three of those "video formats" are actually only generic container types that can hold all manner of encodings of video and audio that can all be mutually exclusive — just because something says it can play MP4 files, doesn't mean it will play my MP4 files. The clearest explanation of this dog's breakfast can be found in Mark Pilgrim's excellent summary Dive Into HTML5: Video On The Web. It is also notable for making me feel like I might not be alone in my despair, and that there may be some justification for the head-scratching I've had to do, with his statement: "HTML5 defines a standard way to embed video in a web page, using a <video> element. Support for the <video> element is still evolving, which is a polite way of saying it doesn't work yet." Yeah. That. Definitely.

For anyone new to this bizarre realm that powers most of our planet's economy and society today, web pages are dynamic entities whose content is specified with HTML5, whose presentation is specified by CSS, and whose interactivity is controlled by ECMAScript. ECMAScript interfaces to the contents of the web pages being displayed through the browser's Data Object Model (DOM). Like all the other technologies discussed, there are many evolutions of this interface as well. For HTML5 video, because there were too many commercial and/or ideological interests (and even technical issues that prevent a "one size fits all" approach), there have been a proliferation of "standard" video formats available to HTML5-compliant browsers. How was this done? It happened by providing nebulous mechanisms that either let the browser pick from a number of different available video formats (it selects the one it likes the most), or let the ECMAScript determine what formats are supported by the browser it's running on, and letting the code pick which one it wants (which works well, until it gets its guess wrong, right?). The W3C then called this horrible idea "the standard" and has moved on to let the world sort it out on its own. Can't blame 'em, but it sucks for anyone who wants to serve up video to more than one kind of web browser. If nothing else, a provider of video content has to encode the video into multiple formats and support them all so the most number of people won't notice how broken this idea is. Again, in my case, that means I have to encode every video to be served up into MP4, WebM, and Ogg. Every single one. Sigh. Then, the code needs to pick one of them to tell the browser to display, and here things get murky beyond belief.

There are any number of web pages and blog entries that provide "guidance" on how to deal with this situation. Almost to the one they deal with it using only "voodoo programming" techniques: "the practice of coding according to superstition, guesses, or anything other than logic. Voodoo programming is a rather broad term for situations where a programmer uses a piece of code without truly understanding how it works." This makes me stabby. I need to know what I'm doing, because invariably something will break and knowing why I used something earlier will usually point to the solution quickly, rather than letting me spend days or weeks bashing my head on my desk trying to solve the problem (pro tip: it feels so good when you stop). This is also what separates boffo programmers from lame coders: a programmer understands what they are doing and why. The DOM provides a method on the video element called canPlayType() — you give it a MIME type string and it tells you whether the browser thinks it can play the video or not. Sort of. "Common" types include (continuing with the examples already used): "video/mp4", "video/webm", and "video/ogg". If the browser is cool with a format, it returns "maybe" (literally, the string "maybe")... as in "maybe" I might be able to play this video, but I'm not really so sure. If it knows it can't play it, it returns a null string ("")... sigh (a bad programming practice in my mind, but it can be tested for at least). Can the browser play it? Dunno... only the person trying to watch the video will be able to answer that (presuming there are no network issues or other problems that might make them think the browser is the problem rather than the signal coming out of the back of their computer/phone/tablet/toaster). I should note, that the only way to find out the above was by hours and hours and searching and comparing bread-crumbs on dozens of web sites looking for information that agreed with any other web site (including reading through many long histories of the subject, which while interesting, were definite productivity sucks). The question then becomes, can we be more sure before we make a choice amongst the possible options for video playback? The answer is, somewhat (but only partially) fortunately, yes. You can also specify what codec support is required to play back the videos you've encoded in the string passed to the canPlayType() method of your video element. If you specify the MIME type and a list of codecs, the browser might respond with the string "probably" to your invocation of canPlayType() if it has those codecs identifiably available to it and installed. I would like to take a moment to emphasize that the "gold standard" of compatibility here is "probably", not "yes", not "barring civil uprisings, earthquakes, or volcanoes", not "yo momma", but "probably"... sigh. And to top it off, codec specification is where the voodoo completely takes over. For example, one of the main information sites suggests (for the three main mime types):
myVideo.canPlayType('video/ogg; codecs="theora, vorbis"');
myVideo.canPlayType('video/mp4; codecs="avc1.4D401E, mp4a.40.2"');
myVideo.canPlayType('video/webm; codecs="vp8.0, vorbis"');
Clear as mud right? Ogg makes at least a little bit of sense (if you somehow magically know the video format is theora and the audio format is vorbis), but why "vp8.0", and what the hell is "avc1.4D401E"??? Another site pointed me to the source code for the Clappr "extensible media player for the web", which contained the following vexing bit of code:
const MIMETYPES = {
'mp4': ["avc1.42E01E", "avc1.58A01E", "avc1.4D401E", "avc1.64001E", "mp4v.20.8",
  "mp4v.20.240", "mp4a.40.2"].map((codec) =>
  { return 'video/mp4; codecs="' + codec + ', mp4a.40.2"'}),
'ogg': ['video/ogg; codecs="theora, vorbis"', 'video/ogg; codecs="dirac"',
  'video/ogg; codecs="theora, speex"'],
'3gpp': ['video/3gpp; codecs="mp4v.20.8, samr"'],
'webm': ['video/webm; codecs="vp8, vorbis"'],
'mkv': ['video/x-matroska; codecs="theora, vorbis"'],
'm3u8': ['application/x-mpegurl']
MIMETYPES['3gp'] = MIMETYPES['3gpp']
Just sticking to the codecs part for the three main types I'm interested in, there are three "avc1" and three "mp4v" types... and WebM support is asking for the "vp8" codec (rather than the "vp8.0" codec specified by the previous site), Ogg also apparently supports something called "dirac" and there's another audio format called "speex". I know there's also a VP9 format that can be used with WebM as well (again, I've read a lot of web sites lately on this).

So, I'm thinking, maybe if I knew the exact codecs I used to encode the videos I'm using, I can just check for that and be done with this rubbish. Back to ffmpeg on Linux and let's give it a try... encode an Ogg video and I get:
Stream mapping:
  Stream #0:0 -> #0:0 (cinepak (native) -> theora (libtheora))
  Stream #0:1 -> #0:1 (pcm_u8 (native) -> vorbis (libvorbis))
Okay... I think I'm safe with
myVideo.canPlayType('video/ogg; codecs="theora, vorbis"');
for that container. How about WebM? Let's see, I get (letting ffmpeg pick the default codecs to use):
Stream mapping:
  Stream #0:0 -> #0:0 (cinepak (native) -> vp9 (libvpx-vp9))
  Stream #0:1 -> #0:1 (pcm_u8 (native) -> vorbis (libvorbis))
Ah, already something "interesting": I have been encoding the video using the VP9 codec, not the VP8 codec. I went to look to see what VLC (a media player for Windoze) thought the codec was, but... it crashed trying to open the file (it plays fine in my browser though). Sigh. The MediaInfo program on Windows claims the video is in "VP9" format, and uses the "V_VP9" codec; and the audio is in "Vorbis" format, and uses the "A_VORBIS" codec. So, I just learned that either I need to explicitly encode with ffmpeg in VP8 format to support older browsers, or check to see if the browser supports the VP9 video format! Going to look at the MP4 videos (sadly, as expected) was no help at sorting this all out... VLC stated that it used the following codecs:
Codec: H264 - MPEG-4 AVC (part 10) (avc1)
Codec: MPEG AAC Audio (mp4a)
and MediaInfo stated that I used the following encodings for video and audio respectively:
Format                      : AVC
Format/Info                 : Advanced Video Codec
Format profile              : High 4:4:4 Predictive@L1.1
Format                      : AAC
Format/Info                 : Advanced Audio Codec
Format profile              : LC
The fact I'm using the "High" profile for AVC rings some bells that not all portable devices (phones, etc. that I need to support) support that "profile" of AVC (even if they can play "MP4/AVC" videos)... I can't remember exactly where I saw that (or when), but I know I filed something away in my noodle about that. Looking at the Wikipedia page for "H.264" seems to back up that impression by stating that there are different support levels possible and that the one I'm using is a "top of the line" profile that might not be supported everywhere. Back to the drawing board on that one I guess too, but I suspect it'll take me a while to figure out.

Makes sense so far? Uh, huh... nope. Not for me either. The "Rosetta Stone" for me was an obscure answer to an obscure question on Stack Overflow. This in turn clarified that the magic value "avc1.4D401E" actually referred to the "H.264 Main Profile Level 3" profile of the AVC video specification (whatever that means... I'm working on it though...). The other hexadecimal gibberish given in the Clappr code was for other AVC profile values. The article also pointed me to an RFC of all insane places, where the encoding of this mysterious string that everybody uses is actually specified: RFC 6381 (see, in particular, section 3.3). And there's the answer to the question I set out to answer: where are these mysterious strings defined and why? It, unsurprisingly, just raises a rash of other questions and concerns of a more practical nature, and is going to require me to become much more fluent at wrangling ffmpeg to do my bidding, but it's at least a start. I'm off to sort out these and many more challenges (including finishing the painting of my kitchen, which I started far too long ago... pictures when it's done!). Oh, and ffmpeg stated, when it did its MP4 encoding for me:
[libx264 @ 0x67bcc0] profile High 4:4:4 Predictive, level 1.1, 4:4:4 8-bit
[libx264 @ 0x67bcc0] 264 - core 138 r2358 9e941d1 - H.264/MPEG-4 AVC codec -
Copyleft 2003-2013 - - options: cabac=1 ref=3 
deblock=1:0:0 analyse=0x3:0x113 me=hex subme=7 psy=1 psy_rd=1.00:0.00 mixed_ref=1 
me_range=16 chroma_me=1 trellis=1 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=1 
chroma_qp_offset=4 threads=6 lookahead_threads=1 sliced_threads=0 nr=0 decimate=1 
interlaced=0 bluray_compat=0 constrained_intra=0 bframes=3 b_pyramid=2 b_adapt=1 
b_bias=0 direct=1 weightb=1 open_gop=0 weightp=2 keyint=250 keyint_min=10 scenecut=40
intra_refresh=0 rc_lookahead=40 rc=crf mbtree=1 crf=23.0 qcomp=0.60 qpmin=0 qpmax=69 
qpstep=4 ip_ratio=1.40 aq=1:1.00

Stream mapping:
  Stream #0:0 -> #0:0 (cinepak (native) -> h264 (libx264))
  Stream #0:1 -> #0:1 (pcm_u8 (native) -> aac (libfaac))
I followed the article's suggestion to try a program called mp4file and it gave me the following on one of the MP4 files I encoded with ffmpeg:
type avcC
  AVCProfileIndication = 244 (0xf4)
  profile_compatibility = 0 (0x00)
  AVCLevelIndication = 11 (0x0b)
type esds
  objectTypeId = 64 (0x40)
    info = <2 bytes>  15 08  |..|
which means my particular "profile" that I encoded to is "avc1.F4000B, mp4a.40.3" (see the article on how to decipher the codes)... which I have not seen on any of the web site I visited to date. Pale Moon (a Firefox fork), my usual browser, definitely can't play it, and Chrome says it can can't play it either (but it says it can "probably" play WebM with VP9/Vorbis, and Ogg with Theora/Vorbis... and btw, it can play the MP4 file, I tested it). Yeah, this is going to take a while.

p.s. Does anyone need to fix their national postal carrier in this digital age? Here's the hint on how to do it:

"Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway."
— Andrew S. Tanenbaum, Computer Networks, 4th ed., p. 91

p.p.s. My configuration line for ffmpeg is as follows (information that doesn't seem to be readily available anywhere else in a compact form either):
./configure --enable-shared --enable-gpl --enable-version3 --enable-nonfree \
   --enable-libmp3lame --enable-libvorbis --enable-libx264 \
   --enable-libxvid --enable-libfaac --enable-libvpx --enable-libtheora


pheloniusfriar: (Default)

September 2017

3456 7 89
10 11 1213 141516
171819202122 23


RSS Atom

Most Popular Tags

Style Credit

Expand Cut Tags

No cut tags
Page generated Sep. 24th, 2017 09:01 pm
Powered by Dreamwidth Studios