I was watching a James Cameron interview on the Science Channel's website and they asked him what he thought the next big thing was. And he mentioned two things: faster frame rates and making things brighter. Unfortunately, the full-length interview is no longer available.
Cameron talks about how they started doing 3D differently. Instead of doing it the old fashioned IMAX way, with a pair of cameras side-by-side, he talks about how they started experimenting with "active convergence"- which means having two cameras and changing the angle between them- parallel for objects out at infinity, angled together to converge on something close. This is the basis of stereoscopic vision. Human eyes do the same thing. When you look at something really close to your face, you go cross-eyed. And that's a big part of what tells you that it's really close. The double image produced in the non-focused background, on the other hand, is an indirect cue. The main reason you don't see double-image artifacts is that film making isn't that close. You could, however, create quite a miniature effect by using it (in the next 3D Bugs movie, such techniques might be used).
Cameron et al quickly discover that active convergence has limitations. Namely, the fact that movie cameras are enormous compared to human eyes. It takes a large lens to match the resolution of our natural optics. And you can't put two large lenses too close together without having them crash into one another. So they started doint things differently: "We completely revolutionized... when we went from a parallel system to a beam splitter system." And that's the big gimmick with Avatar: that 3D isn't a gimmick any more. Unlike past 3D movies, where the 3D was omnipresent (think of Beowulf for instance), Avatar's 3D was designed to go unnoticed. In fact, Avatar represents the first time 3D wasn't being used to create scenes with impossibly deep 3D that don't represent how the eye and brain actually work. Think of the spear scene in Beowulf. The tip of the spear is pointed at the viewer's eye. Part of what tells you that it's so close is that it's huge compared to the background. But if you were actually to have a spear in your face, and you were to focus directly on it (as it appears in the shot), you'd also see a blurry double image of the background. But the makers of Beowulf keep the background clean and singular. After all, the mind can derive 3D from the primary cue without resorting to distracting secondary cues. In making Avatar, Cameron uses active convergence, enabled by beam splitters, to operate in the middle distance. No cross-eyed shots. Avatar is a movie of dense foreground shots- numerous 3D plants- and big sky long shots. There are several shots that are extreme closeups- Jake's face when he emerges from hypersleep, or while he's opening his own eyes after finishing a link session. But these shots have no background cues, so we never experience the lack of secondary cues.
Here's where I get crazy. Cameron says we're finally reaching a plateau.
Asked whether we'll ever be able to take off the glasses, Cameron says no. He says, "For a big theatrical exposition in a movie theater, you're always going to wear glasses." (And then Jon Landau, Avatar's producer, fantasizes about using the glasses to make money: "It's not that I have to wear the glasses, but that I get to wear the glasses" and goes on to spout a fantasy about how your glasses could also be your ticket, paid for at home.)
First of all, Cameron is wrong about the plateau. But just as Cameron represents the (half)generation of filmmakers that were inspired to surpass George Lucas at his own game, there is plenty of room for new innovation. I'd like to say something about George Lucas. People may not have been impressed with his new Star Wars trilogy, but he, and ILM, did a massive amount of innovation a whole ten years before Cameron pulled the trigger on Avatar. Just as Cameron and (many) others spent fifteen years (post Return of the Jedi) moving the tech forward for Lucas.
Cameron mentioned two innovations: faster frame rates and brighter pictures. Both would be significant. By "brighter" he may mean "higher contrast." Film projection allows contrast ratios of thousands to one. Some TV displays allow over a million to one. That's the difference between looking at a photograph of a sunset and actually feeling like you're looking at the setting sun. One of the biggest obstacles to the suspension of disbelief is that movies are relatively dark. That's one of the reasons that the cinema style favors relatively dark scenes, at night, under clouds. Using artful but inherently unnatural lighting (oh, how bright the moon is, and how conveniently located). Higher contrast would both add a level of realism never before imagined and freedom from the constraints of traditional lighting.
Higher frame rates would provide some interesting benefits. The most important thing it would do would be to allow fast POV action shots to be believably achieved. That's because, at 24 fps, you almost can't pan at all without creating stroboscopic artifacts that tell the brain that what it's seeing is not real. In other words, the camera can't turn its head without making the scene unintelligible. There are some tricky ways around this: artificial motion blur, carefully establishing spatiality before making a motion (using an establishing shot for instance). The best way around it is to keep the camera still. That's what movie making equipment was designed to do in the first place.
But if you double or quadruple the frame rate, and you can also quadruple your pan speed and stop relying on tricks. This would allow the film maker to break the proprioception barrier- to get into the head of the viewer, to get them to identify kinesthetically with your characters. Many people who saw Speed Racer (2008) missed the point(s). It was panned by people who called themselves "film critics," but were actually operating on a purely emotional level. What made Speed Racer great was that it made so many of its viewers sick. What kind of sickness? Motion sickness. Because the movie convinced them that they were moving in ways they weren't prepared for. The only way to enjoy a movie like that is to do so actively: to put yourself in the driver's seat. Because it was so innovative in this and other ways, it still managed to make Richard Corliss's top ten list. Just imagine what the Wachowski Bros could have done with a faster frame rate?
And now for the reason I'm writing this blog: Let's talk about how to make 3D more believable.
When you're watching a 3D movie, the film makers have decided what your eye can look at, what is in focus. This used to be done in more subtle ways- the use of eye dominants (the brightest thing on screen should always bee the thing you want to draw attention to- a rule made to be broken). The use of aberrant motion. Simply using color contrast. And the oldest trick of them all: movie stars. Pretty faces to attract the eye.
With 3D techniques, film makers have another trick in the bag: the ability to constrain the focal plane on the object-of-focus. And that works well.
However, the illusion of 3D arises in the brain, not in the eye, not in the picture the eye is looking at. By dictating what the eye can focus on- but giving it only one option- a huge part of the illusion is lost. Unfortunately, there's only two ways around this (that I know of). Both are difficult to do.
One, the oldest and hardest to revive, we'll call the multi-planar staging approach. Imagine instead of projecting your 3D image on a single flat screen, you project images on numerous screens that are actually at different distances from the eye. Besides being very difficult to film for and to execute, it also only works well for only a couple people sitting in the best seats. And projection is out of the question: projection screens have to be physically moved out of the way because they are, of necessity, not transparent. Instead, you use special screens with countless switches allowing flat pixels (super bright LEDs- the brighter, the smaller) to be reoriented so you're seeing them edge on- making them almost transparent when not in use. And you'd use these screens to show 3D images to add extra depth.
This would allow the eye to choose its own focal plane, which would create a sense of 3D that would make Avatar look like King Kong (the first one) in comparison.
You'd really be creating a simplified hologram. But unlike the traditional application of hologram technology, this one would look enormous (even if it was no bigger than a home theater) and at least half-immersive.
The other way is actually easier, even though it isn't technologically feasible (yet).
Instead of projecting multiple screens at multiple distances and letting the eye choose, you simply track what the eye is looking at and provide an in-focus image before in less time than it takes the eye to focus or the mind to take notice- somewhere around 1/15th of a second. You just need fast computers (about a magnitude more powerful than the present crop of affordable GPUs). At this point, the movie theater becomes obsolete. The best way to achieve this would be with head wearable displays. Virtual reality "glasses." So Cameron would be right about not taking off the glasses. He might be wrong about the future of "big theatrical expositions" though. That's likely to change in some slow subtle ways.
It would also pose a considerable challenge to film makers. Cameras would need to collect a distance map (using an array of laser range finders) of the image that could be used to translate the image into a virtual hologram of the image in post. One cool thing is that your could use a single camera to produce a 3D image- you'd just have to have separate motion trackers, display drivers, and displays for each eye.
It would take some very fast computers but, ultimately, movies wouldn't become much more data heavy. It would be like adding an extra audio track. The technique could be used in both live action and CG-based movies. You could even 3D-ize movies that weren't filmed using the techniques (though the mind would suspend believe every time it tried to focus on something in an out-of-focus plane). In that sense, it would be better used for TV shows. Cheap TV shows filmed during the day. Woohoo.
Fast frame rates, much higher contrast, and self-selectable 3D focus planes- together these could produce a visual experience indistinguishable from reality. Of course, if you really want to go there, you'd need to be able to not just move your eyes. You'd need to be able to move your head too. And if we're really going for the next big thing, it should go without saying that the image should be at least 180 x 180 degrees wide and tall.
All that means is filming at a higher resolution and a wider angle. Not necessarily 360 degrees though (your camera crew needs to hide somewhere).
When? Actually quite soon. You'll start seeing cameras capable of encoding distance data in the next five years. Ultra HD will eventually become the next TV standard, but the exact technology hasn't been designed yet. I predict we'll see viable HWD's in the next five, and the technology will mature in the next ten. Fifteen years from now, we may expect feature films using self-selectable 3D. It'll be a test film, nothing like Avatar. Fully mature examples of the technology may take another five to ten years to arrive. So, I place the next plateau somewhere around 2030.
Is there a plateau after that If you've read my last post, you already know the answer. There's usually another level. In this case, I think the next level will arrive concurrently. It requires adding senses, especially breaking the proprioception barrier. Imagine sitting in a chair that sways back and forth to simulate walking, or spins you left and right to cue your inner ear that you're turning. Imagine having a virtual body- arms and legs that move as your point of view moves, or as you choose to move them, and which are lit in ways that are consistent with the scene you're immersed in. Why are arms and legs important if you can't interact with the movie?
We're talking about movies here, after all, not games- though there are already good examples of games that have exceeded the ambitions of theater (Call of Duty, Modern Warfare is a great example- the final battle, at least for me, was emotionally immersive in a way no movie has achieved- I personally felt responsible to save the millions of lives that would be snuffed out if I didn't succeed).
So, how does seeing your limbs break the proprioceptic barrier? Simple. Arms and legs are in the middle territory between the internal and external sensory fields. Your arms are embassies into "reality." The fact that you can move them, using signals from within, to affect part of what you see outside of you lends reality to everything that you can see because it belongs to the same visual continuum.
Try this exercise. Close your eyes in a completely dark room. Now, move your hand in front of your face. If you are indeed in a completely dark room you won't be able to see anything. Right? But you tell me, can you see your hand? Not visually, no. But you "see" something. Try changing the shape of your hand. Make a fist. Wiggle your fingers. Do you have any trouble visualizing the shape your hand has taken? Is it even "visualization" that your mind is engaged in?
The frontier beyond that is probably something internal- jacking into the optic nerve directly, creating false nerve signals, creating technologically-driven conscious dream states. The Holoband of Caprica instead of the Holodeck of Star Trek. And beyond that? For that, read my last post.
No comments:
Post a Comment