Ex Parte Boyce et al

Patent Trial and Appeal Board

Feb 12, 2018

13343266 (P.T.A.B. Feb. 12, 2018)

United States Patent and Trademark Office
UNITED STATES DEPARTMENT OF COMMERCE
United States Patent and Trademark Office
Address: COMMISSIONER FOR PATENTS
P.O.Box 1450
Alexandria, Virginia 22313-1450
www.uspto.gov
APPLICATION NO. FILING DATE FIRST NAMED INVENTOR ATTORNEY DOCKET NO. CONFIRMATION NO.
13/343,266 01/04/2012 Jill Boyce 076569.0388 1086
21003 7590 02/14/2018
RAKFR ROTTST T P
EXAMINER
30 ROCKEFELLER PLAZA AN, SHAWN S
44TH FLOOR
NEW YORK, NY 10112-4498 ART UNIT PAPER NUMBER
2483
NOTIFICATION DATE DELIVERY MODE
02/14/2018 ELECTRONIC
Please find below and/or attached an Office communication concerning this application or proceeding.
The time period for reply, if any, is set in the attached communication.
Notice of the Office communication was sent electronically on above-indicated "Notification Date" to the
following e-mail address(es):
DLNYDOCKET@BAKERBOTTS.COM
PTOL-90A (Rev. 04/07)
UNITED STATES PATENT AND TRADEMARK OFFICE
BEFORE THE PATENT TRIAL AND APPEAL BOARD
Ex parte JILL BOYCE and DANNY HONG
Appeal 2017-009881
Application 13/343,2661
Technology Center 2400
Before JOHN A. EVANS, JASON J. CHUNG, and
STEVEN M. AMUNDSON, Administrative Patent Judges.
CHUNG, Administrative Patent Judge.
DECISION ON APPEAL
This is a decision on appeal under 35 U.S.C. § 134(a) of the Final
Rejection of claims 1, 8, and 17-19.2 We have jurisdiction under 35 U.S.C.
§ 6(b).
We affirm-in-part.
INVENTION
The invention is directed to techniques enabling a video decoder or a
media aware network element to identify non-base layer pictures, or parts of
non-base layer pictures, required for prediction, thereby allowing for
1 According to Appellants, the real party in interest is Vidyo. App. Br. 2.
2 Claims 2-7 and 9-16 have been canceled. App. Br. 19-20.
Appeal 2017-009881
Application 13/343,266
efficient bitstream pruning. Spec. ^ 2. Claim 1 is illustrative of the
invention and is reproduced below:
1. A method for decoding with a decoding device media
encoded using scalable coding with temporal scalability at a pre­
determined maximum temporal level, the method comprising:
decoding a single bit flag temporal_ nesting_ flag in a
sequence parameter set that is indicative of a temporally nested
structure of layers and pertains to all frames of a coded video
sequence,
decoding a coded first picture of the coded video sequence
with a first layer id;
in response to the decoded temporalnestingflag equal to
1, removing at least one second picture of the coded video
sequence from a reference picture list maintained by the decoder,
the at least one second picture in the reference picture list having
a second layerid, wherein a value of the second layerid is
higher than a value of the first layer id and wherein the second
picture is not used for a prediction of the first picture or any
pictures following the first picture in decoding order;
wherein the at least one second picture of the coded video
sequence would be included in the reference picture list if the
decoded temporal nesting flag were equal to 0;
wherein the temporal nesting flag set to one indicates for
any three frames picA, picB, picC included in the coded video
sequence, that picB is not used for reference of picA:
under a first condition that picB is of a temporal level
lower or equal than the temporal level of picA, and
under a second condition that the temporal level of picC is
lower than the temporal level of picB, and
under a third condition that picC follows picB in decoding
order, and
under a fourth condition that picC precedes picA in
decoding order; and wherein picB is not included in a reference
picture list of picA.
2
Appeal 2017-009881
Application 13/343,266
REJECTION AT ISSUE
Claims 1, 8, and 17-19 stand rejected under 35 U.S.C. § 103(a) as
being unpatentable over Eleftheriadis (US 2009/0116562 Al; published
May 7, 2009) (hereinafter, “Eleftheriadis”). Ans. 3-7.
ANALYSIS
Claims 1 and 8 Under 35 U.S.C. § 103(a)
The Examiner finds Eleftheriadis teaches that in response to the
decoded temporal nesting flag equal to 0, removing at least one second
picture of the coded video sequence from a reference picture list, since a
reference picture frame shall not be used. Ans. 7. And the Examiner finds it
would have been obvious to one having ordinary skill in the art at the time of
the invention to modify setting the temporal nesting flag equal to 1 (as
opposed to 0) without changing the limitation that it places on the structure
of the bitstream. Id. at 8 (citing Eleftheriadis ^ 39). Moreover, the
Examiner finds Eleftheriadis’s teaching of not using a flag clearly implies
the limitation “removing at least one second picture” as recited in claim 1.
Id. (citing Eleftheriadis ^ 39).
Appellants argue Eleftheriadis’s teaching of not using a reference
picture fails to teach or suggest the limitation “removing at least one second
picture” recited in claim 1. App. Br. 14-15; Reply Br. 3—4. We agree with
Appellants.
Eleftheriadis teaches not using a reference picture. Eleftheriadis ^ 39.
There are two ways to “not use” something: (1) keep it there and not use it;
and (2) remove it. The Examiner, however, has not proffered an adequate
rationale or motivation as to why one having ordinary skill in the art at the
3
Appeal 2017-009881
Application 13/343,266
time of the invention would choose option (2) discussed supra, which is to
remove a reference picture (as opposed to keeping it and not using it) as
required by claim l.3 Instead, the Examiner provided an insufficient
conclusory statement that Eleftheriadis’s teaching clearly implies the
limitation “removing at least one second picture” recited in claim 1.
Accordingly, for the reasons discussed supra, we do not sustain the
Examiner’s rejection of: (1) independent claim 1; and (2) dependent claim
18.
Claims 8,17, and 19 Under 35 U.S.C. § 103(a)
The Examiner finds Eleftheriadis teaches the limitations in claims 8,
17, and 19. Ans. 3-7.
Appellants argue Eleftheriadis fails to teach “decoding a single bit
flag temporalnestingflag in a sequence parameter” (App. Br. 11-14), “in
response to the decoded temporal nesting flag equal to 1, removing at least
one second picture of the coded video sequence from a reference picture
list” (id. at 14-15), and H.265 (id. at 15-16). Moreover, Appellants argue
independent claims 8 and 17 are allowable for at least the same reasons as
claim 1 is allowable. Id. at 16. We disagree with Appellants.
We note that “[Ajppellanf s arguments fail from the outset because
. . . they are not based on limitations appearing in the claims.” See In re Self,
671 F.2d 1344, 1348 (CCPA 1982). That is, Appellants arguments are
3 In the event of further prosecution, the Examiner should consider Chen
(US 2011/0280316 Al; published Nov. 17, 2011) (hereinafter, “Chen”) and
Auld (US 5,835,636; issued Nov. 10, 1998) (hereinafter, “Auld”). Chen
teaches video decoders may ignore frames by removing them from the
bitstream and discarding. Chen ^ 119. Auld teaches ignoring or otherwise
discarding a frame. Auld, 16:8-26.
4
Appeal 2017-009881
Application 13/343,266
unpersuasive because none of these features are recited in claims 8 and 17.
For instance, claims 8 and 17 recite encoding, whereas claim 1 recites
decoding. Moreover, H.265 is not even recited in claims 8 and 17.
Accordingly, for the reasons discussed supra, we sustain the
Examiner’s rejection of: (1) independent claims 8 and 17; and (2) dependent
claim 19.
DECISION
The Examiner’s decision rejecting claims 1 and 18 under 35 U.S.C.
§ 103(a) is reversed.
The Examiner’s decision rejecting claims 8, 17, and 19 under
35 U.S.C. § 103(a) is affirmed.
No time period for taking any subsequent action in connection with
this appeal may be extended under 37 C.F.R. § 1.136(a)(l)(iv).
AFFIRMED-IN-PART
5
Application/Control No. ApplicantfsVPatent Under Patent
Notice of References Cited
13/343,266
Appeal No.
2017-009881
Examiner Art Unit
2483 Page 1 of 1
U.S. PATENT DOCUMENTS
ie Document NumberCountry Code-Number-KinrJ Code
Date
MM-YYYY Name Classification
A US- US 2011/0280316 A1 11-2011 Chen et al.
B US- US 5,835,636 11-1998 Auld
c US-
D US-
E US-
F US-
G US-
H US-
i US-
J US-
K US-
L US-
M US-
FOREIGN PATENT DOCUMENTS
* Document Number
Country Code-Number-Kind Code
Date
MM-YYYY Country Name Classification
N
O
P
Q
R
s
T
NON-PATENT DOCUMENTS
* include as applicable: Author, Title Date, Publisher, Edition or Volume, Pertinent Pages)
U
V
w
X
’A copy of this reference is not being furnished with this Office action. (See MPFP § 707.05(a).)
Dates in MM-YYYY format are publication dates. Classifications may be US or foreign.
U.S. Patent and Trademark Office
PTO-892 (Rev. 01-2001) Notice of References Cited Part of Paper No.
US 20110280316A1
(i9) United States
(12) Patent Application Publication oo) Pub. No.: US 2011/0280316 Al
Chen et al. (43) Pub. Date: Nov. 17,2011
(54) FRAME PACKING FOR ASYMMETRIC
STEREO VIDEO
(75) Inventors: Ying Chen, San Diego, CA (US);
Marta Karczewicz, San Diego, CA
(US)
(73) Assignee: QUALCOM Incorporated, San
Diego, CA (US)
(21) Appl. No.: 13/035,448
(22) Filed: Feb. 25, 2011
Related U.S. Application Data
(60) Provisional application No. 61/334,253, filed on May
13, 2010, provisional application No. 61/366,436,
filed on Jul. 21, 2010, provisional application No.
61/433,110, filed on Jan. 14, 2011.
Publication Classification
(51) Int. Cl.
H04N 7/26 (2006.01)
(52) U.S. Cl......... . 375/240.25; 375/240.26; 375/E07.027;
375/E07.076
(57) ABSTRACT
An asymmetric frame of a coded video bitstream may include
a full resolution picture of a left view and a reduced resolution
picture of a right view, where the left and right views form a
stereo view pair for three-dimensional video playback. In one
example, an apparatus includes a video encoder configured to
receive a first picture of a first view of a scene having a first
resolution, receive a second picture of a second view of the
scene having a reduced resolution relative to the first resolu­
tion, form an asymmetric frame comprising the first picture
and the second picture, and encode the asymmetric frame. In
this manner, decoders of varying capabilities may receive the
same bitstream, and the bitstream may consume less band­
width than one or more bitstreams having full resolution
pictures of a stereo view pair. The bitstream may have better
quality than a bitstream having subsampled pictures.
Patent Application Publication Nov. 17, 2011 Sheet 1 of 12 US 2011/0280316 A1
10
FIG. 1
Patent Application Publication Nov. 17, 2011 Sheet 2 of 12 US 2011/0280316 A1
1
Patent Application Publication Nov. 17, 2011 Sheet 3 of 12 US 2011/0280316 A1
FI
G
. 3
Patent Application Publication Nov. 17, 2011 Sheet 4 of 12 US 2011/0280316 A1
X X X X X X X X o o o o
X X X X X X X X o o o o
X X X X X X X X o o o o
X X X X X X X X o o o o
X X X X X X X X o o o o
X X X X X X X X o o o o
X X X X X X X X o o o o
X X X X X X X X o o o o
o
o'
i—in
>-w<
inQ HI
£ “■
/\
X X X X X X X X o o o o o o o o
X X X X X X X X CMo o o o o o o o o
X X X X X X X X o o o o o o o o
X X X X X X X X o o o o o o o o
X X X X X X X X o o o o o o o o
X X X X X X X X o o o o o o o o
X X X X X X X X o o o o o o o o
X X X X X X X X o o o o o o o o
FI
G
. 4
Patent Application Publication Nov. 17, 2011 Sheet 5 of 12 US 2011/0280316 A1
o o o o o o o o
o o o o o o o o
o o o o o o o o
o o o o o o o o
X X X X X X X X
X X X X X X X X
X X X X X X X X
X X X X X X X X
X X X X X X X X
X X X X X X X X
X X X X X X X X
X X X X X X X X
g
on
i—
LU
>-
CO<
Q
111
*
o<0.
2
2
LL
X X X X X X X X o o o o o o o o
X X X X X X X X 10
2 o o o o o o o o
X X X X X X X X o o o o o o o o
X X X X X X X X o o o o o o o o
X X X X X X X X o o o o o o o o
X X X X X X X X o o o o o o o o
X X X X X X X X o o o o o o o o
X X X X X X X X o o o o o o o o
FI
G
. 5

Patent Application Publication Nov. 17, 2011 Sheet 7 of 12 US 2011/0280316 A1
CN
CM
CN
X X X X X X X X
X X X X X X X X
X X X X X X X X
X X X X X X X X
X X X X X X X X
X X X X X X X X
X X X X X X X X
X X X X X X X X
X X X X X X X X
X X X X X X X X
X X X X X X X X
X X X X X X X X
X X X X X X X X
X X X X X X X X
X X X X X X X X
X X X X X X X X
X o X o X o X o X o X o X o X o
X o X o X o X o X o X o X o X o
X o X o X o X o X o X o X o X o
X o X o X o X o X o X o X o X o
X o X o X o X o X o X o X o X o
X o X o X o X o X o X o X o X o
X o X o X o X o X o X o X o X o
X o X o X o X o X o X o X o X o
X o X o X o X o X o X o X o X o
X o X o X o X o X o X o X o X o
X o X o X o X o X o X o X o X o
X o X o X o X o X o X o X o X o
X o X o X o X o X o X o X o X o
X o X o X o X o X o X o X o X o
X o X o X o X o X o X o X o X o
X o X o X o X o X o X o X o X o
Q
gw
^ Q O
in
O uj
o z ol > m z)
UJ Q_
UL
FI
G
. 7
Patent Application Publication Nov. 17, 2011 Sheet 8 of 12 US 2011/0280316 A1
HI
l- y
u- >
LU
HI
>-ss
0£
FI
G
. 8
Patent Application Publication Nov. 17, 2011 Sheet 9 of 12 US 2011/0280316 A1
FIG. 9
Patent Application Publication Nov. 17, 2011 Sheet 10 of 12 US 2011/0280316 A1
200
202
204
206
208
210
FIG. 10
Patent Application Publication Nov. 17, 2011 Sheet 11 of 12 US 2011/0280316 A1
RECEIVE ENCODED PICTURE
OF LEFT EYE VIEW
r220
ENCODE PICTURE OF
LEFT EYE VIEW
222
RECEIVE ENCODED PICTURE
OF RIGHT EYE VIEW
.-224
REDUCE RESOLUTION OF
RIGHT EYE VIEW PICTURE
226
ENCODE PICTURE OF RIGHT
EYE VIEW BASED ON A
PICTURE OF LEFT EYE VIEW
228
OUTPUT ENCODED LEFT EYE
VIEW PICTURE
230
OUTPUT ENCODED RIGHT EYE
VIEW PICTURE
232
FIG. 11
Patent Application Publication Nov. 17, 2011 Sheet 12 of 12 US 2011/0280316 A1
RECEIVE ENCODED PICTURE
OF LEFT EYE VIEW
V240
DECODE PICTURE OF
LEFT EYE VIEW
242
RECEIVE ENCODED PICTURE
OF RIGHT EYE VIEW
.244
DECODE PICTURE OF RIGHT
EYE VIEW BASED ON A
PICTURE OF LEFT EYE VIEW
246
UPSAMPLE DECODED RIGHT
EYE VIEW PICTURE
248
OUTPUT DECODED LEFT EYE
VIEW PICTURE
250
OUTPUT DECODED RIGHT EYE
VIEW PICTURE
252
FIG. 12
US 2011/0280316 A1
1
Nov. 17,2011
FRAME PACKING FOR ASYMMETRIC
STEREO VIDEO
[0001] This application claims the benefit of U.S. Provi­
sional Application No. 61/334,253, filed May 13, 2010, U.S.
Provisional Application No. 61/366,436, filed Jul. 21, 2010,
and U.S. Provisional Application No. 61/433,110, filed on
Jan. 14, 2011, each of which is hereby incorporated by refer­
ence in its entirety.
CROSS-REFERENCE TO RELATED
APPLICATIONS
[0002] This application is related to the following co-pend-
ing U.S. patent application:
“ONE-STREAM CODING FOR ASYMMETRIC STEREO
VIDEO” by Ying Chen et al., having Attorney Docket No.
102361, filed concurrently herewith, assigned to the assignee
hereof, and expressly incorporated by reference herein.
TECHNICAL FIELD
[0003] This disclosure relates to video coding.
BACKGROUND
[0004] Digital video capabilities can be incorporated into a
wide range of devices, including digital televisions, digital
direct broadcast systems, wireless broadcast systems, per­
sonal digital assistants (PDAs), laptop or desktop computers,
digital cameras, digital recording devices, digital media play­
ers, video gaming devices, video game consoles, cellular or
satellite radio telephones, video teleconferencing devices,
and the like. Digital video devices implement video compres­
sion techniques, such as those described in the standards
defined by MPEG-2, MPEG-4, ITU-T H.263 or ITU-T
H.264/MPEG-4, Part 10, Advanced Video Coding (AVC),
and extensions of such standards, to transmit and receive
digital video information more efficiently.
[0005] Video compression techniques perform spatial pre­
diction and/ortemporal prediction to reduce orremove redun­
dancy inherent in video sequences. For block-based video
coding, a video frame or slice may be partitioned into mac­
roblocks. Each macroblock can be further partitioned. Mac­
roblocks in an intra-coded (I) frame or slice are encoded using
spatial prediction with respect to neighboring macroblocks.
Macroblocks in an inter-coded (P or B) frame or slice may use
spatial prediction with respect to neighboring macroblocks in
the same frame or slice or temporal prediction with respect to
other reference frames.
[0006] Efforts have been made to develop new video cod­
ing standards based on H.264/AVC. One such standard is the
scalable video coding (SVC) standard, which is the scalable
extension to H.264/AVC. Another standard is the multi-view
video coding (MVC), which has become the multiview exten­
sion to H.264/AVC. A joint draft of MVC is in described in
JVT-AB204, “Joint Draft 8.0 on Multiview Video Coding,”
28* JVT meeting, Hannover, Germany, July 2008, available
at http://wftp3.ltu.int/av-arch/jvt-site/2008_07_Hannover/
JVT-AB204.zip. A version of the AVC standard is described
in JVT-AD007, “Editors’ draft revision to ITU-T Rec.
H.264IISO/IEC 14496-10 Advanced Video Coding—in
preparation for ITU-T SG 16 AAP Consent (in integrated
form),” 30th JVT meeting, Geneva, CH, February 2009,”
available from http://wftp3.ltu.int/av-arch/jvt-site/2009_
01_Geneva/JVT-AD007.zip. The JVT-AD007document
integrates SVC and MVC in the AVC specification.
SUMMARY
[0007] In general, this disclosure describes techniques for
supporting stereo video data, e.g., video data used to produce
a three-dimensional (3D) effect. To produce a three-dimen­
sional effect in video, two views of a scene, e.g., a left eye
view and a right eye view, are shown simultaneously or nearly
simultaneously. The techniques of this disclosure include
forming a bitstream having packed frames, where a packed
frame corresponds to a single frame having data for two views
of a scene. In particular, the techniques of this disclosure
include encoding a packed frame having a full resolution
frame of one view of a scene and a reduced resolution frame
of another view of the scene. The reduced resolution frame
may be encoded with respect to a frame of the other view. In
this manner, this disclosure also provides techniques for per­
forming inter-view prediction for a reduced resolution frame
of a packed frame.
[0008] In one example, a method includes receiving a first
picture of a first view of a scene having a first resolution,
receiving a second picture of a second view of the scene
having a reduced resolution relative to the first resolution,
forming an asymmetric frame comprising the first resolution
picture and the reduced resolution picture, encoding the
asymmetric frame, and outputting the asymmetric frame.
[0009] In another example, an apparatus for encoding video
data includes a video encoder configured to receive a first
picture of a first view of a scene having a first resolution,
receive a second picture of a second view of the scene having
a reduced resolution relative to the first resolution, form an
asymmetric frame comprising the first picture and the second
picture, and encode the asymmetric frame.
[0010] In another example, an apparatus includes means for
receiving a first picture of a first view of a scene having a first
resolution, means for receiving a second picture of a second
view of the scene having a reduced resolution relative to the
first resolution, means for forming an asymmetric frame com­
prising the first picture and the second picture, and means for
encoding the asymmetric frame.
[0011] In another example, a computer program product
includes a computer-readable storage medium having stored
thereon instructions that, when executed, cause a processor to
receive a first picture of a first view of a scene having a first
resolution, receive a second picture of a second view of the
scene having a reduced resolution relative to the first resolu­
tion, form an asymmetric frame comprising the first picture
and the second picture, encode the asymmetric frame, and
output the encoded asymmetric frame.
[0012] In another example, a method includes receiving an
encoded asymmetric frame comprising a first resolution pic­
ture of a first view of a scene and a reduced resolution picture
of a second view of the scene, wherein the reduced resolution
picture has a reduced resolution relative to the first resolution,
decoding the asymmetric frame, separating the decoded
asymmetric frame into the first resolution picture and the
reduced resolution picture, upsampling the reduced resolu­
tion picture to produce a second picture of the scene having
the first resolution, and outputting the first picture and the
second picture, wherein the first picture and the second pic­
ture form a stereo image pair.
[0013] In another example, an apparatus includes a video
decoder configured to receive an encoded asymmetric frame
US 2011/0280316 A1
2
Nov. 17,2011
comprising a first resolution picture of a first view of a scene
and a reduced resolution picture of a second view of the scene,
wherein the reduced resolution picture has a reduced resolu­
tion relative to the first resolution, decode the asymmetric
frame, separate the decoded asymmetric frame into the first
resolution picture and the reduced resolution picture, and
upsample the reduced resolution picture to produce a second
picture of the scene having the first resolution, wherein the
first decoded picture and the second decoded picture form a
stereo image pair.
[0014] In another example, an apparatus includes means for
receiving an asymmetric frame comprising a first resolution
picture of a first view of a scene and a reduced resolution
picture of a second view of the scene, wherein the reduced
resolution picture has a reduced resolution relative to the first
resolution, means for decoding the asymmetric frame, means
for separating the decoded asymmetric frame into the first
resolution picture and the reduced resolution picture, and
means for upsampling the reduced resolution picture to pro­
duce a second picture of the scene having the first resolution,
wherein the first decoded picture and the second decoded
picture form a stereo image pair.
[0015] In another example, a computer program product
includes a computer-readable storage medium having stored
thereon instructions that, when executed, cause a processor to
receive an asymmetric frame comprising a first resolution
picture of a first view of a scene and a reduced resolution
picture of a second view of the scene, wherein the reduced
resolution picture has a reduced resolution relative to the first
resolution, decode the asymmetric frame, separate the
decoded asymmetric frame into the first resolution picture
and the reduced resolution picture, upsample the reduced
resolution picture to produce a second picture of the scene
with the first resolution, and output the first picture and the
second picture, wherein the first picture and the second pic­
ture form a stereo image pair.
[0016] In another example, a method includes encoding a
first picture of a first view of a scene to produce an encoded
picture with a first resolution, encoding at least a portion of a
second picture of a second view of the scene relative to a
reference picture of the first view to produce an encoded
picture with a reduced resolution relative to the first resolu­
tion, and outputting the encoded first resolution picture and
the encoded reduced resolution picture in a common bit-
stream.
[0017] In another example, an apparatus includes a video
encoder configured to encode a first picture of a first view of
a scene to produce an encoded picture with a first resolution,
encode at least a portion of a second picture of a second view
of the scene relative to a reference picture of the first view to
produce an encoded picture with a reduced resolution relative
to the first resolution, and output the encoded first resolution
picture and the encoded reduced resolution picture in a com­
mon bitstream.
[0018] In another example, an apparatus includes means for
encoding a first picture of a first view of a scene to produce an
encoded picture with a first resolution, means for encoding at
least a portion of a second picture of a second view of the
scene relative to a reference picture of the first view to pro­
duce an encoded picture with a reduced resolution relative to
the first resolution, and means for outputting the encoded first
resolution picture and the encoded reduced resolution picture
in a common bitstream.
[0019] In another example, a computer program product
includes a computer-readable storage medium having stored
thereon instructions that, when executed, cause a processor to
encode a first picture of a first view of a scene to produce an
encoded picture with a first resolution, encode at least a
portion of a second picture of a second view of the scene
relative to a reference picture of the first view to produce an
encoded picture with a reduced resolution relative to the first
resolution, and output the encoded first resolution picture and
the encoded reduced resolution picture in a common bit-
stream.
[0020] In another example, a method includes receiving,
from a common bitstream, a first resolution encoded picture
of a first view of a scene and a reduced resolution encoded
picture of a second view of the scene, wherein the reduced
resolution encoded picture has a reduced resolution relative to
the first resolution, decoding the first resolution encoded pic­
ture to produce a first decoded picture, decoding at least a
portion of the reduced resolution encoded picture relative to a
reference picture of the first view, upsampling the reduced
resolution picture to produce a second decoded picture of the
scene with the first resolution, and outputting the first
decoded picture and the second decoded picture, wherein the
first decoded picture and the second decoded picture form a
stereo image pair.
[0021] In another example, an apparatus includes a video
decoder configured to receive, from a common bitstream, a
first resolution encoded picture of a first view of a scene and
a reduced resolution encoded picture of a second view of the
scene, wherein the reduced resolution encoded picture has a
reduced resolution relative to the first resolution, decode the
first resolution encoded picture to produce a first decoded
picture, decode at least a portion of the reduced resolution
encoded picture relative to a reference picture of the first view,
upsample the reduced resolution picture to produce a second
decoded picture of the scene with the first resolution, and
output the first decoded picture and the second decoded pic­
ture, wherein the first decoded picture and the second
decoded picture form a stereo image pair.
[0022] In another example, an apparatus includes means for
receiving, from a common bitstream, a first resolution
encoded picture of a first view of a scene and a reduced
resolution encoded picture of a second view of the scene,
wherein the reduced resolution encoded picture has a reduced
resolution relative to the first resolution, means for decoding
the first resolution encoded picture to produce a first decoded
picture, means for decoding at least a portion of the reduced
resolution encoded picture relative to a reference picture of
the first view, means for upsampling the reduced resolution
picture to produce a second decoded picture of the scene with
the first resolution, and means for outputting the first decoded
picture and the second decoded picture, wherein the first
decoded picture and the second decoded picture form a stereo
image pair.
[0023] In another example, a computer program product
includes a computer-readable storage medium having stored
thereon instructions that, when executed, cause a processor to
receive, from a common bitstream, a first resolution encoded
picture of a first view of a scene and a reduced resolution
encoded picture of a second view of the scene, decode the first
resolution encoded picture to produce a first decoded picture,
decode at least a portion of the reduced resolution encoded
picture relative to a reference picture of the first view,
upsample the reduced resolution picture to produce a second
US 2011/0280316 A1
3
Nov. 17,2011
decoded picture of the scene with the first resolution, and
output the first decoded picture and the second decoded pic­
ture, wherein the first decoded picture and the second
decoded picture form a stereo image pair.
[0024] The details of one or more examples are set forth in
the accompanying drawings and the description below. Other
features, objects, and advantages will be apparent from the
description and drawings, and from the claims.
BRIEF DESCRIPTION OF DRAWINGS
[0025] FIG. 1 is a block diagram illustrating an example
video encoding and decoding system that may utilize tech­
niques for forming asymmetric packed frames including pic­
tures from two corresponding views of a scene.
[0026] FIG. 2 is a block diagram illustrating an example of
a video encoder that may implement techniques for produc­
ing asymmetric packed frames.
[0027] FIG. 3 is a block diagram illustrating an example of
a video decoder, which decodes an encoded video sequence.
[0028] FIG. 4 is a conceptual diagram illustrating pictures
of a left eye view and a right eye view being combined by a
video encoder to form an asymmetric packed frame having a
top-bottom frame packing arrangement.
[0029] FIG. 5 is a conceptual diagram illustrating pictures
of a left eye view and a right eye view being combined by a
video encoder to form an asymmetric packed frame having a
side-by-side frame packing arrangement.
[0030] FIG. 6 is a conceptual diagram illustrating an
example process for forming an asymmetric packed frame
including a reduced resolution picture encoded as a field.
[0031] FIG. 7 is a conceptual diagram illustrating field
encoding of a picture to produce a reduced resolution
encoded picture for inclusion in an asymmetric packed frame.
[0032] FIG. 8 is a conceptual diagram illustrating inter­
view prediction of a block of a reduced resolution encoded
picture of an asymmetric packed frame.
[0033] FIG. 9 is a flowchart illustrating an example method
for encoding two pictures of two different views and combin­
ing the pictures to form an asymmetric packed frame.
[0034] FIG. 10 is a flowchart illustrating an example
method for decoding an asymmetric frame.
[0035] FIG. 11 is a flowchart illustrating an example
method for performing frame field interleaved coding in
accordance with the techniques of this disclosure.
[0036] FIG. 12 is a flowchart illustrating an example
method for decoding a frame field interleaved coded bit-
stream in accordance with the techniques of this disclosure.
DETAILED DESCRIPTION
[0037] In general, this disclosure relates to techniques for
supporting stereo video data, e.g., video data used to produce
a three-dimensional effect. To produce a three-dimensional
effect in video, two views of a scene, e.g., a left eye view and
a right eye view, are shown simultaneously or nearly simul­
taneously. Two pictures of the same scene, corresponding to
the left eye view and the right eye view of the scene, may be
captured from slightly different horizontal positions, repre­
senting the horizontal offset between a viewer’s left and right
eyes. By displaying these two pictures simultaneously or
nearly simultaneously, such that the left eye view picture is
perceived by the viewer’s left eye and the right eye view
picture is perceived by the viewer’s right eye, the viewer may
experience a three-dimensional video effect.
[0038] This disclosure provides techniques for forming a
bitstream including packed frames. A packed frame may cor­
respond to a single frame of video data having data for two
pictures corresponding to different views of a scene. In par­
ticular, the techniques of this disclosure include encoding a
packed frame having a full resolution picture of one view of a
scene and a reduced resolution picture of another view of the
scene. A packed frame including a full resolution picture of a
first view of a scene and a reduced resolution picture of a
second, different view of the scene may be referred to as an
asymmetric packed frame, or simply an asymmetric frame.
[0039] In general, the terms “picture” and “frame” may be
used interchangeably. This disclosure generally refers to a
picture as a sample of a view. This disclosure generally refers
to a frame as comprising one or more pictures, which is to be
coded as an access unit representing a specific time instance.
Accordingly, a frame may correspond to a sample of a view
(that is, a single picture) or, in the case of packed frames,
include samples from multiple views (that is, two or more
pictures).
[0040] As an example, two view pictures may be packed as
a frame with a top-bottom format. In this example, one view
picture may be arranged on top of the other. Each picture may
have the same width of w pixels. The full resolution picture
may have a height of h pixels, while the reduced resolution
picture may have a height of h/2 pixels. As another example,
two view pictures may be packed as a frame with a side-by-
side format. In this example, the two view pictures may be
arranged beside each other. Each picture may have the same
height of h pixels. The full resolution picture may have a
width of w pixels, while the reduced resolution picture may
have a width of w/2 pixels.
[0041] Forming asymmetric frames in this manner may
provide several advantages. For example, the same bitstream
may be sent to devices configured to present three-dimen­
sional video data and to devices that are limited to only
two-dimensional video data. The three-dimensional video
capable devices may separate the asymmetric frames into
constituent views, upsample the reduced resolution view, and
display the two views simultaneously or near simultaneously.
The two-dimensional video capable devices may remove the
reduced resolution view and display only the full resolution
view. In this manner, a video content provider, e.g., a net-
work-based server or broadcaster, need only form one bit-
stream, and devices with varying capabilities may each
receive the same bitstream. Moreover, the bitstream may
require less bandwidth than a bitstream comprising full reso­
lution pictures of each of two or more views, while introduc­
ing negligible subjective quality degradation.
[0042] Accordingly, the techniques of this disclosure may
support backwards compatibility with legacy devices that are
not capable of presenting three-dimensional video data.
Unlike devices that can receive and decode symmetric packed
frames, which include two sub-sampled pictures, devices
receiving asymmetric packed frames in accordance with the
techniques of this disclosure may receive a full resolution
picture and a reduced resolution picture. Accordingly, the
devices need not upsample a picture just to produce a two-
dimensional video presentation. Furthermore, a bitstream in
accordance with the techniques of this disclosure (e.g.,
including asymmetric packed frames) may consume less
bandwidth than a bitstream having two full resolution pic­
tures for three-dimensional video data.
US 2011/0280316 A1
4
Nov. 17,2011
[0043] In some examples, the reduced resolution frame
may be encoded with respect to a frame of the other view. That
is, an encoder may perform inter-view prediction for reduced
resolution pictures of asymmetric packed frames. This dis­
closure describes techniques for encoding the reduced reso­
lution pictures as fields and using displacement vectors to
inter-view encode the reduced resolution pictures. In this
manner, this disclosure also provides techniques for perform­
ing inter-view prediction for a reduced resolution picture of
an asymmetric packed frame. This disclosure further pro­
vides frame field interleaved coding techniques, in which
pictures of one view may be coded as frames, while pictures
of another view may be coded as fields, and the frame pictures
and field pictures of the two views may be interleaved in a
common bitstream. The pictures of each view may form
discrete, independent access units of the same bitstream.
[0044] This disclosure also provides techniques for signal­
ing a frame packing type at the network abstraction layer
(NAL), e.g., in supplemental enhancement information (SEI)
messages of NAL units. Network abstraction layer (NAL)
units may include and/or describe coded audio and video
data, e.g., using SEI messages. In the example ofH.264/AVC
(Advanced Video Coding), coded video segments are orga­
nized into NAL units, which provide a “network-friendly”
video representation addressing applications such as video
telephony, storage, broadcast, or streaming. NAL units can be
categorized as Video Coding Layer (VCL) NAL units and
non-VCL NAL units. VCL units may contain output from the
core compression engine and may include block, macrob­
lock, and/or slice level data. Other NAL units may be non-
VCL NAL units. In some examples, a coded picture in one
time instance, normally presented as a primary coded picture,
may be contained in an access unit, which may include one or
more NAL units.
[0045] In some examples, the techniques of this disclosure
may be applied to H.264/AVC codecs or codecs based on
advanced video coding (AVC), such as scalable video encod­
ing (SVC), multiview video coding (MVC), or other exten­
sions of H.264/AVC. Such codecs may be configured to rec­
ognize SEI messages when the SEI messages are associated
with an access unit, where the SEI message may be encapsu­
lated within the access unit in an ISO base media file format
or MPEG-2 Systems bitstream. The techniques may also be
applied to future coding standards, e.g., H.265/HEVC (high
efficiency video coding).
[0046] SEI messages may contain information that is not
necessary for decoding the coded pictures samples from VCL
NAL units, but may assist in processes related to decoding,
display, error resilience, and other purposes. SEI messages
may be contained in non-VCL NAL units. SEI messages are
the normative part of some standard specifications, and thus
are not always mandatory for standard compliant decoder
implementation. SEI messages may be sequence level SEI
messages orpicture level SEI messages. Some sequence level
information may be contained in SEI messages, such as seal-
ability information SEI messages in the example of SVC and
view scalability information SEI messages in MVC. These
example SEI messages may convey information on, e.g.,
extraction of operation points and characteristics of the
operation points.
[0047] H.264/AVC provides a frame packing SEI message,
which is a codec-level message indicating a frame packing
type for a frame including a two pictures, e.g., a left view and
a right view of a scene. In this manner, H.264/AVC supports
interleaving of two pictures of left view and right view into
one picture and coding such pictures into a video sequence.
The frame packing SEI message is described in “Information
technology—Coding of audio-visual objects—Part 10:
Advanced Video Coding, AMENDMENT 1: Constrained
baseline profile, stereo high profile and frame packing
arrangement SEI message,” N10703, MPEG of ISO/IEC
JTC1/SC29/WG11, Xian, China, October 2009, which is
incorporated into the most recent version of the H.264/AVC
standard.
[0048] In this SEI message, various types of frame packing
methods are supported for spatial interleaving of two frames.
The supported interleaving methods include checkerboard,
column interleaving, row interleaving, side-by-side, top-bot­
tom, and side-by-side with checkerboard upconversion. This
disclosure provides techniques for supporting additional
frame packing types, such as asymmetric frame packing
arrangements. In particular, this disclosure provides a modi­
fied frame packing SEI message that indicates whether asym­
metric packing is enabled for a particular frame, and if so,
whether the asymmetric frame is packed top-bottom or side-
by-side. For example, the frame packing SEI message may
indicate whether the pictures for the two views in the same
frame are arranged with the reduced resolution picture below
the full resolution picture or to the right of the full resolution
picture in the frame. A decoder may use this information to
determine whether the frame is an asymmetric frame and to
properly separate the asymmetric frame into constituent pic­
tures of the two views.
[0049] This disclosure includes techniques for signaling
whether a frame is an asymmetric packed frame in an SEI
message, in some examples, e.g., with respect to H.264/AVC.
As one example, an encoder may signal that a frame is an
asymmetric packed frame in an independent SEI message. As
another example, an encoder may signal that a frame is an
asymmetric packed frame in a modified version of the frame
packing arrangement SEI message. The encoder may also
signal, in video usability information (VUI), an aspect ratio
for the asymmetric packed frame to indicate a packing
arrangement for the asymmetric packed frame. For example,
the encoder may signal an aspect ratio of 4:3 (or one of the
unspecified values of Table E-l of the H.264/AVC specifica­
tion) to indicate a side-by-side packing arrangement. As
another example, the encoder may signal an aspect ratio of 3:4
(or, again, one of the unspecified values of Table E-l of the
H.264/AVC specification) to indicate a top-bottom packing
arrangement.
[0050] It should be understood that methods for sub-sam­
pling and up-sampling of the reduced resolution picture are
not limited to any particular techniques. For purposes of
example, this disclosure generally describes horizontal or
vertical down-sampling and upsampling. However, quincunx
(that is, checkerboard) sampling may also be used.
[0051] In addition, this disclosure provides techniques for
transferring a bitstream including asymmetric packed frames
over a high definition multimedia interface (HDMI). In this
manner, this disclosure provides techniques by which a three-
dimensional video interface, such as HDMI, may accept view
images with asymmetric packing in one or more frames.
[0052] FIG. 1 is a block diagram illustrating an example
video encoding and decoding system 10 that may utilize
techniques for forming asymmetric packed frames including
pictures from two corresponding views of a scene. As shown
in FIG. 1, system 10 includes a source device 12 that transmits
encoded video to a destination device 14 via a communication
channel 16. Source device 12 and destination device 14 may
comprise any of a wide range of devices, such as fixed or
mobile computing devices, set-top boxes, gaming consoles,
digital media players, orthe like. In some cases, source device
12 and destination device 14 may comprise wireless commu­
nication devices, such as wireless handsets, so-called cellular
or satellite radiotelephones, or any wireless devices that can
communicate video information over a communication chan­
nel 16, in which case communication channel 16 is wireless.
US 2011/0280316 A1
5
Nov. 17,2011
[0053] The techniques of this disclosure, however, which
concern forming asymmetric packed frames, are not neces­
sarily limited to wireless applications or settings. For
example, these techniques may apply to over-the-air televi­
sion broadcasts, cable television transmissions, satellite tele­
vision transmissions, Internet video transmissions, encoded
digital video that is encoded onto a storage medium, or other
scenarios. Accordingly, communication channel 16 may
comprise any combination of wireless or wired media suit­
able for transmission of encoded video data.
[0054] In the example of FIG. 1, source device 12 includes
a video source 18, video encoder 20, a modulator/demodula­
tor (modem) 22 and a transmitter 24. Destination device 14
includes a receiver 26, a modem 28, a video decoder 30, and
a display device 32. In accordance with this disclosure, video
encoder 20 of source device 12 may be configured to apply
the techniques for forming a bitstream including asymmetric
packed frames, e.g., frames including coded data for two
pictures, each from a different view of a scene, where one of
the pictures has full resolution and the other picture has a
reduced resolution, e.g., one-half of the resolution of the full
resolution frame. Moreover, video encoder 20 may be con­
figured to inter-view encode the reduced resolution frame. In
other examples, a source device and a destination device may
include other components or arrangements. For example,
source device 12 may receive video data from an external
video source 18, such as an external camera. Likewise, des­
tination device 14 may interface with an external display
device, rather than including an integrated display device.
[0055] The illustrated system 10 of FIG. 1 is merely one
example. Techniques for producing asymmetric packed
frames and splitting asymmetric packed frames into constitu­
ent views may be performed by any digital video encoding
and/or decoding device. Although generally the techniques of
this disclosure are performed by a video encoding device, the
techniques may also be performed by a video encoder/de­
coder, typically referred to as a “CODEC.” Moreover, aspects
of the techniques of this disclosure may also be performed by
a video preprocessor or video postprocessor, such as a file
encapsulation unit, file decapsulation unit, video multiplexer,
or video demultiplexer. Source device 12 and destination
device 14 are merely examples of such coding devices in
which source device 12 generates coded video data for trans­
mission to destination device 14. In some examples, devices
12, 14 may operate in a substantially symmetrical manner
such that each of devices 12, 14 include video encoding and
decoding components. Flence, system 10 may support one­
way or two-way video transmission between video devices
12, 14, e.g., for video streaming, video playback, video
broadcasting, video gaming, or video telephony.
[0056] Video source 18 of source device 12 may include a
video capture device, such as a video camera, a video archive
containing previously captured video, and/or a video feed
from a video content provider. As a further alternative, video
source 18 may generate computer graphics-based data as the
source video, or a combination of live video, archived video,
and computer-generated video. In some cases, if video source
18 is a video camera, source device 12 and destination device
14 may form so-called camera phones or video phones. As
mentioned above, however, the techniques described in this
disclosure may be applicable to video coding in general, and
may be applied to wireless and/or wired applications
executed by mobile or generally non-mobile computing
devices. In any case, the captured, pre-captured, or computer­
generated video may be encoded by video encoder 20.
[0057] Video source 18 may provide pictures from two or
more views to video encoder 20. Two pictures of the same
scene may be captured simultaneously or nearly simulta­
neously from slightly different horizontal positions, such that
the two pictures can be used to produce a three-dimensional
effect. Alternatively, video source 18 (or another unit of
source device 12) may use depth information or disparity
information to generate a second picture of a second view
from a first picture of a first view. The depth or disparity
information may be determined by a camera capturing the
first view, or may be calculated from data in the first view.
[0058] MPEG-C part-3 provides a specified format for
including a depth map for a picture in a video stream. The
specification is described in “Text of ISO/IEC FDIS 23002-3
Representation of Auxiliary Video and Supplemental Infor­
mation,” ISO/IEC JTC 1/SC 29/WG 11, MPEG Doc, N8768,
Marrakech, Morocoo, January 2007. In MPEG-C part 3, aux­
iliary video can be a depth map or a parallax map. When
representing a depth map, MPEG-C part-3 may provide flex­
ibilities, in terms of number of bits used to represent each
depth value and resolution of depth map. For example, the
map may be one-quarter of the width and one-half of the
height of the image described by the map. The map may be
coded as a monochromatic video sample, e.g., within an
F1.264/AVC bitstream with only the luminance component.
Alternatively, the map may be coded as auxiliary video data,
as defined in F1.264/AVC. In the context of this disclosure, a
depth map or a parallax map may have the same resolution as
the primary video data. Although the F1.264/AVC specifica­
tion does not currently specify the usage of auxiliary video
data to code depth map the techniques of this disclosure may
be used in conjunction with techniques for using such a depth
map or parallax map.
[0059] The encoded video information may then be modu­
lated by modem 22 according to a communication standard,
and transmitted to destination device 14 via transmitter 24.
Modem 22 may include various mixers, filters, amplifiers or
other components designed for signal modulation. Transmit­
ter 24 may include circuits designed for transmitting data,
including amplifiers, filters, and one or more antennas.
[0060] Receiver 26 of destination device 14 receives infor­
mation over channel 16, and modem 28 demodulates the
information. Again, the video encoding process may imple­
ment one or more of the techniques described herein to form
an asymmetric packed frame having a full resolution picture
of one view and a reduced resolution picture of another view.
[0061] The information communicated over channel 16
may include syntax information defined by video encoder 20,
which is also used by video decoder 30, that includes syntax
elements that describe characteristics and/or processing of
macroblocks and other coded units, e.g., GOPs. Accordingly,
video decoder 30 may unpack the asymmetric packed frame
into constituent pictures of the views, decode the pictures, and
upsample the reduced resolution picture to the full resolution.
Display device 32 may display the decoded pictures to a user.
[0062] Display device 32 may comprise any of a variety of
display devices such as a cathode ray tube (CRT), a liquid
crystal display (LCD), a plasma display, an organic light
emitting diode (OLED) display, or another type of display
device. Display device 32 may display the two pictures from
the asymmetric packed frame simultaneously or nearly
simultaneously. For example, display device 32 may com­
prise a stereoscopic three-dimensional display device capable
of displaying two views simultaneously or nearly simulta­
neously.
[0063] A user may wear active glasses to rapidly and alter­
natively shutter left and right lenses, such that display device
32 may rapidly switch between the left and the right view in
synchronization with the active glasses. Alternatively, display
device 32 may display the two views simultaneously, and the
user may wear passive glasses (e.g., with polarized lenses)
which filter the views to cause the proper views to pass
US 2011/0280316 A1
6
Nov. 17,2011
through to the user’s eyes. As still another example, display
device 32 may comprise an autostereoscopic display, for
which no glasses are needed.
[0064] In some examples, modem 28 and video decoder 30
may be included in separate devices. The separate devices
may be coupled by a high definition multimedia interface
(HDMI). This disclosure, in some examples, proposes modi­
fying HDMI to support transfer of asymmetric packed
frames. HDMI provides three-dimensional video formats in
Appendix H of version 1.4 of the HDMI specification, which
is available at. This specification supports various formats for
packing three-dimensional video data into one frame, e.g., in
the 3D_Structure field. In accordance with the techniques of
this disclosure, devices may exchange asymmetric packed
frames via HDMI, in addition to those packing arrangements
already provided by HDMI version 1.4.
[0065] As an example, the 3D_Structure field may include
a value indicating that a frame has a frame packing format,
which is similar to a top-bottom arrangement in H.264/AVC,
but without sub-sampling. There may be some blank area in a
frame having a frame packing format for HDMI. As another
example, the 3D_Structure field may include a value indicat­
ing that a frame has a field alternative format, which indicates
that a left-view image and a right-view image are fields of the
corresponding frame. As another example, the 3D_Structure
field may include a value indicating that a frame has a side-
by-side full format, indicating that the views are arranged
side-by-side and not sub-sampled.
[0066] As still another example, the 3D_Structure field
may include a value indicating that a frame has a side-by-side
half format, indicating that the views are sub-sampled with a
half horizontal resolution, and are arranged side-by-side.
When the side-by-side half format is enabled, subsampling
and position information may also be signaled, e.g., in a
3D_Ext_Data field. The frame may support two types of
sub-sampling: horizontal sub-sampling or quincunx (e.g.,
checkerboard) matrix. The position information may provide
data indicating a phase shift of the sub-sampled left and right
views. HDMI also supports texture image plus depth image
information, as well as video content with graphics represen­
tation.
[0067] As noted above, the techniques of this disclosure
include modifying HDMI to support asymmetric packed
frames. For example, in accordance with this disclosure, a
device may set a value for a 3D_Structure field of HDMI data
to indicate that a frame is an asymmetric packed frame. The
3D_Structure field may include a value indicating that a
frame includes a full resolution picture and a reduced resolu­
tion picture that form a stereo pair, and indicating that the
pictures are arranged side-by-side or top-bottom.
[0068] In the example of FIG. 1, communication channel
16 may comprise any wireless or wired communication
medium, such as a radio frequency (RF) spectrum or one or
more physical transmission lines, or any combination of wire­
less and wired media. Communication channel 16 may form
part of a packet-based network, such as a local area network,
a wide-area network, or a global network such as the Internet.
Communication channel 16 generally represents any suitable
communication medium, or collection of different commu­
nication media, for transmitting video data from source
device 12 to destination device 14, including any suitable
combination of wired or wireless media. Communication
channel 16 may include routers, switches, base stations, or
any other equipment that may be useful to facilitate commu­
nication from source device 12 to destination device 14.
[0069] Video encoder 20 and video decoder 3 0 may operate
according to a video compression standard, such as the ITU-T
H.264 standard, alternatively referred to as MPEG-4, Part 10,
Advanced Video Coding (AVC). The techniques of this dis­
closure, however, are not limited to any particular coding
standard. Other examples include MPEG-2 and ITU-T
H.263. Although not shown in FIG. 1, in some aspects, video
encoder 20 and video decoder 3 0 may each be integrated with
an audio encoder and decoder, and may include appropriate
MUX-DEMUX units, or other hardware and software, to
handle encoding of both audio and video in a common data
stream or separate data streams. If applicable, MUX-DE-
MUX units may conform to the ITU H.223 multiplexer pro­
tocol, or other protocols such as the user datagram protocol
(UDP).
[0070] The ITU-T H.264/MPEG-4 (AVC) standard was
formulated by the ITU-T Video Coding Experts Group
(VCEG) together with the ISO/IEC Moving Picture Experts
Group (MPEG) as the product of a collective partnership
known as the Joint Video Team (JVT). In some aspects, the
techniques described in this disclosure may be applied to
devices that generally conform to the H.264 standard. The
H.264 standard is described in ITU-T Recommendation
H.264, Advanced Video Coding for generic audiovisual ser­
vices, by the ITU-T Study Group, and dated March, 2005,
which may be referred to herein as the H.264 standard or
H.264 specification, or the H.264/AVC standard or specifica­
tion. The Joint Video Team (JVT) continues to work on exten­
sions to H.264/MPEG-4 AVC.
[0071] Video encoder 20 and video decoder 30 each may be
implemented as any of a variety of suitable encoder circuitry,
such as one or more microprocessors, digital signal proces­
sors (DSPs), application specific integrated circuits (ASICs),
field programmable gate arrays (FPGAs), discrete logic, soft­
ware, hardware, firmware or any combinations thereof. Each
of video encoder 20 and video decoder 30 may be included in
one or more encoders or decoders, either of which may be
integrated as part of a combined encoder/decoder (CODEC)
in a respective camera, computer, mobile device, subscriber
device, broadcast device, set-top box, server, or the like.
[0072] A video sequence typically includes a series of
video frames. A group of pictures (GOP) generally comprises
a series of one or more video frames. A GOP may include
syntax data in a header of the GOP, a header of one or more
frames of the GOP, or elsewhere, that describes a number of
frames included in the GOP. Each frame may include frame
syntax data that describes an encoding mode for the respec­
tive frame. Video encoder 20 typically operates on video
blocks within individual video frames in order to encode the
video data. A video block may correspond to a macroblock or
a partition of a macroblock. The video blocks may have fixed
or varying sizes, and may differ in size according to a speci­
fied coding standard. Each video frame may include a plural­
ity of slices. Each slice may include a plurality of macrob­
locks, which may be arranged into partitions, also referred to
as sub-blocks.
[0073] As an example, the ITU-T H.264 standard supports
intra prediction in various block sizes, such as 16 by 16, 8 by
8, or 4 by 4 for luma components, and 8x8 for chroma com­
ponents, as well as inter prediction in various block sizes,
such as 16x16, 16x8, 8x16, 8x8, 8x4, 4x8 and 4x4 for luma
components and corresponding scaled sizes for chroma com­
ponents. In this disclosure, “NxN” and “N by N” may be used
interchangeably to refer to the pixel dimensions of the block
in terms of vertical and horizontal dimensions, e.g., 16x16
pixels or 16 by 16 pixels. In general, a 16x16 block will have
16 pixels in a vertical direction (y=16) and 16 pixels in a
horizontal direction (x=16). Likewise, an NxN block gener­
ally has N pixels in a vertical direction and N pixels in a
US 2011/0280316 A1
7
Nov. 17,2011
horizontal direction, where N represents a nonnegative inte­
ger value. The pixels in a block may be arranged in rows and
columns. Moreover, blocks need not necessarily have the
same number of pixels in the horizontal direction as in the
vertical direction. For example, blocks may comprise NxM
pixels, where M is not necessarily equal to N.
[0074] Block sizes that are less than 16 by 16 may be
referred to as partitions of a 16 by 16 macroblock. Video
blocks may comprise blocks of pixel data in the pixel domain,
or blocks of transform coefficients in the transform domain,
e.g., following application of a transform such as a discrete
cosine transform (DCT), an integer transform, a wavelet
transform, or a conceptually similar transform to residual
video block data representing pixel differences between
coded video blocks and predictive video blocks. In some
cases, a video block may comprise blocks of quantized trans­
form coefficients in the transform domain.
[0075] Smaller video blocks can provide better resolution,
and may be used for locations of a video frame that include
high levels of detail. In general, macroblocks and the various
partitions, sometimes referred to as sub-blocks, may be con­
sidered video blocks. In addition, a slice may be considered to
be a plurality of video blocks, such as macroblocks and/or
sub-blocks. Each slice may be an independently decodable
unit of a video frame. Alternatively, frames themselves may
be decodable units, or other portions of a frame may be
defined as decodable units. The term “coded unit” or “coding
unit” may refer to any independently decodable unit of a
video frame such as an entire frame, a slice of a frame, a group
of pictures (GOP) also referred to as a sequence, or another
independently decodable unit defined according to applicable
coding techniques.
[0076] In accordance with the techniques of this disclosure,
video encoder 20 may form asymmetric packed frames from
received video data of two views. That is, video encoder 20
may receive raw image data of two views from, e.g., video
source 18. In general, the two views may include a sequence
of pictures, such that for each picture of one view, there exists
a picture of the other view that forms a stereo pair with the
picture of the first view. A stereo pair generally corresponds to
two pictures that, when displayed simultaneously or nearly
simultaneously, produce a three-dimensional video effect.
Pictures that form a stereo pair may include descriptive data,
such as timestamps, to indicate a corresponding picture of
another view with which a current picture forms a stereo pair.
[0077] In any case, video encoder 20 may encode a picture
of a first view normally, e.g., in accordance with ITU-T
F1.264/AVC encoding standards or with another encoding
standard such as MPEG-2, MPEG-4, El.265, or the like. Video
encoder 20, or a video preprocessing unit of source device 12
(which may comprise a processor, processing unit, ASIC,
DSP, FPGA, or other processing circuitry coupled between
video source 18 and video encoder 20), may spatially down-
sample a picture of a second view that forms a stereo pair with
the encoded picture of the first view. Spatial downsampling
may comprise reducing spatial resolutions, e.g., by reducing
vertical and/or horizontal pixel resolution. In one example,
video encoder 20 may reduce the vertical pixel resolution of
the picture by one-half.
[0078] Video encoder 20 may then encode the reduced
resolution picture of the other view. In some examples, video
encoder 20 may encode the reduced resolution picture in an
intra-prediction mode (e.g., as an I-Picture) or in an inter­
prediction mode (e.g., as a P-Picture or a B-Picture). In this
manner, video encoder 20 may encode the reduced resolution
picture relative to other pictures in the same view that occur
earlier (in decoding-time order) in a bitstream produced by
video encoder 20. In some examples, video encoder 20 may
implement inter-view prediction, in which video encoder 20
may encode the reduced resolution view relative to pictures of
the view including the full resolution picture. For example,
video encoder 20 may encode the reduced resolution view
picture relative to previously encoded pictures of the view
including the full resolution encoded picture. Video encoder
20 may encode the reduced resolution view picture relative to
the full resolution picture of the same packed frame, or of
previously coded frames.
[0079] As one example, video encoder 20 may encode the
reduced resolution picture as a field. Techniques for inter­
laced video data coding may be employed to encode the
reduced resolution picture as a field, in which case horizontal
rows of pixels of the reduced resolution picture may be pre­
dicted from alternate rows of pixels of a full resolution pic­
ture. That is, video encoder 20 may encode the reduced reso­
lution picture as either a top field or a bottom field. In some
examples, video encoder 20 may output the full resolution
picture of one view as an access unit and a corresponding
reduced resolution picture of a different view as a separate
access unit. Thus, video encoder 20 need not necessarily
combine the two pictures into an asymmetric frame to per­
form techniques for combining data for two views into a
single, common bitstream.
[0080] As another example, video encoder 20 may encode
the reduced resolution picture using displacement vectors.
The displacement vectors may be relative to reduced resolu­
tion pictures in the same view or full resolution pictures in the
view including full resolution pictures. When the displace­
ment vector refers to a full resolution picture, video encoder
20 may account for the position of the reduced resolution
picture in the asymmetric frame. Suppose, for example, that
the asymmetric packed frame includes the pictures in a top-
bottom arrangement with the reduced resolution picture
below the full resolution picture in the frame. Video encoder
20 may modify a vertical component of the displacement
vector by subtracting the height of the full resolution picture
from the vertical component and multiplying the resulting
difference by two, assuming that the reduced resolution pic­
ture has one-half the resolution of the full resolution picture.
[0081] Following intra-predictive or inter-predictive cod­
ing to produce predictive data and residual data, and follow­
ing any transforms (such as the 4x4 or 8x8 integer transform
used in F1.264/AVC or a discrete cosine transform DCT)
applied to residual data to produce transform coefficients,
quantization of transform coefficients may be performed.
Quantization generally refers to a process in which transform
coefficients are quantized to possibly reduce the amount of
data used to represent the coefficients. The quantization pro­
cess may reduce the bit depth associated with some or all of
the coefficients. For example, an n-bit value may be rounded
down to an m-bit value during quantization, where n is greater
than m.
[0082] Following quantization, entropy coding of the quan­
tized data may be performed, e.g., according to content adap­
tive variable length coding (CAVLC), context adaptive binary
arithmetic coding (CABAC), or another entropy coding
methodology. A processing unit configured for entropy cod­
ing, or another processing unit, may perform other processing
functions, such as zero run length coding of quantized coef-
US 2011/0280316 A1
8
Nov. 17,2011
ficients and/or generation of syntax information such as
coded block pattern (CBP) values, macroblock type, coding
mode, maximum macroblock size for a coded unit (such as a
frame, slice, macroblock, or sequence), or the like.
[0083] Video encoder 20 may further send syntax data,
such as block-based syntax data, frame-based syntax data,
and/or GOP-based syntax data, to video decoder 30, e.g., in a
frame header, a block header, a slice header, or a GOP header.
The GOP syntax data may describe a number of frames in the
respective GOP, and the frame syntax data may indicate an
encoding/prediction mode used to encode the corresponding
frame. Video decoder 30 may therefore comprise a standard
video decoder and need not necessarily be specially config­
ured to effect or utilize the techniques of this disclosure.
[0084] Video encoder20 and video decoder30 eachmay be
implemented as any of a variety of suitable encoder or
decoder circuitry, as applicable, such as one or more micro­
processors, digital signal processors (DSPs), application spe­
cific integrated circuits (ASICs), field programmable gate
arrays (FPGAs), discrete logic circuitry, software, hardware,
firmware or any combinations thereof. Each of video encoder
20 and video decoder 30 may be included in one or more
encoders or decoders, either of which may be integrated as
part of a combined video encoder/decoder (CODEC). An
apparatus including video encoder 20 and/or video decoder
30 may comprise an integrated circuit, a microprocessor, a
computing device, and/or a wireless communication device,
such as a mobile telephone.
[0085] Video decoder 30 may be configured to receive a
bitstream including asymmetric packed frames. Video
decoder 30 may further be configured to unpack such a frame
into corresponding pictures, e.g., a full resolution picture of
one view and a reduced resolution picture of another view.
Video decoder 30 may decode the pictures and upsample
(e.g., through interpolation) the reduced resolution picture to
produce two decoded, full resolution pictures. In some
examples, video decoder 30 may decode the reduced resolu­
tion picture with reference to a decoded picture from the view
corresponding to the full resolution picture. That is, video
decoder 30 may also support inter-view prediction.
[0086] In some examples, video decoder 30 may be con­
figured to determine whether destination device 14 is capable
of decoding and displaying three-dimensional data. If not,
video decoder 30 may unpack a received asymmetric packed
frame, but discard the reduced resolution picture. Video
decoder 30 may decode the full resolution picture and other
pictures of the same view, and cause video display 32 to
display the pictures from this view to present two-dimen­
sional video data. Thus, video decoder 30 may decode the full
resolution picture and provide the decoded full resolution
picture to display device 32, without attempting to decode the
reduced resolution picture.
[0087] In this manner, whether or not destination device 14
is capable of displaying three-dimensional video data, desti­
nation device 14 may receive a bitstream including asymmet­
ric packed frames. Thus, various destination devices with
various decoding and rendering capabilities may be config­
ured to receive the same bitstream from source device 20.
That is, some destination devices may be capable of decoding
and rendering three-dimensional video data while others may
not be capable of decoding and/or rendering three-dimen­
sional video data, yet each of the devices may be configured
to receive and use data from the same bitstream including
asymmetric packed frames.
[0088] FIG. 2 is a block diagram illustrating an example of
video encoder 20 that may implement techniques for produc­
ing asymmetric packed frames. Video encoder 20 may per­
form intra- and inter-coding of blocks within video frames,
including macroblocks, or partitions or sub-partitions of mac­
roblocks. Intra-coding relies on spatial prediction to reduce or
remove spatial redundancy in video within a given video
frame. Inter-coding relies on temporal prediction to reduce or
remove temporal redundancy in video within adj acent frames
of a video sequence. Intra-mode (I-mode) may refer to any of
several spatial based compression modes and inter-modes
such as uni-directional prediction (P-mode) or bi-directional
prediction (B-mode) may refer to any of several temporal-
based compression modes. Video encoder 20 may also, in
some examples, be configured to perform inter-view predic­
tion of reduced resolution pictures in an asymmetric packed
frame.
[0089] As shown in FIG. 2, video encoder 20 receives a
current video block within a video picture to be encoded. In
the example of FIG. 2, video encoder 20 includes motion
compensation unit 44, motion estimation unit 42, reference
frame store 64, summer 50, transform unit 52, quantization
unit 54, and entropy coding unit 56. For video block recon­
struction, video encoder 20 also includes inverse quantization
unit 58, inverse transform unit 60, and summer 62. A deblock­
ing filter (not shown in FIG. 2) may also be included to filter
block boundaries to remove blockiness artifacts from recon­
structed video. If desired, the deblocking filter would typi­
cally filter the output of summer 62.
[0090] During the encoding process, video encoder 20
receives a video picture or slice to be coded. The picture or
slice may be divided into multiple video blocks. Motion esti­
mation unit 42 and motion compensation unit 44 perform
inter-predictive coding of the received video block relative to
one or more blocks in one or more reference frames to provide
temporal compression. Intra prediction unit 46 may perform
intra-predictive coding of the received video block relative to
one or more neighboring blocks in the same frame or slice as
the block to be coded to provide spatial compression. Mode
select unit 40 may select one of the coding modes, intra or
inter, e.g., based on error results, and provides the resulting
intra- or inter-coded block to summer 50 to generate residual
block data and to summer 62 to reconstruct the encoded block
for use in a reference frame.
[0091] In particular, video encoder 20 may receive pictures
from two views forming a stereo view pair. The two views
may be referred to as view 0 and view 1. Without loss of
generality, assume that view 0 is a left eye view and view 1 is
a right eye view. It should be understood that the views may be
labeled differently, and that instead, view 1 may correspond
to the left eye view and view 0 may correspond to the right eye
view. In one example, video encoder 20 may encode pictures
of view 0 at a full resolution and pictures of view 1 at a
reduced resolution. Video encoder 20 may downsample pic­
tures of view 1 by a factor of one-half in the horizontal or the
vertical direction.
[0092] Video encoder 20 may further pack the encoded
pictures into an asymmetric packed frame. Assume, for
example, that video encoder 20 receives a view 0 picture and
a view 1 picture, each having a height of h pixels and a width
of w pixels, where w and h are non-negative, non-zero inte­
gers. Video encoder 20 may form a top-bottom arranged
asymmetric packed frame by downsampling the height of the
view 1 picture. For example, following downsampling and
US 2011/0280316 A1
9
Nov. 17,2011
encoding of the view 1 picture, the encoded, downsampled
view 1 picture may have a height of h/2 pixels and a width of
w pixels. Video encoder 20 may then form an asymmetric
packed frame including the encoded view 0 picture and the
encoded, downsampled view 1 picture below the encoded
view 0 picture, such that the asymmetric frame has a height of
3/2 h pixels and a width of w pixels.
[0093] As another example, video encoder 20 may form a
side-by-side arranged asymmetric packed frame by down-
sampling the view 1 picture. For example, following down-
sampling and encoding of the view 1 picture, the view 1
picture may have a width of w/2 pixels and a height of h
pixels. Video encoder 20 may then form an asymmetric
packed frame including the encoded view 0 picture and the
encoded, downsampled view 1 picture to the right of the
encoded view 0 picture, such that the asymmetric frame has a
height of h pixels and a width of 3/2 w pixels.
[0094] Video encoder 20 may further provide information
indicating a packing arrangement for an asymmetric packed
frame. The information may indicate whether the frame is an
asymmetric packed frame, and if so, whether the packing
arrangement is side-by-side or top-bottom. As one example,
video encoder 20 may provide this information in the form of
a frame packing arrangement SEI message. The frame pack­
ing arrangement SEI message may be defined according to
the example data structure of Table 1, below:
TABLE 1
frame Dackins; arransement SEI messase
frame_packing_arrangement( payloadSize ) { C Descriptor
frame_packing_arrangement_id 5 ue(v)
frame_packing_arrangement_cancel_flag
if( ! frame_packing_arrangement_cancel_flag ) {
5 u(l)
asymmetric_packing_idc 5 u(2)
frame_packing_arrangement_type 5 u(5)
quincunx_sampling_flag 5 u(l)
content_interpretation_type 5 u(6)
spatial_flipping_flag 5 u(l)
frameO_flipped_flag 5 u(l)
field_views_flag 5 u(l)
current_frame_is_frameO_flag 5 u(l)
frameO_self_contained_flag 5 u(l)
frame 1_self_contained_flag
If(! quincunx_sampling_flag &&
5 u(l)
frame_packing_arrangement_type !=5 ) {
frameO_grid_position_x 5 u(4)
frameO_grid_position_y 5 u(4)
framel_grid_position_x 5 u(4)
framel grid position y
}
frame_packing_arrangement_reserved_byte
5 u(4)
5 u(8)
frame packing_arrangement repetition period
}
frame packing arrangement extension flag
}
5 ue(v)
5 u(l)
[0095] The frame packing arrangement SEI message may
inform a video decoder, such as video decoder 30, that the
output decoded picture contains samples of a frame consist­
ing of multiple distinct spatially packed constituent frames
using an indicated frame packing arrangement scheme. In
accordance with the techniques of this disclosure, the frame
may comprise an asymmetric packed frame. The information
of the SEI message can be used by the decoder to rearrange
the samples and process the samples of the constituent frames
appropriately for display or other purposes. This SEI message
may be associated with pictures that are either frames or
fields. The frame packing arrangement of the samples may be
specified in terms of the sampling structure of a frame in order
to define a frame packing arrangement structure that is invari­
ant with respect to whether a picture is a single field of such a
packed frame or is a complete packed frame.
[0096] Video encoder 20 may set frame_packing_arrange-
ment_id to a value containing an identifying number that may
be used to identify the usage of the frame packing arrange­
ment SEI message. Video encoder 20 may set value of frame_
packing_arrangement_id in the range of 0 to 232-2, inclusive.
Values of frame_packing_arrangement_id from 0 to 255 and
from 512 to 231-1 may be used as determined by video
encoder 20. Values of frame_packing_arrangement_id from
256 to 511 and from 231 to 232-2 may be reserved for future
use by ITU-TIISO/IEC. Video decoders may ignore (e.g.,
remove from the bitstream and discard) all frame packing
arrangement SEI messages containing a value of frame_
packing_arrangement_id in the range of 256 to 511 or in the
range of 231 to 232-2.
[0097] Video encoder 20 may set the value of frame_pack-
ing_arrangement_cancel_flag equal to 1 to indicate that the
frame packing arrangement SEI message cancels the persis­
tence of any previous frame packing arrangement SEI mes­
sage in output order. Video encoder 20 may set the value of
frame_packing_arrangement_cancel_flag equal to 0 to indi­
cate that frame packing arrangement information follows.
[0098] Video encoder 20 may set the value of asymmetric_
packing_idc (asymmetric packing indicator) to indicate a
type of asymmetric coding. For example, video encoder 20
may set asymmetric_packing_idc to a value of 0 to indicate
that two constituent frames have the same resolution, that is,
that the corresponding frame is not an asymmetric packed
frame. Video encoder 20 may set the value of asymmetric_
packing_idc laiger than 0 (e.g., 1 or 2) to indicate that two
constituent frames have with different resolutions. For
example, one of the frames may be one-half of the other.
[0099] In one example, video encoder 20 may set the value
of asymmetric_packing_idc equal to 1 to indicate that two
constituent frames have different resolutions, and that frame
1 has a half resolution of frame 0. In one example, video
encoder 20 may set the value of asymmetric_packing_idc
equal to 2 to indicate that two constituent frames have differ­
ent resolutions, and that frame 0 has a half resolution of frame
1. The value 3 for asymmetric_packing_idc is currently
unspecified and reserved for future use. Table 2 below pro­
vides one example for interpreting the value of asymmetric_
packing_idc:
TABLE 2
asvmmetric oackinp idc
Value Example Interpretation
0 Frame 0 and frame 1 have the same resolution
1 Indicates frame 1 being half resolution of frame 0: when the
frame_packing_arrangement_type is 3, frame 1 has the same
height as frame 0 and frame 1 has a half width of frame 0;
when the frame_packing_arrangement_type is 4, frame
1 has the same width as frame 0 and frame 1 has a half
height of frame 0.
2 Indicates frame 0 being half resolution of frame 1: when the
frame_packing_arrangement_type is 3, frame 0 has the same
height as frame 1 and frame 0 has a half width of frame 1,
when the frame_packing_arrangement_type is 4,
frame 0 has the same width as frame 1 and frame 0 has a
half height of frame 1.
US 2011/0280316 A1
10
Nov. 17,2011
[0100] Video encoder 20 may set the value of frame_pack-
ing_arrangement_type to indicate the type of packing
arrangement of the frames as specified in Table 3, below.
When video encoder 20 sets the value of asymmetric_pack-
ing_idc to a value larger than 0 (e.g., 1 or 2), video encoder 20
may set the value of frame_packing_arrangement_type to
either 6, 7, 8, or 9.
TABLE 3
frame p ackin g arran gement typ e
Value Example Interpretation
0 Each component plane of the decoded frames contains a “checkerboard” based
interleaving of corresponding planes of two constituent frames.
1 Each component plane of the decoded frames contains a column based
interleaving of corresponding planes of two constituent frames.
2 Each component plane of the decoded frames contains a row based
interleaving of corresponding planes of two constituent frames.
3 Each component plane of the decoded frames contains a side-by-side packing
arrangement of corresponding planes of two constituent frames.
4 Each component plane of the decoded frames contains top-bottom packing
arrangement of corresponding planes of two constituent frames.
5 The component planes of the decoded frames in output order form a temporal
interleaving of alternating first and second constituent frames.
6, 7 Each component plane of the decoded frames contains side-by-side packing
arrangement of corresponding planes of two constituent frames as illustrated in
FIG. 5, wherein only the bottom frame needs upconversion (in this example).
Frame 0 and frame 1 have the same height. This value equal to 6 indicates that
frame 1 has a half width of frame 0; this value equal to 7 indicates that frame 0
has a half width of frame 1.
8, 9 Each component plane of the decoded frames contains a top-bottom packing
arrangement of corresponding planes of two constituent frames as illustrated in
FIG. 4, wherein only the right frame needs upconversion (in this example).
Frame 0 and frame 1 have the same width. This value equal to 8 indicates that
frame 1 has a half height of frame 0; this value equal to 9 indicates that
frame 0 has a half height of frame 1.
[0101] Video encoder 20 may set the value of quincunx_
sampling_flag equal to 1 to indicate that each color compo­
nent plane of each constituent frame is quincunx sampled.
Video encoder 20 may set the value of quincunx_sampling_
flag equal to 0 to indicate that the color component planes of
each constituent frame are not quincunx sampled. When
video encoder 20 sets the value of frame_packing_arrange-
ment_type is equal to 0, video encoder 20 may also set the
value of quincunx_sampling_flag equal to 1. When video
encoder 20 sets the value of frame_packing_arrangement_
type equal to 5, video encoder 20 may also set the value of
quincunx_sampling_flag equal to 0.
[0102] Video encoder 20 may set the value of content_
interpretation_type to indicate the intended interpretation of
the constituent frames as specified in Table 4. Values of con-
tent_interpretation_type that do not appear in Table 4 may be
reserved for future specification by ITU-TIISO/IEC. For each
specified frame packing arrangement scheme, there may be
two constituent frames (pictures), referred to in Table 4 as
frame 0 and frame 1.
TABLE 4
content interpretation type
Value Example Interpretation
0 Unspecified relationship between the frame packed constituent
frames
TABLE 4-continued
content interpretation tvne
Value Example Interpretation
1 Indicates that the two constituent frames form the left and right
views of a stereo view scene, with frame 0 being associated with
the left view and frame 1 being associated with the right view
2 Indicates that the two constituent frames form the right and left
views of a stereo view scene, with frame 0 being associated with
the right view and frame 1 being associated with the left view
[0103] Video encoder 20 may set the value of spatial_flip-
ping_flag equal to 1 when the value of frame_packing_ar-
rangement_type is equal to 3 or 4, to indicate that one of the
two constituent frames is spatially flipped relative to its
intended orientation for display or other such purposes. When
frame_packing_arrangement_type is equal to 3 or 4 and spa-
tial_flipping_flag is equal to 1, the type of spatial flipping that
is indicated may be as follows. If frame_packing_arrange-
ment_type is equal to 3, the indicated spatial flipping is hori­
zontal flipping. Otherwise (that is, when the value of frame_
packing_arrangement_type is equal to 4), the indicated
spatial flipping is vertical flipping.
[0104] When frame_packing_arrangement_type is not
equal to 3 or 4, video encoder 20 may set the value of spatial_
flipping_flag equal to 0. When frame_packing_arrangement_
type is not equal to 3 or 4, the value 1 for frame_packing_
arrangement_type may be reserved for future use by ITU-
US 2011/0280316 A1
11
Nov. 17,2011
TIISO/IEC. When frame_packing_arrangement_type is not
equal to 3 or 4, video decoders may ignore the value 1 for
spatial_flipping_flag.
[0105] Video encoder 20 may set the value of frame0_
flipped_flag equal to 1 to indicate which one of the two
constituent frames is flipped. When spatial_flipping_flag is
equal to 1, video encoder 20 may set the value of frame0_
flipped_flag equal to 0 to indicate that frame 0 is not spatially
flipped and frame 1 is spatially flipped, or video encoder 20
may set the value of frameO_flipped_flag equal to 1 to indi­
cate that frame 0 is spatially flipped and frame 1 is not spa­
tially flipped.
[0106] When video encoder 20 sets the value of spatial_
flipping_flag equal to 0, video encoder 20 may set the value of
frameO_flipped_flag equal to 0. When video encoder 20 sets
the value of spatial_flipping_flag is equal to 0, the value 1 for
spatial_flipping_flag may be reserved for future use by ITU-
TIISO/IEC. When spatial_flipping_flag is equal to 0, video
decoders may ignore the value of frameO_flipped_flag.
[0107] When video encoder 20 sets the value of quincunx_
sampling_flag equal to 0, video encoder 20 may provide
spatial location reference information to specify the location
of the upper left luma sample of each constituent frame rela­
tive to a spatial reference point. Video encoder 20 may indi­
cate the location of chroma samples relative to luma samples
by the chroma_sample_loc_type_top_field and chroma_
sample_loc_type_bottom_field syntax elements in video
usability information (VUI) parameters.
[0108] Video encoder 20 may set the value of field_views_
flag equal to 1 to indicate that all pictures in the current coded
video sequence are coded as complementary field pairs. All
fields of a particular parity may be considered a first constitu­
ent frame and all fields of the opposite parity may be consid­
ered a second constituent frame. When video encoder 20 does
not set the value of frame_packing_arrangement_type equal
to 2, video encoder 20 may set the value of field_views_flag
equal to 0. When video encoder 20 does not set the value of
frame_packing_arrangement_type equal to 2, the value 1 for
field_views_flag may be reserved for future use by ITU-
TIISO/IEC. When frame_packing_arrangement_type is not
equal to 2, video decoders may ignore the value of field_
views_flag.
[0109] Video encoder 20 may set the value of current_
frame_is_frameO_flag equal to 1, when frame_packing_ar-
rangement is equal to 5, to indicate that the current decoded
frame is constituent frame 0 and the next decoded frame in
output order is constituent frame 1, and the display time of the
constituent frame 0 should be delayed to coincide with the
display time of constituent frame 1. Accordingly, a video
decoder, such as video decoder 30, may delay the display time
of constituent frame 0 to coincide with the display time of
constituent frame 1. Video encoder 20 may set the value of
current_frame_is_frameO_flag equal to 0, when frame_pack-
ing_arrangement is equal to 5, to indicate that the current
decoded frame is constituent frame 1 and the previous
decoded frame in output order is constituent frame 0, and the
display time of the constituent frame 1 should not be delayed
for purposes of stereo-view pairing. Accordingly, a video
decoder, such as video decoder 3 0, need not delay the display
time of constituent frame 1 when the value of current_frame_
is_frame0_flag is equal to 0.
[0110] When video encoder 20 does not set the value of
frame_packing_arrangement_type equal to 5, the constituent
frame associated with the upper-left sample of the decoded
frame may be considered to be constituent frame 0 and the
other constituent frame may be considered to be constituent
frame 1. When frame_packing_arrangement_type is not
equal to 5 video encoder 20 may set the value of current_
frame_is_frameO_flag equal to 0. When frame_packing_ar-
rangement_type is not equal to 5, the value 1 for current_
frame_is_frameO_flag may be reserved for future use by ITU-
TIISO/IEC. When frame_packing_arrangement_type is not
equal to 5, decoders may ignore the value of current_frame_
is_frame0_flag.
[0111] Video encoder 20 may set the value of frame0_self_
contained_flag equal to 1 to indicate that no inter prediction
operations within the decoding process for the samples of
constituent frame 0 of the coded video sequence refer to
samples of any constituent frame 1. Video encoder 20 may set
the value of frameO_self_contained_flag equal to 0 to indicate
that some inter prediction operations within the decoding
process for the samples of constituent frame 0 of the coded
video sequence may or may not refer to samples of some
constituent frame 1. When frame_packing_arrangement_
type is equal to 0 or 1, video encoder 20 may set the value of
frameO_self_contained_flag equal to 0. When frame_packin-
g_arrangement_type is equal to 0 or 1, the value 1 for frame0_
self_contained_flag may be reserved for future use by ITU-
TIISO/IEC. When frame_packing_arrangement_type is
equal to 0 or 1, video decoders may ignore the value of
frameO_self_contained_flag. Within a coded video sequence,
video encoder 20 may set the value of frame0_self_con-
tained_flag in all frame packing arrangement SEI messages to
the same value.
[0112] Video encoder 20 may set the value of framel_self_
contained_flag equal to 1 to indicate that no inter prediction
operations within the decoding process for the samples of
constituent frame 1 of the coded video sequence refer to
samples of any constituent frame 0. Video encoder 20 may set
the value of framel_self_contained_flag equal to 0 to indicate
that some inter prediction operations within the decoding
process for the samples of constituent frame 1 of the coded
video sequence may or may not refer to samples of some
constituent frame 0. When frame_packing_arrangement_
type is equal to 0 or 1, it is a requirement of bitstream con­
formance that framel_self_contained_flag shall be equal to
0. When frame_packing_arrangement_type is equal to 0 or 1,
the value 1 for framel_self_contained_flag may be reserved
for future use by ITU-TIISO/IEC. When frame_packing_
arrangement_type is equal to 0 or 1, video decoders may
ignore the value of framel_self_contained_flag. Within a
coded video sequence, video encoder 20 may set the value of
framel_self_contained_flag in all frame packing arrange­
ment SEI messages to the same value.
[0113] When frameO_self_contained_flag is equal to 1 or
framel_self_contained_flag is equal to 1, and frame_packin-
g_arrangement_type is equal to 2, the decoded frame may be
a non-macroblock-level adaptive frame/field (MBAFF)
frame.
[0114] In some examples, video encoder 20 may set both
the value of frameO_self_contained_flag equal to 1 and
framel_self_contained_flag equal to 1. In this manner, video
encoder 20 may signal that the respective views can be
decoded and rendered separately.
[0115] Video encoder 20 may set the value of frame0_grid_
position_x (when present) to specify the horizontal location
of the upper left sample of constituent frame 0 to the right of
the spatial reference point in units of one sixteenth of the luma
US 2011/0280316 A1
12
Nov. 17,2011
sample grid spacing between the samples of the columns of
constituent frame 0 that are present in the decoded frame
(prior to any upsampling for display or other purposes).
[0116] Video encoder 20 may set the value of frame0_grid_
position_y (when present) to specify the vertical location of
the upper left sample of constituent frame 0 below the spatial
reference point in units of one sixteenth of the luma sample
grid spacing between the samples of the rows of constituent
frame 0 that are present in the decoded frame (prior to any
upsampling for display or other purposes).
[0117] Video encoder 20 may set the value of framel_grid_
position_x (when present) specifies the horizontal location of
the upper left sample of constituent frame 1 to the right of the
spatial reference point in units of one sixteenth of the luma
sample grid spacing between the samples of the columns of
constituent frame 1 that are present in the decoded frame
(prior to any upsampling for display or other purposes).
[0118] Video encoder 20 may set the value of framel_grid_
position_y (when present) specifies the vertical location of
the upper left sample of constituent frame 1 below the spatial
reference point in units of one sixteenth of the luma sample
grid spacing between the samples of the rows of constituent
frame 1 that are present in the decoded frame (prior to any
upsampling for display or other purposes).
[0119] Frame_packing_arrangement_reserved_byte may
be reserved for future use by ITU-TIISO/IEC. Video encoder
20 may set the value of frame_packing_arrangement_re-
served_byte equal to 0. All other values of frame_packing_
arrangement_reserved_byte may be reserved for future use
by ITU-TIISO/IEC. Video decoders may ignore (e.g., remove
from the bitstream and discard) the value of frame_packing_
arrangement_reserved_byte.
[0120] Video encoder 20 may set the value of frame_pack-
ing_arrangement_repetition_period to specify the persis­
tence of the frame packing arrangement SEI message, which
may specify a frame order count interval within which
another frame packing arrangement SEI message with the
same value of frame_packing_arrangement_id or the end of
the coded video sequence video encoder 20 has made present
in the bitstream. Video encoder 20 may set the value of frame_
packing_arrangement_repetition_period in the range of 0 to
16,384, inclusive.
[0121] Video encoder 20 may set the value of frame_pack-
ing_arrangement_repetition_period equal to 0 to specify that
the frame packing arrangement SEI message applies to the
current decoded frame only. Video encoder 20 may set the
value of frame_packing_arrangement_repetition_period
equal to 1 to specify that the frame packing arrangement SEI
message persists in output order until any of the following
conditions are true: a new coded video sequence begins, or a
frame in an access unit containing a frame packing arrange­
ment SEI message with the same value of frame_packing_
arrangement_id is output having PicOrderCnt() greater than
PicOrderCnt(CurrPic).
[0122] Video encoder 20 may set the value of frame_pack-
ing_arrangement_repetition_period equal to 0 or equal to 1 to
indicate that another frame packing arrangement SEI mes­
sage with the same value of frame_packing_arrangement_id
may or may not be present. Video encoder 20 may set the
value of frame_packing_arrangement_repetition_period
greater than 1 to specify that the frame packing arrangement
SEI message persists until any of the following conditions are
true: a new coded video sequence begins, or a frame in an
access unit containing a frame packing arrangement SEI mes­
sage with the same value of frame_packing_arrangement_id
is output having PicOrderCnt( ) greater than PicOrderCnt
(CurrPic) and less than or equal to PicOrderCnt(CurrPic)+
frame_packing_arrangement_repetition_period.
[0123] Video encoder 20 may set the value of frame_pack-
ing_arrangement_repetition_period greater than 1 to indicate
that another frame packing arrangement SEI message with
the same value of frame_packing_arrangement_frames_id is
present for a frame in an access unit that is output having
PicOrderCnt( ) greater than PicOrderCnt(CurrPic) and less
than or equal to PicOrderCnt(CurrPic)+frame_packing_ar-
rangement_repetition_period, unless the bitstream ends or a
new coded video sequence begins without output of such a
frame.
[0124] Video encoder 20 may set the value of frame_pack-
ing_arrangement_extension_flag equal to 0 to indicate that
no additional data follows within the frame packing arrange­
ment SEI message. In this case, video encoder 20 may set the
value of frame_packing_arrangement_extension_flag equal
to 0. The value 1 for frame_packing_arrangement_exten-
sion_flag may be reserved for future use by ITU-TIISO/IEC.
Video decoders may ignore the value 1 for frame_packing_
arrangement_extension_flag in a frame packing arrangement
SEI message and may ignore all data that follows within a
frame packing arrangement SEI message after the value 1 for
frame_packing_arrangement_extension_flag.
[0125] Mode select unit 40 may receive raw video data in
the form of blocks from the view 0 picture. After encoding the
view 0 picture, video encoder 20 may downsample a view 1
picture that corresponds to the view 0 picture. That is, the
view 0 picture and the view 1 picture may have been captured
at substantially the same time. After downsampling the view
1 picture, video encoder 20 may encode the view 1 picture.
Video encoder 20 may also store a decoded version of the
view 0 picture in reference frame store 64, such that motion
estimation unit 42 and motion compensation unit 44 may
perform inter-view prediction with respect to the view 0 pic­
ture when encoding the view 1 picture.
[0126] Motion estimation unit 42 and motion compensa­
tion unit 44 may be highly integrated, but are illustrated
separately for conceptual purposes. Motion estimation is the
process of generating motion vectors, which estimate motion
for video blocks. A motion vector, for example, may indicate
the displacement of a predictive block within a predictive
reference frame (or other coded unit) relative to the current
block being coded within the current frame (or other coded
unit). A predictive block is a block that is found to closely
match the block to be coded, in terms of pixel difference,
which may be determined by sum of absolute difference
(SAD), sum of square difference (SSD), or other difference
metrics. A motion vector may also indicate displacement of a
partition of a macroblock. Motion compensation may involve
fetching or generating the predictive block based on the
motion vector (or displacement vector) determined by motion
estimation unit 42. Again, motion estimation unit 42 and
motion compensation unit 44 may be functionally integrated,
in some examples.
[0127] Motion estimation unit 42 may calculate a motion
vector for a video block of an inter-coded picture by compar­
ing the video block to video blocks of a reference frame in
reference frame store 64. Motion compensation unit 44 may
also interpolate sub-integerpixels of the reference frame, e.g.,
an I-frame or a P-frame. The ITU-T H.264 standard refers to
“lists” of reference frames, e.g., list 0 and list 1. List 0
US 2011/0280316 A1
13
Nov. 17,2011
includes reference frames having a display order earlier than
the current picture, while list 1 includes reference frames
having a display order later than the current picture. Motion
estimation unit 42 compares blocks of one or more reference
frames from reference frame store 64 to a block to be encoded
of a current picture, e.g., a P-picture or a B-picture. When the
reference frames in reference frame store 64 include values
for sub-integer pixels, a motion vector calculated by motion
estimation unit 42 may refer to a sub-integer pixel location of
a reference frame. Motion estimation unit 42 sends the cal­
culated motion vector to entropy coding unit 56 and motion
compensation unit 44. The reference frame block identified
by a motion vector may be referred to as a predictive block.
Motion compensation unit 44 calculates residual error values
for the predictive block of the reference frame.
[0128] Motion estimation unit 42 may be configured to
perform inter-view prediction for view 1 pictures, in which
case motion estimation unit 42 may calculate displacement
vectors between blocks of the view 1 picture and correspond­
ing blocks of a reference frame of view 0. When calculating a
displacement vector, motion estimation unit 42 may set the
value of the motion vector relative to the position of the
current block in the reduced resolution picture separate from
the asymmetric frame, rather than a position of the current
block as positioned within the asymmetric packed frame.
[0129] Suppose, for example, that the position of the cur­
rent block in the reduced resolution picture is (x0, y0). Sup­
pose further that video encoder 20 will pack the asymmetric
frame with a top-bottom frame packing arrangement. The full
resolution picture may have a height of h pixels and a width of
w pixels. Accordingly, motion estimation unit 42 may calcu­
late the displacement vector relative to (x0, 2*(y0-h)). As
another example, suppose instead that video encoder 20 will
pack the asymmetric frame with a side-by-side frame packing
arrangement. In this example, motion estimation unit 42 may
calculate the displacement vector relative to (2*(x0-w), y0).
Thus, motion estimation unit 42 may calculate the displace­
ment vector relative to the position of the current block in the
reduced resolution frame standing alone, rather than the posi­
tion of the current block in the asymmetric frame. Motion
compensation unit 44 may calculate prediction data based on
the predictive block. Video encoder 20 forms a residual video
block, indicating residual error between the pixels values of
the block to be coded and the predictive block, by subtracting
the prediction data from motion compensation unit 44 from
the original video block being coded. Summer 50 represents
the component or components that perform this subtraction
operation.
[0130] Alternatively, video encoder 20 may be configured
to encode view 1 pictures as fields. Rather than encoding a
pair of interlaced top and bottom fields for view 1 pictures,
however, video encoder 20 may be configured to encode only
a single field for each of the view 1 pictures. Video encoder 20
may further encode the view 1 pictures as fields relative to
either fields of previously coded view 1 pictures or top or
bottom fields of view 0 pictures. Each of the previously coded
view 0 pictures may include both a top field and a bottom
field. It should be understood that although video encoder 20
may be configured to encode view 1 pictures as fields, video
encoder 20 may still encode view 0 pictures as frames.
[0131] To encode a picture of view 1 as a field, motion
estimation unit 42 may be configured to compare the picture
of view 1 to previously coded pictures of view 1 that have
been decoded, or top or bottom fields of previously coded
view 0 pictures that have been decoded. Field coded pictures
may comprise one-half of the vertical resolution of the full
resolution picture. In general, a field of a picture may com­
prise a top field, comprising even-numbered lines of the pic­
ture, or a bottom field, comprising odd-numbered lines of the
picture. Accordingly, to encode a view 1 picture relative to a
previously coded view 1 picture that is now decoded, video
encoder 20 may select a field for the view 1 picture (e.g., a top
field or a bottom field of the view 1 picture), select a previ­
ously coded view 1 picture that is now decoded as a reference
picture, and calculate the difference between the selected
field and the reference picture. Similarly, to encode a view 1
picture relative to a previously coded view 0 picture that is
now decoded, video encoder 20 may perform similar steps,
but additionally determine whether to encode the view 1
picture relative to a top field or a bottom field of the view 0
reference picture.
[0132] Transform unit 52 applies a transform, such as a
discrete cosine transform (DCT), integer transform, or a con­
ceptually similar transform, to the residual block, producing
a video block comprising residual transform coefficient val­
ues. Transform unit 52 may perform other transforms, such as
those defined by the F1.264 standard, which are conceptually
similar to DCT. Wavelet transforms, integer transforms, sub­
band transforms or other types of transforms could also be
used. In any case, transform unit 52 applies the transform to
the residual block, producing a block of residual transform
coefficients. Transform unit 52 may convert the residual
information from a pixel value domain to a transform domain,
such as a frequency domain. Quantization unit 54 quantizes
the residual transform coefficients to further reduce bit rate.
The quantization process may reduce the bit depth associated
with some or all of the coefficients. The degree of quantiza­
tion may be modified by adjusting a quantization parameter.
[0133] Following quantization, entropy coding unit 56
entropy codes the quantized transform coefficients. For
example, entropy coding unit 56 may perform content adap­
tive variable length coding (CAVLC), context adaptive binary
arithmetic coding (CABAC), or another entropy coding tech­
nique. Following the entropy coding by entropy coding unit
56, the encoded video may be transmitted to another device or
archived for later transmission or retrieval. In the case of
context adaptive binary arithmetic coding (CABAC), context
may be based on neighboring macroblocks.
[0134] In some cases, entropy coding unit 56 or another
unit of video encoder 20 may be configured to perform other
coding functions, in addition to entropy coding. For example,
entropy coding unit 56 may be configured to determine the
CBP values for the macroblocks and partitions. Also, in some
cases, entropy coding unit 56 may perform run length coding
of the coefficients in a macroblock or partition thereof. In
particular, entropy coding unit 56 may apply a zig-zag scan or
other scan pattern to scan the transform coefficients in a
macroblock or partition and encode runs of zeros for further
compression. Entropy coding unit 56 also may construct
header information with appropriate syntax elements for
transmission in the encoded video bitstream.
[0135] Inverse quantization unit 58 and inverse transform
unit 60 apply inverse quantization and inverse transforma­
tion, respectively, to reconstruct the residual block in the pixel
domain, e.g., for later use as a reference block. Motion com­
pensation unit 44 may calculate a reference block by adding
the residual block to a predictive block of one of the frames of
reference frame store 64. Motion compensation unit 44 may
US 2011/0280316 A1
14
Nov. 17,2011
also apply one or more interpolation filters to the recon­
structed residual block to calculate sub-integer pixel values
for use in motion estimation. Summer 62 adds the recon­
structed residual block to the motion compensated prediction
block produced by motion compensation unit 44 to produce a
reconstructed video block for storage in reference frame store
64. The reconstructed video block may be used by motion
estimation unit 42 and motion compensation unit 44 as a
reference block to inter-code a block in a subsequent video
frame.
[0136] FIG. 3 is a block diagram illustrating an example of
video decoder 30, which decodes an encoded video sequence.
In the example of FIG. 3, video decoder 30 includes an
entropy decoding unit 70, motion compensation unit 72, intra
prediction unit 74, inverse quantization unit 76, inverse trans­
formation unit 78, reference frame store 82 and summer 80.
Video decoder 30 may, in some examples, perform a decod­
ing pass generally reciprocal to the encoding pass described
with respect to video encoder 20 (FIG. 2).
[0137] In particular, video decoder 30 may be configured to
receive a bitstream including asymmetric packed frames.
Video decoder 30 may receive information indicative of
whether the bitstream includes asymmetric packed frames,
and if so, a frame packing arrangement for the asymmetric
packed frames. For example, video decoder 30 may be con­
figured to interpret frame packing arrangement SEI mes­
sages. Video decoder 30 may also be configured to determine
whether to decode both pictures in an asymmetric packed
frame, or only one of the two pictures, e.g., the full resolution
picture. This determination may be based on whether video
display 32 (FIG. 1) is able to display three-dimensional video
data, whether video decoder 30 has the capability to decode
two views (and upsample a reduced resolution view) of a
particular bitrate and/or framerate, or other factors regarding
video decoder 30 and/or video display 32.
[0138] When destination device 40 is not able to decode
and/or display three-dimensional video data from asymmet­
ric packed frames, video decoder 30 may unpack received
asymmetric frames into constituent full resolution encoded
pictures and reduced resolution encoded pictures, then dis­
card the reduced resolution encoded pictures. Thus, video
decoder 30 may elect to only decode the full-resolution pic­
tures of, e.g., view 0. On the other hand, when destination
device 40 is capable of decoding and displaying three-dimen­
sional video data of asymmetric packed frames, video
decoder 30 may unpack received asymmetric frames into
constituent full and reduced resolution encoded pictures,
decode the full and reduced resolution encoded pictures,
upsample the reduced resolution picture, and send the pic­
tures to video display 32. In some examples, video decoder 30
may receive asymmetric packed frames via F1DMI.
[0139] Video encoder 30 may further receive information
indicating whether a reduced resolution encoded picture of an
asymmetric frame is encoded as a field or as a picture. When
encoded as a picture, video encoder 30 may retrieve displace­
ment vectors for inter-view encoded reduced resolution pic­
tures, or motion vectors for intra-view, inter-prediction
encoded reduced resolution pictures. Video encoder 30 may
use the displacement or motion vectors to retrieve a predic­
tion block to decode a block of the reduced resolution picture.
After decoding the reduced resolution picture, video encoder
30 may upsample the decoded picture to the same resolution
as the full resolution picture of the same asymmetric frame.
[0140] Motion compensation unit 72 may generate predic­
tion data based on motion vectors received from entropy
decoding unit 70. Motion compensation unit 72 may use
motion vectors received in the bitstream to identify a predic­
tion block in reference frames in reference frame store 82.
Intra prediction unit 74 may use intra prediction modes
received in the bitstream to form a prediction block from
spatially adj acent blocks. Inverse quantization unit 76 inverse
quantizes, i.e., de-quantizes, the quantized block coefficients
provided in the bitstream and decoded by entropy decoding
unit 70. The inverse quantization process may include a con­
ventional process, e.g., as defined by the FI.264 decoding
standard. The inverse quantization process may also include
use of a quantization parameter QP^calculated by encoder 20
for each macroblock to determine a degree of quantization
and, likewise, a degree of inverse quantization that should be
applied.
[0141] Inverse transform unit 58 applies an inverse trans­
form, e.g., an inverse DCT, an inverse integer transform, or a
conceptually similar inverse transform process, to the trans­
form coefficients in order to produce residual blocks in the
pixel domain. Motion compensation unit 72 produces motion
compensated blocks, possibly performing interpolation
based on interpolation filters. Identifiers for interpolation fil­
ters to be used for motion estimation with sub-pixel precision
may be included in the syntax elements. Motion compensa­
tion unit 72 may use interpolation filters as used by video
encoder 20 during encoding of the video block to calculate
interpolated values for sub-integer pixels of a reference block.
Motion compensation unit 72 may determine the interpola­
tion filters used by video encoder 20 according to received
syntax information and use the interpolation filters to produce
predictive blocks.
[0142] Motion compensation unit 72 uses some of the syn­
tax information to determine sizes of macroblocks used to
encode frame(s) of the encoded video sequence, partition
information that describes how each macroblock of a frame of
the encoded video sequence is partitioned, modes indicating
how each partition is encoded, one or more reference frames
(or lists) for each inter-encoded macroblock or partition, and
other information to decode the encoded video sequence.
[0143] Summer 80 sums the residual blocks with the cor­
responding prediction blocks generated by motion compen­
sation unit 72 or intra-prediction unit to form decoded blocks.
If desired, a deblocking filter may also be applied to filter the
decoded blocks in order to remove blockiness artifacts. The
decoded video blocks are then stored in reference frame store
82, which provides reference blocks for subsequent motion
compensation and also produces decoded video for presenta­
tion on a display device (such as display device 32 of FIG. 1).
[0144] When a reduced resolution picture of an asymmetric
frame is encoded as a field, video encoder 30 may use a top
field or a bottom field of a previously decoded picture of the
other view as a reference field for decoding the reduced
resolution encoded picture. Video encoder 30 may also use a
previously decoded reduced resolution picture of the same
view as a reference field, where the previously decoded
reduced resolution pictures may be stored in reference frame
store 82 prior to upsampling. In this manner, video decoder 30
may decode the reduced resolution encoded picture relative to
a reduced resolution decoded picture of the same view, or
relative to a top or bottom field of a full resolution decoded
picture of the opposite view. After decoding the reduced
resolution picture, video decoder 30 may store the reduced
US 2011/0280316 A1
15
Nov. 17,2011
resolution decoded picture in reference frame store 82, then
upsample the reduced resolution decoded picture to form a
full resolution picture of the corresponding view.
[0145] FIG. 4 is a conceptual diagram illustrating pictures
100, 102 of a left eye view and a right eye view being com­
bined by video encoder 20 to form an asymmetric packed
frame 104. In this example, video encoder 20 receives picture
100, including raw video data of a left eye view of a scene, and
picture 102, including raw video data of a right eye view of the
scene. The left eye view may correspond to view 0, while the
right eye view may correspond to view 1. Pictures 100, 102
may correspond to two pictures of the same temporal
instance. For example, pictures 100,102 may have been cap­
tured by cameras at substantially the same time.
[0146] In the example of FIG. 4, samples of picture 100 are
indicated with X’s, while samples (e.g., pixels) of picture 102
are indicated with O’s. In this example, video encoder 20
encodes picture 100, downsamples and encodes picture 102,
and combines the pictures to form asymmetric packed frame
104. In this example, video encoder 20 arranges the full
resolution encoded picture for picture 100 and the reduced
resolution encoded picture for picture 102 in a top-bottom
arrangement within asymmetric packed frame 104. To down-
sample picture 102, video encoder 20 may decimate alternate
rows of picture 102. As another example, video encoder 20
may entirely remove alternate rows of picture 102 to produce
a downsampled version of picture 102. As still another
example, video encoder 20 may quincunx (checkerboard)
sample picture 102, and arrange these samples in rows within
asymmetric packed frame 104.
[0147] In the illustration of FIG. 4, asymmetric packed
frame 104 includes X’s corresponding to data from picture
100 and O’s corresponding to data from picture 102. Fiow-
ever, it should be understood that the data of asymmetric
packed frame 104 corresponding to picture 102 will not nec­
essarily align exactly with data of picture 102 following
downsampling. Likewise, following encoding, the data of the
pictures in asymmetric packed frame 104 will likely be dif­
ferent than the data of pictures 100, 102. Accordingly, it
should not be assumed that the data of one X in asymmetric
packed frame 104 is necessarily identical to a corresponding
X in picture 100. Similarly, it should not be assumed that the
data of one O in asymmetric packed frame 104 is identical to
a corresponding O in picture 102, or that the O’s of asymmet­
ric packed frame 104 have the same resolution as O’s of
picture 102.
[0148] Asymmetric packed frame 104 may correspond to a
top-bottom frame packing arrangement. That is, data corre­
sponding to picture 100 is placed on top of data corresponding
to picture 102 in asymmetric packed frame 104. Although
illustrated in rows, data corresponding to picture 102 in asym­
metric packed frame 104 may be quincunx (checkerboard)
sampled, and thus, may be upsampled using a quincunx
arrangement as well. Alternatively, data corresponding to
picture 102 in asymmetric packed frame 104 may be sampled
from alternate rows of picture 102, in which case the data may
be upsampled by, e.g., interpolating alternate rows of the data
following decoding.
[0149] FIG. 5 is a conceptual diagram illustrating pictures
100, 102 of a left eye view and a right eye view being com­
bined by video encoder 20 to form an asymmetric packed
frame 106. In this example, video encoder 20 receives picture
100, including raw video data of a left eye view of a scene, and
picture 102, including raw video data of a right eye view of the
scene. Pictures 100, 102 may correspond to two pictures of
the same temporal instance. For example, pictures 100, 102
may have been captured by cameras at substantially the same
time.
[0150] In the example of FIG. 5, samples of picture 100 are
indicated with X’s, while samples of picture 102 are indicated
with O’s. In this example, video encoder 20 encodes picture
100, downsamples and encodes picture 102, and combines
the pictures to form asymmetric packed frame 106. In this
example, video encoder 20 arranges the full resolution
encoded picture for picture 100 and the reduced resolution
encoded picture for picture 102 in a side-by-side arrangement
within asymmetric packed frame 106. To downsample picture
102, video encoder 20 may decimate alternate columns of
picture 102. Alternatively, video encoder 20 may entirely
remove alternate columns of picture 102 to produce a down-
sampled version of picture 102.
[0151] In the illustration of FIG. 5, asymmetric packed
frame 106 includes X’s corresponding to data from picture
100 and O’s corresponding to data from picture 102. Fiow-
ever, it should be understood that the data of asymmetric
packed frame 106 corresponding to picture 102 will not nec­
essarily align exactly with data of picture 102 following
downsampling. Likewise, following encoding, the data of the
pictures in asymmetric packed frame 106 will likely be dif­
ferent than the data of pictures 100, 102. Accordingly, it
should not be assumed that the data of one X in asymmetric
packed frame 106 is necessarily identical to a corresponding
X in picture 100. Similarly, it should not be assumed that the
data of one O in asymmetric packed frame 106 is identical to
a corresponding O in picture 102, orthatthe O’s of asymmet­
ric packed frame 106 have the same resolution as O’s of
picture 102.
[0152] Asymmetric packed frame 106 may correspond to a
side-by-side frame packing arrangement. That is, data corre­
sponding to picture 100 is arranged side-by-side with data
corresponding to picture 102. Although illustrated in col­
umns, data corresponding to picture 102 in asymmetric
packed frame 106 may be quincunx (checkerboard) sampled,
and thus, may be upsampled using a quincunx arrangement as
well. Alternatively, data corresponding to picture 102 in
asymmetric packed frame 106 may be sampled from alternate
columns of picture 102, in which case the data may be
upsampled by, e.g., interpolating alternate columns of the
data following decoding.
[0153] FIG. 6 is a conceptual diagram illustrating an
example process for encoding pictures 110A-110D (pictures
110) of a left eye view as frames, while encoding pictures
112A-112D (pictures 112) of a right eye view as fields. In this
example, pictures 110 correspond to a left eye view (e.g.,
view 0), while pictures 112 correspond to a right eye view
(e.g., view 1). In general, pictures 112 may comprise down-
sampled pictures of the right eye view. For example, a video
processing unit may decimate rows of incoming pictures of
the right eye view to produce pictures 112.
[0154] A video encoder, such as video encoder 20, or a
video preprocessing unit coupled to the video encoder, may
receive full resolution, unencoded pictures of the left eye
view and the right eye view. Video encoder 20 may reduce the
resolution of pictures of the right eye view by decimating the
pictures of the right eye view. In this manner, video encoder
20 may produce pictures 112 that have one-half the vertical
resolution of pictures 110, but the same horizontal resolution
(width) as pictures 110.
US 2011/0280316 A1
16
Nov. 17,2011
[0155] The video encoder may encode pictures 110 nor­
mally, that is, as frames. However, in this example, video
encoder 20 may encode pictures 112 as fields. Video encoder
20 may encode pictures 112 relative to previously encoded
(and subsequently decoded) pictures of the right eye view, or
previously encoded (and subsequently decoded) pictures of
the left eye view. For example, video encoder 20 may encode
picture 112s' relative to either the top field of one of pictures
110 or the bottom field of one of pictures 110. That is, video
encoder 20 may use the top field of one of pictures 110 as a
reference field to encode one of pictures 112, e.g., by calcu­
lating differences between rows of the field of the one of
pictures 112 and alternate rows (starting with a top row) of the
one of pictures 110. Alternatively, video encoder 20 may use
the bottom field of one of pictures 110 as a reference field, in
which case video encoder 20 may calculate differences
between rows of the one of pictures 112 and alternate rows
(starting with the row after the top row) of the one of pictures
110. In general, video encoder 20 may encode pictures 112
relative to previously coded pictures 112 and top and/or bot­
tom fields of previously coded pictures 110.
[0156] Video encoder 20 may form independent access
units from pictures 110 and pictures 112. Together, pictures
110A and 112A may form a stereo image pair. Likewise,
pictures HOB and 112B may form a stereo image pair, pic­
tures HOC and 112C may form a stereo image pair, and
pictures 110D and 112D may form a stereo image pair. How­
ever, rather than forming an asymmetric frame including two
images forming a stereo image pair, video encoder 20 may
form independent access units from each of pictures 110 and
112. Video encoder 20 may output pictures 110 and 112
alternately, as illustrated in the example of FIG. 6. This tech­
nique may be referred to as frame field interleaved coding.
Thus, video encoder 20 may form a bitstream including both
pictures coded as frames and pictures coded as fields, and the
field coded pictures may have reduced resolution relative to
the frame coded pictures. Moreover, the field coded pictures
may be coded relative to one or more of the frame or field
coded pictures that occurs earlier in the bitstream.
[0157] Frame field interleaved coding is one example for
allowing prediction of a reduced resolution picture from a full
resolution picture. By coding full resolution pictures as
frames and coding reduced resolution pictures as fields, a
relatively high coding and bitstream efficiency may be
achieved. Decoded pictures 110 may be treated as comple­
mentary field pairs and used as reference pictures when a
reduced resolution field, that is, one of pictures 112, is coded.
In some examples, each picture of one view (e.g., the left view
or view 0) may be coded as a frame, while each picture of the
other view (e.g., the right view or view 1) may be coded as a
field. Accordingly, the view including full resolution encoded
pictures may be referred to as a full resolution view or a high
resolution view, while the view including reduced resolution
encoded pictures may be referred to as a reduced resolution
view or a low resolution view.
[0158] This technique may be used as an extension of
H.264/AVC in some examples. In some examples, this tech­
nique may be used as an extension to future coding standards,
such as H.265, assuming these standards support both frame
and field coding. Thus, these techniques do not necessarily
require new coding tools at the block level.
[0159] FIG. 7 is a conceptual diagram illustrating field
encoding of a picture to produce a reduced resolution
encoded picture for inclusion in an asymmetric packed frame.
FIG. 7 illustrates picture 120 as a view 0 (e.g., left eye view)
reference picture and picture 122 as a view 1 (e.g., right eye
view) picture to be coded as a field. In this example, rows of
pixels of the view 0 reference picture corresponding to the top
field of picture 120 are illustrated with X’s, while rows of
pixels of picture 120 corresponding to the bottom field of
picture 120 are illustrated with O’s.
[0160] In this example, picture 122 is encoded as a field
relative to the top field of picture 120. Thus, rows of picture
122 may be predicted from the top field of picture 120. In
other words, an encoder may use the top field of picture 122
as a reference field. For each pixel in picture 122, the video
encoder may calculate the difference between the pixel and a
collocated pixel in the corresponding row of the top field of
picture 122. The video encoder may then encode an identifier
of picture 122, an indication that the top field of picture 122
was used to predict the encoded version of picture 122, and
the residual values (that is, the calculated differences between
picture 122 and the top field of picture 120) to encode picture
122. The video encoder may then output the encoded version
of picture 122, e.g., interleaved between two frame-coded
pictures of view 0, as shown in FIG. 6.
[0161] FIG. 8 is a conceptual diagram illustrating inter­
view prediction of a block 148 of a reduced resolution
encoded picture 144 of an asymmetric packed frame 140.
FIG. 8 illustrates two example asymmetric packed frames
130, 140. Asymmetric packed frame 130 includes full reso­
lution encoded picture 132, corresponding to a left eye view
(e.g., view 0), and reduced resolution encoded picture 134,
corresponding to a right eye view (e.g., view 1). Asymmetric
packed frame 140 includes full resolution encoded picture
142, corresponding to the left eye view, and reduced resolu­
tion encoded picture 144, corresponding to the right eye view.
[0162] Reduced resolution encoded picture 144 includes
block 148, which may be intra-view predicted, e.g., relative to
block 138 of reduced resolution encoded picture 134 of asym­
metric frame 130. The example of FIG. 8 illustrates motion
vector 154 that indicates a location of block 138 relative to
block 148. Alternatively, block 148 may be inter-view pre­
dicted relative to, e.g., block 146 of full resolution encoded
picture 142 of asymmetric packed frame 140 (as shown by
displacement vector 150) or block 136 of full resolution
encoded picture 132 of asymmetric packed frame 130 (as
shown by displacement vector 152).
[0163] Displacement vector 150 may indicate the location
of block 146 relative to block 148 in full resolution encoded
picture 142. Displacement vector 152 may indicate the loca­
tion of block 136 relative to block 148 in full resolution
encoded picture 132 of asymmetric frame 130. Displacement
vector 154 (which may be considered a motion vector) may
indicate the location of block 138 relative to block 148 in
reduced resolution encoded picture 134. In this manner, block
148 may be intra-view inter-frame encoded, inter-view intra­
frame encoded, or inter-view inter-frame encoded. Accord­
ingly, three encoding modes may exist: prediction of block
148 from the same view, (e.g., the right eye view) in different
frames as illustrated by the example of displacement vector
154, prediction of block 148 from the same frame in the other
view (e.g., the left eye view) as illustrated by the example of
displacement vector 150, and prediction of block 148 from a
different frame and the other view (e.g., the left eye view) as
illustrated by the example of displacement vector 152.
[0164] As noted above, a video encoder, such as video
encoder 20, may calculate displacement vectors 150, 152,
US 2011/0280316 A1
17
Nov. 17,2011
154 relative to the location of reference block 148 external to
asymmetric packed frame 140. That is, displacement vectors
150, 152, 154 may be calculated relative to the location of
block 148 as if picture 144 was not combined with picture
142, but was a separate picture. To do so, let the location of
block 148 within asymmetric frame 140 be identified at posi­
tion (x0, y0). Let full resolution picture 142 have a height of h
pixels and width of w pixels.
[0165] In one example, assuming that asymmetric packed
frame 140 has a top-bottom packing arrangement, as illus­
trated in the example of FIG. 8, picture 144 may have the
same width as full resolution picture 142 (that is, a width of w
pixels), but a height less than the height of full resolution
picture 142. For example, picture 144 may have a height of
h/2 pixels. In this example, displacement vectors 150, 152
may be calculated relative to location (x0, 2*(y0-h)). More
generally, if reduced resolution picture 144 has a height of
n*h/d, displacement vectors 150,152 maybe calculated rela­
tive to location (x0, (d/n)*(y0-h)) and displacement vector
154 may be calculated relative to location (x0, y0).
[0166] As another example, assuming that asymmetric
packed frame 140 has a side-by-side packing arrangement,
picture 144 may have the same height as full resolution pic­
ture 142 (that is, a height of h pixels), but a width less than the
width of full resolution picture 142. For example, picture 144
may have a width of w/2 pixels. In this example, displacement
vectors 150, 152 may be calculated relative to location (2*
(x0-w), y0). More generally, if reduced resolution picture 144
has a width of n*w/d, displacement vectors 150,152 may be
calculated relative to location ((d/n)*(x0-w), y0).
[0167] FIG. 9 is a flowchart illustrating an example method
for combining two pictures of two different views into an
asymmetric packed frame and encoding the asymmetric
packed frame. Although generally described with respect to
the example components of FIGS. 1 and 2, it should be
understood that other encoders, encoding units, and encoding
devices may be configured to perform the method of FIG. 9.
Moreover, the steps of the method of FIG. 9 need not neces­
sarily be performed in the order shown in FIG. 9, and addi­
tional or alternative steps may be performed.
[0168] In the example of FIG. 9, video encoder 20 first
receives a picture of a left eye view (160), e.g., view 0. Video
encoder 20 may also receive a picture of a right eye view, e.g.,
view 1, (162), such that the two received pictures form a
stereo image pair. The left eye view and the right eye view
may form a stereo view pair, also referred to as a complemen­
tary view pair. The received right eye view picture may cor­
respond to the same temporal location as the received left eye
view picture. That is, the left eye view picture and the right
eye view picture may have been captured or generated at
substantially the same time. Video encoder 20 may then
reduce the resolution of the right eye view picture (166). In
some examples, a preprocessing unit of video encoder 20 may
receive the pictures. In some examples, the video preprocess­
ing unit may be external to video encoder 20.
[0169] In the example of FIG. 9, video encoder 20 reduces
the resolution of the right eye view picture (164). For
example, video encoder 20 may subsample the received right
eye view picture (e.g., using row-wise, column-wise, or quin­
cunx (checkerboard) subsampling), decimate rows or col­
umns of the received right eye view picture, or otherwise
reduce the resolution of the received right eye view picture. In
some examples, video encoder 20 may produce a reduced
resolution picture having either half of the width or half of the
height of the full resolution picture of the left eye view. In
other examples including a video preprocessor, the video
preprocessor may be configured to reduce the resolution of
the right eye view picture.
[0170] Video encoder 20 may then form an asymmetric
frame including both the received left eye view picture and
the downsampled right eye view picture (166). For example,
video encoder 20 may form an asymmetric frame having a
top-bottom arrangement, assuming that the right eye view
picture has the same width as the left eye view picture. In
some examples, video encoder 20 may form an asymmetric
frame with a top-bottom arrangement in which the full reso­
lution picture is above the reduced resolution picture, e.g.,
where the left eye view picture is placed above the right eye
view picture with a reduced resolution. In other examples,
video encoder 20 may form an asymmetric frame with a
top-bottom arrangement in which the full resolution picture is
below the reduced resolution picture, e.g., where the left eye
view picture is placed below the right eye view picture with a
reduced resolution. In still other examples, e.g., where the
reduced resolution picture has the same height but a reduced
width relative to the full resolution picture, video encoder 20
may form an asymmetric frame with a side-by-side arrange­
ment, and the full resolution picture may be placed either to
the left or to the right of the reduced resolution picture.
[0171] Video encoder 20 may then encode the asymmetric
frame (168). In some examples, video encoder 20 may be
configured to encode the right eye view picture portion of the
asymmetric frame only relative to previously coded data of
the right eye view. Thus, video encoder 20 may encode the
reduced resolution picture in an intra-prediction (I-predic-
tion) mode, relative to other data of the same picture, or in an
inter-prediction (P-prediction or B-prediction) mode, relative
to data of one or more previously encoded pictures of the right
eye view.
[0172] In other examples, video encoder 20 may be config­
ured to encode the reduced resolution right eye view picture
portion of the asymmetric frame relative to either data of the
right eye view or of the left eye view. For example, video
encoder 20 may encode the reduced resolution right eye view
picture relative to the left eye view portion of the asymmetric
frame. Video encoder 20 may also encode the reduced reso­
lution right eye view portion of the asymmetric frame relative
to left eye view portions of previously encoded asymmetric
frames.
[0173] Video encoder 20 may encode the reduced resolu­
tion right eye view picture relative to either a picture of the
right eye view or relative to a picture of the left eye view of a
previously encoded asymmetric frame. Thus, video encoder
20 may encode each block of the right eye view portion of the
current asymmetric frame in an inter-mode relative to blocks
of previously encoded right eye view pictures, blocks of the
left eye view picture portion of the same asymmetric frame, or
blocks of previously encoded left eye view portions of previ­
ously encoded asymmetric frames. As noted above, to encode
the blocks of the current picture, video encoder 20 may cal­
culate displacement vectors relative to the location of the
block in the reduced resolution right eye view picture, rather
than to the location of the block positioned within the asym­
metric packed frame.
[0174] After encoding the asymmetric frame, video
encoder 20 may signal whether inter-view prediction is used
to encode the right eye view picture (170). For example, video
encoder 20 may generate a frame packing arrangement SEI
US 2011/0280316 A1
18
Nov. 17,2011
message that indicates both whether asymmetric packed
frames are present in a bitstream formed by video encoder 20,
and if so, whether any of the asymmetric packed frames
includes a reduced resolution picture encoded in an inter­
view prediction mode.
[0175] Video encoder 20 may also signal a frame packing
type for the asymmetric packed frame (172). For example,
video encoder 20 may include information in the frame pack­
ing arrangement SEI message discussed above indicating a
frame packing arrangement for the asymmetric packed frame,
e.g., side-by-side or top-bottom packing. Moreover, video
encoder 20 may include information indicating the relative
locations of the data for the full resolution picture and the data
for the reduced resolution picture, e.g., in the frame packing
arrangement SEI message.
[0176] Video encoder 20 may then output the asymmetric
frame (174). For example, video encoder 20, or a unit coupled
to video encoder 20, may store the asymmetric frame to a
computer-readable storage medium, broadcast the asymmet­
ric frame, transmit the asymmetric frame via network trans­
mission or network broadcast, or otherwise provide the
encoded video data. In some examples, video encoder 20, or
a unit coupled to video encoder 20, may output the asymmet­
ric frame via a high definition multimedia interface (F1DMI).
[0177] It should also be understood that video encoder 20
need not necessarily provide information indicating whether
a bitstream includes asymmetric packed frames, and frame
packing arrangements and indications of locations of full and
reduced resolution pictures in the frames, for each frame of
the bitstream. In some examples, video encoder 20 may pro­
vide a single set of information, e.g., a single frame packing
SEI message, for the entire bitstream indicating this informa­
tion for each frame of the bitstream. In some examples, video
encoder 20 may provide the information periodically, e.g.,
after each video fragment, group of pictures (GOP), video
segment, every certain number of frames, or at other periodic
intervals. Video encoder 20, or another unit associated with
video encoder 20, may also provide the frame packing
arrangement SEI message on demand in some examples, e.g.,
in response to a request from a client device for the frame
packing arrangement SEI message or a general request for
header data of the bitstream.
[0178] FIG. 10 is a flowchart illustrating an example
method for decoding an asymmetric frame. Although gener­
ally described with respect to the example components of
FIGS. 1 and 3, it should be understood that other decoders,
decoding units, and decoding devices may be configured to
perform the method of FIG. 10. Moreover, the steps of the
method of FIG. 10 need not necessarily be performed in the
order shown in FIG. 10, and additional or alternative steps
may be performed.
[0179] Initially, video decoder 30 may receive an asymmet­
ric frame (200). In some examples, video decoder 30, or a unit
coupled to video decoder 30, may receive the asymmetric
frame via a high definition multimedia interface (F1DMI).
Video decoder 30 may then determine a frame packing type
for the asymmetric frame (202). For example, video decoder
30 may receive a frame packing arrangement SEI message
indicating the frame packing type for the asymmetric frame
(e.g., top-bottom or side-by-side), as well as locations of a full
resolution picture and a reduced resolution picture in the
asymmetric frame. In some examples, video decoder 30 may
have previously received a frame packing arrangement SEI
message for the bitstream, prior to receiving the asymmetric
frame, in which case video decoder 30 may have determined
the frame packing type for frames of the bitstream (including
the most recently received asymmetric frame) prior to receiv­
ing the asymmetric frame.
[0180] Based on the frame packing type information, video
decoder 30 may decode the asymmetric frame (204). Video
decoder 30 may first decode the left eye view portion of the
asymmetric frame, followed by the right eye view portion of
the asymmetric frame. Video decoder 30 may determine the
locations of the left eye view and right eye view portions of
the asymmetric frame based on the frame packing type infor­
mation. In some examples, video decoder 30 may decode the
right eye view picture relative to a left eye view picture.
[0181] After decoding the asymmetric frame, video
decoder 30 may separate the decoded frame into constituent
pictures, e.g., the left eye view picture and the right eye view
picture (206). Video decoder 30 may store a copy of the left
eye view picture for reference to decode other left eye view
pictures and, in some examples, right eye view pictures.
Video decoder 30 may also store a copy of the decoded right
eye view picture, e.g., before upsampling, for use as a refer­
ence picture for decoding right eye view portions of subse­
quently received asymmetric frames.
[0182] Maintaining the example above, the right eye view
picture may also have a reduced resolution, although in other
examples the right eye view picture may have full resolution
and the left eye view picture may have reduced resolution.
Accordingly, video decoder 30 may upsample the right eye
view picture (208), e.g., by interpolating missing information
to form a full resolution version of the right eye view picture.
In this manner, video decoder 30 may form a right eye view
picture having the same resolution as the left eye view picture.
Video decoder 30 may then send the decoded left and right
eye view pictures to video display 32, which may display the
left and right eye view pictures simultaneously or nearly
simultaneously (212).
[0183] FIG. 11 is a flowchart illustrating an example
method for performing frame field interleaved coding in
accordance with the techniques of this disclosure. Although
generally described with respect to the example components
of FIGS. 1 and 2, it should be understood that other encoders,
encoding units, and encoding devices may be configured to
perform the method of FIG. 11. Moreover, the steps of the
method of FIG. 9 need not necessarily be performed in the
order shown in FIG. 11, and additional or alternative steps
may be performed.
[0184] Initially, video encoder 20 may receive a left eye
view picture, e.g., a picture of view 0 (220). Video encoder 20
may then encode the left eye view picture (222), e.g., as a
frame in either an intra- or an inter-prediction mode. Thus,
video encoder 20 may encode the left eye view picture rela­
tive to other data of the same picture, or relative to one or more
reference pictures of the left eye view.
[0185] Video encoder 20 may also receive a picture of a
right eye view, e.g., view 1, (224), such that the right eye view
picture and the left eye view picture form a stereo image pair.
The left eye view and the right eye view may form a stereo
view pair, also referred to as a complementary view pair. The
received right eye view picture may correspond to the same
temporal location as the received left eye view picture. That
is, the left eye view picture and the right eye view picture may
have been captured or generated at substantially the same
time. Video encoder 20 may then reduce the resolution of the
right eye view picture (226). In some examples, a video
US 2011/0280316 A1
19
Nov. 17,2011
preprocessing unit of video encoder 20 may receive the right
eye view picture and reduce the resolution of the right eye
view picture prior to encoding. In some examples, the video
preprocessing unit may be external to video encoder 20.
[0186] To reduce the resolution of the right eye view pic­
ture, video encoder 20 (or a video preprocessing unit) may
decimate the right eye view picture, in some examples. In this
manner, video encoder 20 may reduce the resolution of the
right eye view picture, in this example, which may have
one-half the vertical resolution of the left eye view picture.
[0187] Video encoder 20 may then encode the reduced
resolution picture of the right eye view picture based on a
picture of the left eye view (228). That is, video encoder 20
may use a previously coded left eye view picture as a refer­
ence picture for encoding the right eye view picture. Although
in some cases video encoder 20 may use the left eye view
picture encoded at step 222 as a reference picture for encod­
ing the right eye view picture, in general, video encoder 20
may use any previously encoded picture of the left eye view as
a reference picture. Thus, video encoder 20 is not limited to
using the left eye view picture encoded at step 222 as the
reference picture for encoding the right eye view picture. In
some examples, video encoder 20 may use a previously
encoded right eye view picture as the reference picture for
encoding the current right eye view picture. That is, video
encoder 20 may determine whether to use a previously
encoded left eye view picture or a previously encoded right
eye view picture as a reference picture for encoding the cur­
rent right eye view picture. Furthermore, in some examples,
video encoder 20 may select between intra and inter-mode
encode of the current right eye view picture.
[0188] Video encoder 20 may encode the right eye view
picture as a field. Accordingly, to encode the right eye view
picture, video encoder 20 may calculate the difference
between rows of the right eye view picture and alternate rows
of the referenced left eye view picture. In this manner, video
encoder 20 may encode the right eye view picture as a field
referring to either a top field or a bottom field of a previously
encoded left eye view picture.
[0189] Video encoder 20 may then output the encoded left
eye view picture (230) and the encoded right eye view picture
(232). In this example, video encoder 20 may output the
encoded pictures into the same bitstream as separate access
units, rather than forming an asymmetric packed frame. The
bitstream may therefore include full resolution encoded pic­
tures of the left eye view and reduced resolution encoded
pictures of the right eye view, where the left eye view pictures
are encoded as frames and the right eye view pictures are
encoded as fields. The bitstream may resemble the illustration
of FIG. 6, such that the bitstream is frame field interleaved
encoded.
[0190] FIG. 12 is a flowchart illustrating an example
method for decoding a frame field interleaved coded bit-
stream in accordance with the techniques of this disclosure.
Although generally described with respect to the example
components of FIGS. 1 and 3, it should be understood that
other decoders, decoding units, and decoding devices may be
configured to perform the method of FIG. 12. Moreover, the
steps of the method of FIG. 10 need not necessarily be per­
formed in the order shown in FIG. 12, and additional or
alternative steps may be performed.
[0191] Video decoder 30 may be configured to receive and
decode a frame field interleaved encoded bitstream. Accord­
ingly, video decoder 30 may receive an encoded picture of a
left eye view, e.g., view 0 (240). Video decoder 30 may then
decode the left eye view picture (242). Video decoder 30 may
also receive an encoded picture of a right eye view, e.g., view
1 (244). The left eye view and right eye view may form a
stereo view pair, also referred to as a complementary view
pair. In this example, the left eye view picture and the right eye
view picture may form independent access units, even though
the two pictures may correspond to the same temporal period.
For example, the two pictures may have been captured nearly
simultaneously, such that the two pictures form a stereo
image pair for producing a three-dimensional video play­
back.
[0192] Video decoder 30 may decode the right eye view
picture based on a previously decoded left eye view picture
(246). That is, video decoder 30 may use a left eye view
picture as a reference picture when decoding the right eye
view picture. Although the reference picture may comprise
the picture decoded at step 242, the reference picture may
generally comprise any previously decoded picture of the left
eye view. To decode the right eye view picture, video decoder
30 may add values of rows of the received, encoded right eye
view picture to alternate rows of the reference picture, e.g., a
top field or a bottom field of the reference picture. The bit-
stream may include information indicating a reference pic­
ture for the right eye view picture, as well as whether to use
the top field or the bottom field as the reference field for
decoding the right eye view picture. In other examples, video
decoder 3 0 may further be configured to determine whether to
decode the right eye view picture relative to a top field or a
bottom field of a left eye view picture, or relative to a previ­
ously decoded right eye view picture.
[0193] After decoding the right eye view picture, video
decoder 30 may upsample the decoded right eye view picture
(248). For example, video decoder 30 may be configured to
interpolate missing rows of information of the decoded right
eye view picture. Video decoder 30 may output the decoded
left eye view picture (250) and the decoded and upsampled
right eye view picture (252). For example, video decoder 30
may send the decoded pictures to a display, which may dis­
play the pictures simultaneously or nearly simultaneously.
[0194] In some examples, video decoder 30 may be
included within a device that is not capable of three-dimen­
sional video playback. In such examples, video decoder 30
may simply decode the left eye view pictures and skip (e.g.,
discard) the right eye view pictures. In this manner, devices
may be capable of receiving and decoding a frame field inter­
leaved encoded bitstream whether or not the devices are
capable of decoding and/or rendering three-dimensional
video data.
[0195] Although generally described with respect to a
video encoder and a video decoder, the techniques of this
disclosure may be implemented in other devices and coding
units. For example, the techniques for forming an asymmetric
packed frame may be performed by a transcoder configured to
receive two separate, complementary bitstreams and to
transcode the two bitstreams to form a single bitstream
including asymmetric packed frames. As another example,
the techniques for disassembling an asymmetric packed
frame may be performed by a transcoder configured to
receive a bitstream including asymmetric packed frames and
to produce two separate bitstreams corresponding to respec­
tive views of the asymmetric packed frame, each including
encoded video data for a respective view.
US 2011/0280316 A1
20
Nov. 17,2011
[0196] In one or more examples, the functions described
may be implemented in hardware, software, firmware, or any
combination thereof. If implemented in software, the func­
tions may be stored on or transmitted over as one or more
instructions or code on a computer-readable medium and
executed by a hardware-based processing unit. Computer-
readable media may include computer-readable storage
media, which corresponds to a tangible medium such as data
storage media, or communication media including any
medium that facilitates transfer of a computer program from
one place to another, e.g., according to a communication
protocol. In this manner, computer-readable media generally
may correspond to (1) tangible computer-readable storage
media which is non-transitory or (2) a communication
medium such as a signal or carrier wave. Data storage media
may be any available media that can be accessed by one or
more computers or one ormore processors to retrieve instruc­
tions, code and/or data structures for implementation of the
techniques described in this disclosure. A computer program
product may include a computer-readable medium.
[0197] By way of example, and not limitation, such com­
puter-readable storage media can comprise RAM, ROM,
EEPROM, CD-ROM or other optical disk storage, magnetic
disk storage, or other magnetic storage devices, flash
memory, or any other medium that can be used to store
desired program code in the form of instructions or data
structures and that can be accessed by a computer. Also, any
connection is properly termed a computer-readable medium.
For example, if instructions are transmitted from a website,
server, or other remote source using a coaxial cable, fiber
optic cable, twisted pair, digital subscriber line (DSL), or
wireless technologies such as infrared, radio, and microwave,
then the coaxial cable, fiber optic cable, twisted pair, DSL, or
wireless technologies such as infrared, radio, and microwave
are included in the definition of medium. It should be under­
stood, however, that computer-readable storage media and
data storage media do not include connections, carrier waves,
signals, or other transient media, but are instead directed to
non-transient, tangible storage media. Disk and disc, as used
herein, includes compact disc (CD), laser disc, optical disc,
digital versatile disc (DVD), floppy disk and blu-ray disc
where disks usually reproduce data magnetically, while discs
reproduce data optically with lasers. Combinations of the
above should also be included within the scope of computer-
readable media.
[0198] Instructions may be executed by one or more pro­
cessors, such as one or more digital signal processors (DSPs),
general purpose microprocessors, application specific inte­
grated circuits (ASICs), field programmable logic arrays (FP-
GAs), or other equivalent integrated or discrete logic cir­
cuitry. Accordingly, the term “processor,” as used herein may
refer to any of the foregoing structure or any other structure
suitable for implementation of the techniques described
herein. In addition, in some aspects, the functionality
described herein may be provided within dedicated hardware
and/or software modules configured for encoding and decod­
ing, or incorporated in a combined codec. Also, the tech­
niques could be fully implemented in one or more circuits or
logic elements.
[0199] The techniques of this disclosure may be imple­
mented in a wide variety of devices or apparatuses, including
a wireless handset, an integrated circuit (IC) or a set of ICs
(e.g., a chip set). Various components, modules, or units are
described in this disclosure to emphasize functional aspects
of devices configured to perform the disclosed techniques,
but do not necessarily require realization by different hard­
ware units. Rather, as described above, various units may be
combined in a codec hardware unit or provided by a collec­
tion of interoperative hardware units, including one or more
processors as described above, in conjunction with suitable
software and/or firmware.
[0200] Various examples have been described. These and
other examples are within the scope of the following claims.
1. A method of encoding video data, the method compris­
ing:
receiving a first picture of a first view of a scene having a
first resolution;
receiving a second picture of a second view of the scene
having a reduced resolution relative to the first resolu­
tion;
forming an asymmetric frame comprising the first resolu­
tion picture and the reduced resolution picture;
encoding the asymmetric frame; and
outputting the asymmetric frame.
2. The method of claim 1, wherein the reduced resolution
of the second picture is one-half of the first resolution of the
first picture, the method further comprising outputting infor­
mation indicating that the second picture in the asymmetric
frame comprises one-half of the first resolution of the first
picture.
3. The method of claim 1, wherein forming the asymmetric
frame comprises arranging the first picture and the second
picture in a side-by-side arrangement, the method further
comprising outputting information indicating that the asym­
metric frame is formed in the side-by-side arrangement and
that the first picture and the second picture have the same
height.
4. The method of claim 1, wherein asymmetric packed
pictures form a frame in a top-bottom arrangement, the
method further comprising outputting information indicating
that the asymmetric frame is formed in the top-bottom
arrangement and that the first picture and the second picture
have the same width.
5. The method of claim 1, further comprising outputting a
supplemental enhancement information (SEI) message com­
prising an asymmetric packing indicator value that indicates
that the asymmetric frame is an asymmetric frame and a
frame packing arrangement type value indicating that the first
picture and the second picture have one of a side-by-side
arrangement and a top-bottom arrangement, and a location of
the first picture relative to the second picture in the asymmet­
ric frame.
6. The method of claim 1, wherein outputting the encoded
asymmetric frame comprises outputting at least one of a file
conforming to ISO base media file format, a file conforming
to an extension of the ISO base media file format, data con­
forming to an MPEG-2 transport stream, header information
for an MPEG-2 transport stream, and data in accordance with
hypertext transfer protocol (HTTP) streaming format.
7. An apparatus for encoding video data, the apparatus
comprising a video encoder configured to receive a first pic­
ture of a first view of a scene having a first resolution, receive
a second picture of a second view of the scene having a
reduced resolution relative to the first resolution, form an
asymmetric frame comprising the first picture and the second
picture, and encode the asymmetric frame.
8. The apparatus of claim 7, wherein the reduced resolution
of the second picture is one-half of the first resolution of the
US 2011/0280316 A1
21
Nov. 17,2011
first picture, and wherein the video encoder is configured to
output information indicating that the second picture in the
asymmetric frame comprises one-half of the first resolution
of the first picture.
9. The apparatus of claim 7, wherein the video encoder is
configured to arrange the first picture and the second picture
in a side-by-side arrangement in the asymmetric frame, and to
generate information indicating that the asymmetric frame is
formed in the side-by-side arrangement and that the first
picture and the second picture have the same height.
10. The apparatus of claim 7, wherein the video encoder is
configured to arrange the first picture and the second picture
in a top-bottom arrangement in the asymmetric frame, and to
generate information indicating that the asymmetric frame is
formed in the top-bottom arrangement and that the first pic­
ture and the second picture have the same width.
11. The apparatus of claim 7, wherein the video encoder is
configured to generate a supplemental enhancement informa­
tion (SEI) message comprising an asymmetric packing indi­
cator value that indicates that the asymmetric frame is an
asymmetric frame and a frame packing arrangement type
value indicating that the first picture and the second picture
have one of a side-by-side arrangement and a top-bottom
arrangement, and a location of the first picture relative to the
second picture in the asymmetric frame.
12. The apparatus of claim 7, wherein the video encoder is
configured to output the encoded asymmetric frame as at least
one of a file conforming to ISO base media file format, a file
conforming to an extension of the ISO base media file format,
data conforming to an MPEG-2 transport stream, header
information for an MPEG-2 transport stream, and data in
accordance with hypertext transfer protocol (HTTP) stream­
ing format.
13. The apparatus of claim 7, wherein the apparatus com­
prises at least one of:
an integrated circuit;
a microprocessor; and
a wireless communication device that includes the video
encoder.
14. An apparatus for encoding video data, the apparatus
comprising:
means for receiving a first picture of a first view of a scene
having a first resolution;
means for receiving a second picture of a second view of
the scene having a reduced resolution relative to the first
resolution;
means for forming an asymmetric frame comprising the
first picture and the second picture; and
means for encoding the asymmetric frame.
15. The apparatus of claim 14, wherein the reduced reso­
lution of the second picture is one-half of the first resolution
of the first picture, further comprising means for outputting
information indicating that the second picture in the asym­
metric frame comprises one-half of the first resolution of the
first picture.
16. The apparatus of claim 14, wherein the means for
forming the asymmetric frame comprises means for arrang­
ing the first picture and the second picture in a side-by-side
arrangement in the asymmetric frame, further comprising
means for generating information indicating that the asym­
metric frame is formed in the side-by-side arrangement and
that the first picture and the second picture have the same
height.
17. The apparatus of claim 14, wherein the means for
forming the asymmetric frame comprises means for arrang­
ing the first picture and the second picture in a top-bottom
arrangement in the asymmetric frame, further comprising
means for generating information indicating that the asym­
metric frame is formed in the top-bottom arrangement and
that the first picture and the second picture have the same
width.
18. The apparatus of claim 14, further comprising means
for generating a supplemental enhancement information
(SEI) message comprising an asymmetric packing indicator
value that indicates that the asymmetric frame is an asymmet­
ric frame and a frame packing arrangement type value indi­
cating that the first picture and the second picture have one of
a side-by-side arrangement and a top-bottom arrangement,
and a location of the first picture relative to the second picture
in the asymmetric frame.
19. A computer program product comprising a computer-
readable storage medium having stored thereon instructions
that, when executed, cause a processor of a device for encod­
ing video data to:
receive a first picture of a first view of a scene having a first
resolution;
receive a second picture of a second view of the scene
having a reduced resolution relative to the first resolu­
tion;
form an asymmetric frame comprising the first picture and
the second picture;
encode the asymmetric frame; and
output the encoded asymmetric frame.
20. The computer program product of claim 19, wherein
the reduced resolution of the second picture is one-half of the
first resolution of the first picture, further comprising instruc­
tions that cause the processor to output information indicating
that the second picture in the asymmetric frame comprises
one-half of the first resolution of the first picture.
21. The computer program product of claim 19, wherein
the instructions that cause the processor to form the asym­
metric frame comprise instructions that cause the processor to
arrange the first picture and the second picture in a side-by-
side arrangement, further comprising instructions that cause
the processor to output information indicating that the asym­
metric frame is formed in the side-by-side arrangement and
that the first picture and the second picture have the same
height.
22. The computer program product of claim 19, wherein
the instructions that cause the processor to form the asym­
metric frame comprise instructions that cause the processor to
arrange the first picture and the second picture in a top-bottom
arrangement, further comprising instructions that cause the
processor to output information indicating that the asymmet­
ric frame is formed in the top-bottom arrangement and that
the first picture and the second picture have the same width.
23. The computer program product of claim 19, further
comprising instructions that cause the processor to output a
supplemental enhancement information (SEI) message com­
prising an asymmetric packing indicator value that indicates
that the asymmetric frame is an asymmetric frame and a
frame packing arrangement type value indicating that the first
picture and the second picture have one of a side-by-side
arrangement and a top-bottom arrangement, and a location of
the first picture relative to the second picture in the asymmet­
ric frame.
US 2011/0280316 A1
22
Nov. 17,2011
24. The computer program product of claim 19, wherein
the instructions that cause the processor to output the encoded
asymmetric frame comprise instructions that cause the pro­
cessor to output at least one of a file conforming to ISO base
media file format, a file conforming to an extension of the ISO
base media file format, data conforming to an MPEG-2 trans­
port stream, header information for an MPEG-2 transport
stream, and data in accordance with hypertext transfer proto­
col (HTTP) streaming format.
25. A method of decoding video data, the method compris­
ing:
receiving an encoded asymmetric frame comprising a first
resolution picture of a first view of a scene and a reduced
resolution picture of a second view of the scene, wherein
the reduced resolution picture has a reduced resolution
relative to the first resolution;
decoding the asymmetric frame;
separating the decoded asymmetric frame into the first
resolution picture and the reduced resolution picture;
upsampling the reduced resolution picture to produce a
second picture of the scene having the first resolution;
and
outputting the first picture and the second picture, wherein
the first picture and the second picture form a stereo
image pair.
26. The method of claim 25, further comprising receiving
information indicating that the asymmetric frame has one of
a top-bottom packing arrangement and a side-by-side pack­
ing arrangement.
27. The method of claim 26, wherein, when the informa­
tion indicates that the asymmetric frame has the top-bottom
packing arrangement, separating the decoded asymmetric
frame includes separating the asymmetric frame into a top
portion and a bottom portion, wherein the top portion corre­
sponds to the first resolution picture, and wherein the bottom
portion corresponds to the reduced resolution picture.
28. The method of claim 26, wherein when the information
indicates that the asymmetric frame has the side-by-side
packing arrangement, separating the decoded asymmetric
frame includes separating the asymmetric frame into a left
portion and a right portion, wherein the left portion corre­
sponds to the first resolution picture, and wherein the right
portion corresponds to the reduced resolution picture.
29. The method of claim 26, wherein receiving the infor­
mation comprises receiving a supplemental enhancement
information (SEI) message comprising an asymmetric pack­
ing indicator value that indicates that the asymmetric frame is
an asymmetric frame and a frame packing arrangement type
value indicating that the first resolution picture and the
reduced resolution picture have one of the side-by-side
arrangement and the top-bottom arrangement, and indicating
a location of the first resolution picture relative to the reduced
resolution picture in the asymmetric frame.
30. The method of claim 25, wherein receiving the asym­
metric frame comprises receiving the asymmetric frame via a
high definition multimedia interface (HDMI) configured to
receive asymmetric frames.
31. An apparatus for decoding video data, the apparatus
comprising a video decoder configured to receive an encoded
asymmetric frame comprising a first resolution picture of a
first view of a scene and a reduced resolution picture of a
second view of the scene, wherein the reduced resolution
picture has a reduced resolution relative to the first resolution,
decode the asymmetric frame, separate the decoded asym­
metric frame into the first resolution picture and the reduced
resolution picture, and upsample the reduced resolution pic­
ture to produce a second picture of the scene having the first
resolution, wherein the first decoded picture and the second
decoded picture form a stereo image pair.
32. The apparatus of claim 31, further comprising a three-
dimensional video display configured to display the first
decoded picture and the second decoded picture at substan­
tially the same time to display three-dimensional video data.
33. The apparatus of claim 31, wherein the video decoder is
configured to receive information indicating that the asym­
metric frame has one of a top-bottom packing arrangement
and a side-by-side packing arrangement.
34. The apparatus of claim 33, wherein the video decoder is
configured to, when the information indicates that the asym­
metric frame has the top-bottom packing arrangement, sepa­
rate the decoded asymmetric frame into a top portion and a
bottom portion, wherein the top portion corresponds to the
first resolution picture, and wherein the bottom portion cor­
responds to the reduced resolution picture.
35. The apparatus of claim 33, wherein the video decoder is
configured to, when the information indicates that the asym­
metric frame has the side-by-side packing arrangement, sepa­
rate the decoded asymmetric frame into a left portion and a
right portion, wherein the left portion corresponds to the first
resolution picture, and wherein the right portion corresponds
to the reduced resolution picture.
36. The apparatus of claim 33, wherein the information
comprises a supplemental enhancement information (SEI)
message comprising an asymmetric packing indicator value
that indicates that the asymmetric frame is an asymmetric
frame and a frame packing arrangement type value indicating
that the first resolution picture and the reduced resolution
picture have one of the side-by-side arrangement and the
top-bottom arrangement, and indicating a location of the first
resolution picture relative to the reduced resolution picture in
the asymmetric frame.
37. The apparatus of claim 31, further comprising a high
definition multimedia interface (HDMI) configured to
receive asymmetric frames and configured to provide the
asymmetric frames to the video decoder.
38. The apparatus of claim 31, wherein the apparatus com­
prises at least one of:
an integrated circuit;
a microprocessor; and
a wireless communication device that includes the video
decoder.
39. An apparatus for decoding video data, the apparatus
comprising:
means for receiving an asymmetric frame comprising a
first resolution picture of a first view of a scene and a
reduced resolution picture of a second view of the scene,
wherein the reduced resolution picture has a reduced
resolution relative to the first resolution;
means for decoding the asymmetric frame;
means for separating the decoded asymmetric frame into
the first resolution picture and the reduced resolution
picture; and
means for upsampling the reduced resolution picture to
produce a second picture of the scene having the first
resolution, wherein the first decoded picture and the
second decoded picture form a stereo image pair.
40. The apparatus of claim 39, further comprising means
for receiving information indicating that the asymmetric
US 2011/0280316 A1
23
Nov. 17,2011
frame has one of a top-bottom packing arrangement and a
side-by-side packing arrangement.
41. The apparatus of claim 40, wherein the means for
separating comprises means for separating the asymmetric
frame into a top portion and a bottom portion, wherein the top
portion corresponds to the first resolution picture, and
wherein the bottom portion corresponds to the reduced reso­
lution picture, when the information indicates that the asym­
metric frame has the top-bottom packing arrangement.
42. The apparatus of claim 40, wherein the means for
separating comprises means for separating the asymmetric
frame into a left portion and a right portion, wherein the left
portion corresponds to the first resolution picture, and
wherein the right portion corresponds to the reduced resolu­
tion picture, when the information indicates that the asym­
metric frame has the side-by-side packing arrangement.
43. The apparatus of claim 40, wherein the information
comprises a supplemental enhancement information (SEI)
message comprising an asymmetric packing indicator value
that indicates that the asymmetric frame is an asymmetric
frame and a frame packing arrangement type value indicating
that the first resolution picture and the reduced resolution
picture have one of the side-by-side arrangement and the
top-bottom arrangement, and indicating a location of the first
resolution picture relative to the reduced resolution picture in
the asymmetric frame.
44. The apparatus of claim 39, wherein the means for
receiving the asymmetric frame comprises means for receiv­
ing the asymmetric frame via a high definition multimedia
interface (HDMI) configured to receive asymmetric frames.
45. A computer program product comprising a computer-
readable storage medium having stored thereon instructions
that, when executed, cause a processor of a device for decod­
ing video data to:
receive an asymmetric frame comprising a first resolution
picture of a first view of a scene and a reduced resolution
picture of a second view of the scene, wherein the
reduced resolution picture has a reduced resolution rela­
tive to the first resolution;
decode the asymmetric frame;
separate the decoded asymmetric frame into the first reso­
lution picture and the reduced resolution picture;
upsample the reduced resolution picture to produce a sec­
ond picture of the scene with the first resolution; and
output the first picture and the second picture, wherein the
first picture and the second picture form a stereo image
pair.
46. The computer program product of claim 45, further
comprising instructions that cause the processor to receive
information indicating that the asymmetric frame has one of
a top-bottom packing arrangement and a side-by-side pack­
ing arrangement.
47. The computer program product of claim 46, wherein
when the information indicates that the asymmetric frame has
the top-bottom packing arrangement, the instructions cause
the processor to separate the asymmetric frame into a top
portion and a bottom portion, wherein the top portion corre­
sponds to the first resolution picture, and wherein the bottom
portion corresponds to the reduced resolution picture.
48. The computer program product of claim 46, wherein
when the information indicates that the asymmetric frame has
the side-by-side packing arrangement, the instructions cause
the processor to separate the asymmetric frame into a left
portion and a right portion, wherein the left portion corre­
sponds to the first resolution picture, and wherein the right
portion corresponds to the reduced resolution picture.
49. The computer program product of claim 46, wherein
receiving the information comprises receiving a supplemen­
tal enhancement information (SEI) message comprising an
asymmetric packing indicator value that indicates that the
asymmetric frame is an asymmetric frame and a frame pack­
ing arrangement type value indicating that the first resolution
picture and the reduced resolution picture have one of the
side-by-side arrangement and the top-bottom arrangement,
and indicating a location of the first resolution picture relative
to the reduced resolution picture in the asymmetric frame.
50. The computer program product of claim 45, wherein
the instructions that cause the processor to receive the asym­
metric frame comprise instructions that cause the processor to
receive the asymmetric frame via a high definition multime­
dia interface (HDMI) configured to receive asymmetric
frames.
US005835636A
United States Patent [w]
Auld
[ii] Patent Number: 5,835,636
[45] Date of Patent: Nov. 10, 1998
[54] METHOD AND APPARATUS FOR
REDUCING THE MEMORY REQUIRED FOR
DECODING BIDIRECTIONALLY
PREDICTIVE-CODED FRAMES DURING
PULL-DOWN
[75] Inventor: David R. Auld, San Jose, Calif.
[73] Assignee: LSI Logic Corporation, Milpitas,
Calif.
[21] Appl. No.: 653,845
[22] Filed: May 28, 1996
[51]
[52]
[58]
lilt. C l. ............................
U.S. Cl...............................
Field of Search ...............
. G06K 9/36; G06K 9/46
........... 382/233; 348/714
.................. 382/232, 233,
382/236; 348/446, 714, 715, 716, 718,
719; 395/507, 509, 512, 521, 432, 433
[56] References Cited
U.S. PATENT DOCUMENTS
5,646,693 7/1997 Cismas ..................................... 348/402
Primary Examiner—Phuoc Tran
Attorney, Agent, or Firm—Conley, Rose & Tayon, PC; B.
Noel Kivlin
[57] ABSTRACT
A video decoder system for reconstructing, storing and
retrieving bidirectionally predictive-coded (B) frames for
display including pull-down conversion includes a recon­
struction unit for reconstructing the frames, where the recon­
struction unit reconstructs the top-upper field of every other
frame twice. The frame is conceptually divided into four
sections, including top-upper, top-lower, bottom-upper and
bottom-lower sections. A memory having only three seg­
ments for storing pixel data is provided, where each segment
is sized to store any one of the frame sections. A segmentor
receives and separates the pixel data according to the top and
bottom fields for each section of each frame, and stores pixel
data from the top field into one segment pixel data from the
bottom field into another segment. The segrnentor initially
selects any two segments for the upper half of the first frame,
and then selects a segment being retrieved for display and
the third segment for the bottom half of the first frame.
Thereafter, the segmentor selects the segment currently
being retrieved for display and whichever segment contains
pixel data that has already been retrieved and that will not be
re-displayed. The segmentor also selects one segment for
receiving the upper portion of the top field being recon­
structed again for pull-down, which is that segment cur­
rently being retrieved for display. The decoder system also
includes retrieval circuitry for retrieving pixel data from one
of said three segments at a time for interlaced display.
14 Claims, 7 Drawing Sheets
( START
____ 4 ____
Reconstruct upper half of frame
into 1st, 2nd segments
702
*
Retrieve top field for
display
704
Reconstruct lower half of
frame into 1st, 3rd segments
706
+
Retrieve bottom field for
display
708
Reconstruct upper half of top
field into 2nd segment
710
i
Pulldown and retrieve top
field again for display
712
Reconstruct upper half of
frame into 1st, 2nd segments
714
Retneve top field for
display
Z16
Reconstruct lower half of
frame into 1st, 3rd segments
718
*
Retrieve bottom field for
display
720
Reconstruct upper half of
frame into 2nd, 3rd segments
722
*
Retrieve top field for
display
724
Reconstruct lower half of
frame into 1st, 3rd segments
726
*
Retrieve bottom field for
display
728
Reconstruct upper half of top
field into 2nd, segment
730
±
Pulldown and retrieve top
field again for display
732
Reconstruct upper half of
frame into 2nd, 3rd segments
734
Retrieve top field for
display
736
Reconstruct lower half of
frame into 1st, 3rd segments
738
±
Retrieve bottom field for
display
740
Reconstruct upper half of
frame into 1st, 2nd segments
742
4 _____________*
sc
an
lin
e#
_
sc
an
lin
e
U.S. Patent Nov. 10, 1998 Sheet 1 of 7 5,835,636
(PRIOR ART)
FIG. 2
(PRIOR ART)
U.S. Patent Nov. 10, 1998 Sheet 2 of 7 5,835,636
FIG. 3A
FIG. 6
D
is
pl
ay
C
od
e:
U.S. Patent Nov. 10,1998 Sheet 3 of 7 5,835,636
8SS
c 'c "e
Hi
O <1-
juauifias
FI
G
. 5
U.S. Patent Nov. 10, 1998 Sheet 4 of 7 5,835,636
FIG. 7A
U.S. Patent Nov. 10, 1998 Sheet 5 of 7 5,835,636
FIG. 7B
U.S. Patent Nov. 10, 1998 Sheet 6 of 7 5,835,636
FIG. 8
st
at
us
U.S. Patent Nov. 10, 1998 Sheet 7 of 7 5,835,636
° ro
E
CB
2
4-*
(/)
s
5,835,636
1
METHOD AND APPARATUS FOR
REDUCING THE MEMORY REQUIRED FOR
DECODING BIDIRECTIONALLY
PREDICTIVE-CODED FRAMES DURING
PULL-DOWN
FIELD OF THE INVENTION
The present invention relates to video systems, and more
particularly to memory segmentation and field reconstruc­
tion repeat for reducing the amount of memory required in
a video decoder for decoding and displaying bidirectionally
predictive-coded frames with pull-down.
DESCRIPTION OF THE RELATED ART
A video program signal is converted to a digital format,
and then compressed and encoded in accordance with one of
several known compression algorithms or methodologies.
This compressed digital system signal, or bitstream, which
includes a video portion, an audio portion, and other infor­
mational portion, is then transmitted to a receiver. Trans­
mission may be over existing television channels, cable
television channels, satellite communications channels, and
the like. A decoder is then typically employed at the receiver
to decompress and decode the received system signal in
accordance with the same compression algorithm used to
encode the signal. The decoded video information may then
be output to a display device, such as a television (TV)
monitor.
Video compression and encoding is typically performed
by a video encoder. The video encoder normally implements
a selected data compression algorithm that conforms to a
recognized standard or specification agreed to among the
senders and receivers of digital video signals. One such
emerging standard developed by the Moving Pictures
Experts Group (MPEG), is generally referred to as the
MPEG-1 Standard. A newer standard, referred to as MPEG-
2, is similar to MPEG-1 but includes extensions to cover a
wider range of applications. More particularly, MPEG-2
concerns high-quality coding of possibly interlaced video,
including high definition television (HDTV). A wide range
of applications, bit rates, resolutions, signal qualities and
services are addressed, including all forms of digital storage
media, TV broadcasting and communications.
In order to compress a video signal, it is typically neces­
sary to sample the analog data and represent this data with
digital values of luminance and color difference. The MPEG
standard specifies that a luminance component (Y) of a
video signal may be sampled with respect to a color differ­
ence signals (Cr, Cb) by a ratio of two-to-one (2:1). That is,
for every two samples of the luminance component Y, there
is one sub-sample each of the color difference components
Cr and Cb. A 2:1 sampling ratio is generally considered
acceptable because the human eye is much more sensitive to
luminance (brightness) components than to color compo­
nents. Video sampling typically is performed in both the
vertical and horizontal directions.
Once the video signal is sampled, it is typically formatted
into a non-interlaced signal that contains all of the picture
content. More particularly, the video signal includes a plu­
rality of pictures or frames, where each frame includes a
plurality of horizontal scan lines for display. An interlaced
signal, in contrast, is one that contains only part of the
picture content for each complete display scan. In an inter­
laced signal, each frame is divided into two fields. The two
fields are often referred to as the even and odd or the top and
bottom fields. Each field spans the length of the frame, but
only includes every other scan line. The purpose for such
field division is that most TVs today display the video
information in interlaced format, by displaying one field
first, such as the entire top field, then displaying the entire
bottom field.
2
After a video signal is sampled and formatted, the encoder
may process it further by converting it to a different reso­
lution in accordance with the image area to be displayed. In
doing so, the encoder must determine how to encode each
picture. A picture may be considered as corresponding to a
single frame of motion video, or to a frame of movie film.
However, different encoding schemes may be employed for
each picture. The most prevalent picture coding types are:
I-pictures (intra-coded pictures) which are coded without
reference to any other pictures and are often referred to as
anchor frames; P-pictures (predictive-coded pictures) which
are coded using motion-compensated prediction from the
past I- or P-reference picture, and may also be considered
anchor frames; and B-pictures (bidirectionally predictive-
coded pictures) which are coded using motion compensation
from a previous and a future I- or P-picture. These picture
types will be referred to as I, P or B frames.
A typical coding scheme may employ a mixture of I, P,
and B frames. Typically, an I frame may occur every half a
second, with two B frames inserted between each pair of I
or P frames. I frames provide random access points within
the coded sequence of pictures where decoding can begin,
but are coded with only a moderate degree of compression.
P frames are coded more efficiently using motion compen­
sated prediction from a past I or P frame and are generally
used as a reference for further prediction. B frames provide
the highest degree of compression but require both past and
future reference pictures for motion compensation. B frames
are generally not used as references for prediction. The
organization of the three picture types in a particular video
sequence is very flexible. A fourth picture type is defined by
the MPEG standard as a D-picture, or DC-picture, which is
provided to allow a simple, but limited quality, Fast-Forward
mode.
Once the picture types have been defined, the encoder
may estimate motion vectors for each 16 by 16 macroblock
in a picture. A macroblock (MB) is the basic coding unit for
the MPEG standard. A macroblock consists of a 16-pixel by
16-line portion, or four 8-pixel by 8-line blocks, of lumi­
nance components (Y) and several spatially corresponding 8
by 8 blocks of chrominance components Cr and Cb. The
number of blocks of chrominance values depends upon
which particular format is used. Common color space sam­
pling schemes include 4:4:4 for maximum quality but rela­
tively low compression, 4:2:2 including two Cb chromi­
nance blocks and Cr chrominance blocks, and 4:2:0
including two chrominance blocks. A plurality of such
macroblocks form a horizontal slice within a frame, where
the slice is the basic processing unit in an MPEG coding
scheme. A plurality of such slices form each picture or
frame, which is the basic unit of display. As described
previously, however, each frame is typically interlaced and
displayed as two separate fields.
Motion vectors provide displacement information
between a current picture and a previously stored picture. P
frames use motion compensation to exploit temporal
redundancy, or lack or motion, between picture frames in the
video. Apparent motion between sequential pictures is
caused by pixels in a previous picture occupying different
positions with respect to the pixels in a current macroblock.
This displacement between pixels in a previous and a current
macroblock is represented by motion vectors encoded in the
MPEG bitstream. Typically, the encoder chooses which
picture type is to be used for each given frame. Having
defined the picture type, the encoder then estimates motion
vectors for each macroblock in the picture. Typically in P
frames, one vector is employed for each macroblock, and in
B frames, one or two vectors are used.
When the encoder processes B frames, it usually re-orders
the picture sequence so that a video decoder receiving the
5
10
15
20
25
30
35
40
45
50
55
60
65
5,835,636
3
digital video signal operates properly. Since B frames are
usually coded using motion compensation based on previ­
ously sent I or P frames, the B frames can only be decoded
after the subsequent anchor pictures (an I or P frame) have
been received and decoded. Thus, the sequence of the series
of pictures may be re-ordered by the encoder so that the
pictures arrive at the decoder in a proper sequence for
decoding of the video signal. The decoder may then re-order
the pictures in proper sequence for viewing.
For a given macroblock of video data, the encoder is
programmed to select a coding mode depending on the
picture type, the effectiveness of motion compensation in the
particular region of the picture, and the nature of the signal
within the block. After the coding method is selected, the
encoder performs a motion-compensated prediction of the
block contents based on past and/or future reference pic­
tures. The encoder then produces an error signal by sub­
tracting the prediction from the actual data in the current
macroblock. The error signal is similarly separated into 8 by
8 blocks (four luminance blocks and two chrominance
blocks). A Discrete Cosine Transform (DCT) may then be 20
performed on each block to achieve further compression.
The DCT operation converts an 8 by 8 block of pixel values
to an 8 by 8 matrix of horizontal and vertical coefficients of
spatial frequency. Coefficients representing one or more
non-zero horizontal or non-zero vertical spatial frequencies
are called AC coefficients. An 8 by 8 block of pixel values
can subsequently be reconstructed by a video decoder per­
forming an Inverse DCT (IDCT) on the spatial frequency
coefficients.
Additional compression is provided through predictive
coding since the difference in the average value of neigh­
boring 8 by 8 blocks tends to be relatively small. Predictive
coding is a technique employed to improve compression
based on the blocks of pixel information previously operated
on by an encoder. A prediction of the pixel values for a block
yet to be encoded may be performed by the encoder. The
difference between the predicted and actual pixel values may
then be computed and encoded. The different values repre­
sent prediction errors which may later be used by a video
decoder to correct the information of a predicted block of
pixel values.
In addition to the signal compression that is achieved by
the encoding process itself, a substantial degree of inten­
tional signal compression is achieved by a process of
selecting a quantization step size, where the quantization
intervals or steps are identified by an index. The quantization 45
level of frequency coefficients corresponding to the higher
spatial frequencies favors the creation of coefficient values
of zero by choosing an appropriate quantization step size
4
statistical encoding of the expected runs of consecutive
zeroed-valued coefficients corresponding to the higher-order
coefficients accounts for considerable compression gain.
In order to cluster non-zero coefficients early in the series
5 and to encode as many zero coefficients as possible follow­
ing the last non-zero coefficient in the ordering. The coef­
ficient sequence is often organized in a specified orientation
termed zigzag ordering. Zigzag ordering concentrates the
highest spatial frequencies at the end of the series. Once the
10 zigzag ordering has been performed, the encoder typically
performs “run-length coding” on the AC coefficients. This
process reduces each 8 by 8 block of DCT coefficients to a
number of events represented by a non-zero coefficient and
the number of preceding zero coefficients. Because the
high-frequency coefficients are more likely to be zero,
run-length coding results in additional video compression.
The encoder may then include a variable length decoder
(VCD) to perform variable-length coding (VLC) on the
resulting data. VLC is a reversible procedure for coding data
that assigns shorter code words to frequent events and longer
code words to less frequent events, thereby achieving addi­
tional video compression. Fluffinan encoding is a particu­
larly well-known form of VLC that reduces the number of
bits necessary to represent a data set without losing any
information. The final compressed video data is then ready
25 to be transmitted to a storage device or over a transmission
medium for reception and decompression by a remotely
located decoder. The MPEG standard specifies a particular
syntax for a compressed bitstream. The MPEG video syntax
comprises six layers, each of which supports either a signal
30 processing Unction or a system function. The MPEG syntax
layers correspond to a hierarchical structure. A “sequence”
is the top layer of the video coding hierarchy and consists of
a header and some number of “Groups-of frames” (GOPs).
The sequence header generally initializes the state of the
decoder, which allows the decoder to decode any sequence
without being affected by past decoding history. A GOP is a
random access point, that is, it is the smallest coding unit
that can be independently decoded within a sequence. A
GOP typically consists of a header and some number of
“pictures.” The GOP header contains time and editing infor­
mation.
As discussed previously, there are at least three types of
pictures or frames: I frames, P frames and B frames. Because
of the picture dependencies, the order in which the frames
are transmitted, stored, or retrieved, is not necessarily the
display order, but rather an order required by the decoder to
properly decode the pictures in the bitstream. For example,
a typical sequence of frames, in display order, might be
shown as follows:
I B B P B B P B B P B B I B B P B B P
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
55
based on the human visual perception system. In particular, By contrast, the bitstream order corresponding to the
the step size is chosen sot that the human visual perception given display order would be as follows:
I P B B P B B P B B I B B P B B P B B
0 3 1 2 6 4 5 9 7 8 12 10 11 15 13 14 18 16 17
system is unlikely to notice the loss of a particular spatial 65 Because the B frame depends on a subsequent I or P frame
frequency unless the coefficient value for that spatial fre- in display order, the I or P frame must be transmitted and
quency rises above the particular quantization level. The decoded before the dependent B frame.
5,835,636
5
Each of the “picture” portions of a GOP consists of a
header and one or more slices. The picture header contains
time stamp, picture type, and coding information. A slice
consists of an integral number of macroblocks from a picture
and can be used by a video decoder to recover from
decoding errors. If the bitstream becomes unreadable within
a picture, the decoder will normally be able to recover by
waiting for the next slice, without having to drop the entire
picture. A slice also includes a header that contains position
and quantizer scale information. Since blocks are the basic
coding unit, the DCT is applied at the block level. Each
block typically contains 64 component pixels arranged in an
8 by 8 matrix. The pixel values are not individually coded,
but are components of the coded block. A macroblock
header is included which contains quantizer scale and
motion compensation information.
The video decoding process is generally the inverse of the
video encoding process and is employed to reconstruct a
motion picture sequence from a compressed and encoded
bitstream. The data in the bitstream is decoded according to
a syntax that is itself defined by the data compression
algorithm. The decoder must first identify the beginning of
a coded picture, identify the type of picture, then decode
each individual macroblock within a particular picture. If
there are motion vectors and macroblock types (each of the
frame types I, P, and B have their own macroblock types)
present in the bitstream, they can be used to construct a
prediction of the current macroblock based on past and
future reference frames that the decoder has already stored.
Coefficient data is then inverse quantized and operated on by
an IDCT process so as to transform the macroblock data
from the frequency domain to data in the time and space
domain.
After all of the macroblocks have been processed by the
decoder, the picture reconstruction is complete. If a recon­
structed frame is a reference or anchor frame, such as an I
or a P frame, it replaces the oldest stored anchor frame and
is used as the new anchor for subsequent frames. As noted
above, the frames may also need to be re-ordered before they
are displayed in accordance with their display order instead
of their coding order. After the frames are re-ordered, they
may then be displayed on an appropriate display device.
In general, encoded video data is received and stored in a
rate or channel buffer. The data is then retrieved from the
channel buffer by a decoder or reconstruction device for
performing the decoding process and the decoded data is
stored into a picture store buffer. In some configurations, the
channel and picture buffers are incorporated into a single
integrated memory device. The decoded data is in the form
of I, P or B frames, where a display controller retrieves the
picture data for display by an appropriate display device,
such as a TV monitor or the like. It is noted that prior to a
system according to the present invention, the picture buffer
had to be capable of storing at least three frames or
“framestores” of video information. Two frames of storage
was necessary for storing two anchor frames. The two
anchor frames were then used to reconstruct a B frame into
the third frame storage area.
In particular, to reconstruct a B frame, its two related
anchor frames must be decoded and available in the picture
buffer. This is true since the B frame is interpolated using
both anchor frames during the reconstruction process.
Because macroblocks include information for both even and
odd fields for each frame, reconstruction of each B frame is
performed progressively, or in a non-interlaced order.
Elowever, display of each frame is in an interlaced order,
where an entire top field is displayed first before beginning
6
the bottom field. Thus, B frame reconstruction must lead the
display of the B frame by at least half a frame, and the
reconstruction process had to be completed to finish the
display of the entire B frame. Due to the progressive versus
interlaced order between reconstruction and display, an
entire frame had to be available in prior art systems to
complete reconstruction of each B frame for display. Thus,
prior art systems had to include at least three frame stores of
memory.
The present disclosure primarily concerns MPEG-2
decoders compliant with the International Standards
Organization/International Electro-technical Commission
(ISO/IEC) 2-13818 standard for supporting NTSC (National
Television Systems Committee) or PAL (Phase Alternating
Line) standards. The NTSC resolution is 720x480 picture
elements (pixels) and the PAL resolution is 720x576 pixels
per frame. The picture rate is 24 to 30 frames per second. At
a sampling rate of 4:2:0, each frame requires an average of
12 bits per pixel. For memory devices such as a dynamic
random access memory (DRAM), each PAL type frame
requires 4,976,640 bits of storage and each NTSC frame
requires 4,147,200 bits of storage. The NTSC standard is
primarily for use in the United States (U.S.), whereas the
PAL standard is primarily for use in Europe. Since a
minimum of three frames of storage were required, a PAL
system would require at least 3x4,976,640=14,929,920 bits
(14.9 Mb) of memory. An NTSC system would require at
least 3x4,147,200=12,441,600 bits (12.5 Mb) of memory.
Furthermore, additional memory was required for the chan­
nel buffer, as well as overhead storage area for performing
a variety of miscellaneous overhead functions, where such
overhead memory was either distributed in the decoder
system or incorporated into the integrated memory.
A further complication is introduced if a decoder is
required to include 3:2 pull-down capability, which is cur­
rently required for NTSC decoders. The NTSC format
specifies a frame rate of 30 frames per second, whereas the
frame rate for a movie is 24 frames per second. Thus, ratio
of frame rates between the NTSC format and a movie is 5
to 4 frames. Although it would be possible to convert the
frame rates at the encoder, this is undesirable since this
would result in substantially less compression of video data
being transmitted or stored. Thus, it is desired to convert the
frame rate at the decoder when receiving a bitstream of
encoded video data. In order to convert a movie for display
for NTSC, one of every four fields of the movie is repeated
to convert to the NTSC format. In the MPEG-2 video coding
standard for NTSC, for example, it is necessary to display
data stored in the top field of a frame more than once if the
bitstream signals repeat top field. Since a frame includes two
fields, one of the fields of every other frame is repeated,
resulting in a pattern of 3 fields, 2 fields, 3 fields, 2 fields,
etc., hence the name “3:2 pull-down.” To achieve 3:2
pull-down, a field of every other frame had to remain in
memory long enough to be displayed a second time before
being overwritten. Up till now, a single framestore for B
frames has still been sufficient to include 3:2 pull-down
capability. However, a decoder with a single framestore
further required that the decoding or reconstruction process
be stalled one field time for every other frame to prevent
overwriting the repeated field.
The amount of memory is a major cost item in the
production of such decoders. Thus, it is desired to reduce the
memory requirements of the decoder system as much as
possible to reduce its size and cost. Since practical memory
devices are implemented using particular convenient dis­
crete sizes, it is important to stay within a particular size if
5
10
15
20
25
30
35
40
45
50
55
60
65
5,835,636
7
possible for commercial reasons. For example, it is desired
to keep the memory requirements below a particular size of
memory, such as 16 Mb, since otherwise a memory device
of 24 or 32 Mb would have to be used resulting in greater
cost and a waste of extraneous storage area. Although it is
possible to implement an NTSC decoder including three full
frames of storage within a 16 Mb DRAM memory device,
it is still desirable to reduce the memory requirements for
NTSC decoders.
SUMMARY OF THE INVENTION
A decoder system according to the present invention
overcomes these limitations and allows reduction of
memory for decoding and displaying B frames during 3:2
pull-down. In general, B frames are not used to predict other
frames, so that once the B frame data is retrieved for display,
it may be discarded. To enable re-use of memory, data from
the top and bottom fields is separated into different segments
of memory. Thus, once retrieval begins of the data in a
segment, that segment becomes free for reconstruction.
However, this initial scheme is not completely sufficient for
the top field of every other frame during 3:2 pull-down,
since the top field must be somehow made available again
for re-display. Rather than requiring additional memory to
save the top field of every other frame, the top field is
reconstructed again during the period when reconstruction is
typically stalled. In this manner, the amount of memory
required is reduced, and the field requiring re-display for 3:2
pull-down is simply reconstructed again. It is noted that the
term “display” refers to retrieving data for eventual display,
since the data might also be retrieved for transmittal,
encoding, storage, etc., and not necessarily for immediate
display.
In the preferred embodiment, a frame is conceptually
divided into four different sections, including the top-upper,
the top-lower, the bottom-upper and the bottom-lower fields.
The memory includes only three segments, where each
segment is the same size as each of the sections. Thus, each
segment of memory is sufficient for storing any one of the
four different sections of the frame. The portion of the
memory used for B frames is three-fourths the size of a full
framestore. During the reconstruction process, the pixel data
is separated between top and bottom fields and stored in a
respective one of two available segments of memory. In
particular, the data from each macroblock is separated into
two respective fields and stored in a corresponding segment
of the memory, so that each consecutive line in each segment
corresponds to consecutive lines of one field. During
retrieval, the data within each segment of memory is
retrieved line by line for display until the entire segment is
retrieved. The entire memory segment is available for recon­
struction as soon as retrieval of that segment is initiated.
Since each segment includes a plurality of scan lines stored
consecutively and retrieved consecutively, the DRAMs typi­
cally used to implement the memory are addressed in a
standard manner. Therefore, segmentation of the memory
allows the DRAM page hit rate to be maintained at the same
level, so that storage and retrieval of data from the memory
is not slowed. Furthermore, such segmentation of the
memory keeps the data organized and prevents undesirable
fragmentation of the memory.
For each frame, reconstruction of the upper half of the
frame involves the top-upper and bottom-upper sections.
Thus, the top-upper section is stored in one segment and the
bottom-upper section is stored in another segment. After half
the frame is reconstructed in this manner, retrieval of the top
field of that frame begins. The segment storing the top-upper
8
section is retrieved first according to interlaced display.
Almost immediately after retrieval of a segment begins, that
segment may be used for reconstruction of another section
of the frame. This is true since retrieval of a segment occurs
faster than its reconstruction. Thus, the third segment and the
segment being retrieved for display are used to store the
top-lower and bottom-lower sections during reconstruction
of the lower half of the frame. Approximately half way
through reconstruction of the lower half of the frame,
retrieval of the segment holding the top-upper section is
completed and retrieval for display of the segment contain­
ing the top-lower section begins, even though its reconstruc­
tion is not yet completed. Nonetheless, reconstruction of
each pixel in the segment is completed before its retrieval for
display.
At this point, two segments contain the data for the
bottom-upper and bottom-lower sections of the frame, so
that retrieval for display of the bottom field begins. One
segment still contains the data for the top-lower section of
the frame, so that re-display of this section is possible for 3:2
pull-down. However, the segment originally containing the
top-upper section has been overwritten and used for the
bottom field. Thus, the segment storing the bottom-upper
section is used to reconstruct the top-upper section while
being retrieved, so that reconstruction is not stalled.
Although reconstruction is typically performed with two
segments, the data for the bottom-upper section is discarded
while reconstructing the top-upper field again. In this
manner, reconstruction of the top-upper section of the frame
is reconstructed during retrieval of the bottom field for
display. Then, after the bottom field is retrieved for display,
the two segments of memory containing the top field are
retrieved to re-display the top field. Reconstruction of the
second frame begins during retrieval of the top field of the
first frame for re-display. Operation is similar for the second
frame, except that the top field is not reconstructed twice
since it is not re-displayed for the second frame.
Since only three segments are necessary for a frame
typically requiring four segments of memory, the portion of
the memory for B frames is three-fourths or 0.75 the size of
a full framestore. An NTSC frame including 720 pixelsx480
lines using 4:2:0 sampling, where each pixel includes, on the
average, 12 bits, has a full framestore of 4,147,200 bits. The
B frame is thus reduced to 3,110,400 bits, where each of the
segments of memory is preferably 1,036,800 bits. The
present invention, therefore, allows reduction of the picture
buffer memory for NTSC to 11,404,800 bits or approxi­
mately 11.4 Mb. Of course, these specific sizes of memory
correspond to the NTSC format. The present invention is not
limited to any particular video format or framestore size and
may be used to reduce the memory requirements of any
video system using 3:2 pull-down capability. Furthermore,
the present invention is applicable for different types of
video systems using pull-down schemes with different
ratios. Such variations are simply a matter of design choice.
More particularly, a decoder system according to the
present invention for reconstructing, storing and retrieving B
frames for display including pull-down conversion includes
a reconstruction unit for reconstructing the B frames, where
the reconstruction unit reconstructs the top-upper field of
every other frame twice. The decoder system includes a
memory having only three segments for storing pixel data,
where each segment is sized to store any one of the frame
sections. A segmentor is included for receiving and separat­
ing the pixel data according to the top and bottom fields for
each section of each frame, where the segmentor stores pixel
data from the top field into one segment and stores pixel data
5
10
15
20
25
30
35
40
45
50
55
60
65
5,835,636
9
from the bottom field into another segment of the memory.
The segmentor initially selects any two segments for the
upper half of the first frame, and then selects a segment
being retrieved for display and the remaining segment for
the bottom half of the first frame. Thereafter, the segmentor
selects the segment currently being retrieved for display and
whichever segment contains pixel data that has already been
retrieved and that will not be re-displayed. The segmentor
also selects one segment for receiving the upper portion of
the top field being reconstructed again for pull-down. The
selected segment for pull-down is that segment being
retrieved for display. The decoder system also includes
retrieval circuitry for retrieving pixel data from one of said
three segments at a time for interlaced display.
A method of reconstructing, storing and retrieving pixel
data for display of B frames using pull-down conversion
according to the present invention uses three segments of
memory, where each segment is one-fourth the size of a
framestore. The method comprises the steps of reconstruct­
ing the upper half of each frame into pixel data, reconstruct­
ing the lower half of each frame into pixel data, and for
every other frame, reconstructing the upper half of the frame
into pixel data again. During each of the reconstruction
steps, the method also includes the steps of separating the
pixel data according to top and bottom fields of the frame,
selecting two available segments for reconstructing the
upper half of the frame, selecting another two available
segments for reconstructing the lower half of the frame and
further selecting one available segment for reconstructing
the upper half again. The method further includes the steps
of storing top field pixel data into one selected segment and
storing bottom field data into the other selected segment
while reconstructing the upper half of the frame and while
reconstructing the lower half of the frame, and further
storing top field pixel data into the selected segment for
reconstructing the upper half of the frame again. Finally, the
stored pixel data from the segments of memory is retrieved
for interlaced display.
The step of selecting segments in the above method may
further include the steps of selecting any two segments for
the top half of the first frame, selecting a segment being
retrieved and the third segment for the bottom half of the
first frame, and during either of reconstructing steps, select­
ing the segment being retrieved for display and whichever
segment contains pixel data that has already been retrieved
and that will not be retrieved for re-display, and while
reconstructing the upper half of the frame again, selecting
the segment being retrieved for receiving top field pixel data.
The above method may further include steps of updating a
segment table including a list of pointers to segments
according to interlaced display, and retrieving pointers from
the segment table.
BRIEF DESCRIPTION OF THE DRAWINGS
A better understanding of the present invention can be
obtained when the following detailed description of the
preferred embodiment is considered in conjunction with the
following drawings, in which:
FIG. 1 is a graph illustrating operation of a decoder
system according to prior art;
FIG. 2 is a graph illustrating operation of a decoder
system operating according to prior art while performing 3:2
pull-down;
FIG. 3A is a diagram is shown of a frame conceptually
divided according to the present invention;
FIG. 3B is a diagram illustrating a memory used for
reconstructing and displaying B frames including 3:2 pull­
down;
10
FIG. 4 is a graph is shown illustrating reconstruction and
display operations of a decoder system according to the
present invention;
FIG. 5 is a tabular diagram illustrating operation of a
decoder system according to the present invention;
FIG. 6 is a state diagram illustrating operation according
to the present invention;
FIG. 7A is a flowchart diagram illustrating a method
according to the present invention;
FIG. 7B is another flowchart diagram illustrating a
method according to the present invention;
FIG. 8 is a simplified block diagram illustrating a decoder
system implemented according to the present invention; and
FIG. 9 is a block diagram of another decoder system
implemented according to the present invention.
DETAILED DESCRIPTION OF THE
PREFERRED EMBODIMENT
FIG. 1 is a graph illustrating operation of a decoder
system operating according to prior art without performing
3:2 pull-down. Each of the scan lines forming a B frame are
referenced along the y-axis and time is plotted along the
x-axis of the graph. Between times TO and T4, a first B
frame, referred to as FRO, is reconstructed as illustrated with
a solid line 100 plotted between the origin (O) and a point
A of the graph. Such reconstruction occurs in a progressive
manner, where each macroblock for each slice is recon­
structed one at a time into a picture buffer (not shown). Each
macroblock includes data for a contiguous 16-pixel by
16-line portion of the frame. Thus, each macroblock incor­
porates data for both fields, where the consecutive lines of
data are stored at consecutive locations within the picture
buffer. For both NTSC and PAL type decoders, 45 such
macroblocks are reconstructed for each slice having a width
of 720 pixels. For NTSC systems including 480 total scan
lines, 30 slices are reconstructed for each frame for a total
of 1,350 macroblocks per frame. For PAL systems including
576 total scan lines, 36 slices are reconstructed for each
frame for a total of 1,620 macroblocks per frame. It is noted
that PAL systems display 25 frames per second whereas
NTSC systems display 30 frames per second, so that each
system reconstructs and displays about 40,500 macroblocks
per second on the average.
At time T2, after reconstruction of approximately half of
the first frame FRO is completed as indicated by a point HI,
a display device (not shown) begins retrieving and display­
ing the first or top field of the first frame FRO, referred to as
FR0-FD0. A dash-dot line 102 plotted between points B and
A illustrates display of the top field FR0-FD0 of the first
frame FRO, where such display completes at approximately
time T4. On the average, reconstruction and display of each
pixel occurs at roughly the same rate. However, reconstruc­
tion of each frame occurs progressively, line by line,
whereas display occurs in an interlaced manner, or every
other line corresponding to a field. Thus, the effective rate of
display of each slice of scan lines is twice as fast as
reconstruction for each macroblock row of pixel data. Since
the display of the top field FR0-FD0 of the frame FRO takes
half the time as reconstruction of the entire frame FRO, the
slope of the line 100 is about half that of the slope of line
102. However, only half of the frame FRO has been dis­
played at time T4. Furthermore, it is noted that reconstruc­
tion of the last several lines of the last slice of the frame FRO
corresponding to the field FR0-FD0 is completed just prior
to those same lines being displayed. In this manner, recon­
struction and display are substantially locked together.
5
10
15
20
25
30
35
40
45
50
55
60
65
5,835,636
11
Beginning approximately at time T4, the display device
begins retrieving and displaying the second or bottom field
of the first frame FRO, referred to as field FR0-FD1. This is
illustrated with a dashed line 104 plotted between points C
and D, where display of the bottom field FR0-FD1 of the
first frame FRO is completed at approximately time T6. Also
between times T4 and T8, reconstruction of a second frame
FR1 occurs, as illustrated by a solid line 106 plotted between
points C and E. Since a single framestore of memory is used,
such reconstruction of the second frame FR1 overwrites the
data for the first frame FRO. Thus, the displaying of the
bottom field FR0-FD1 of the first frame FRO must begin
slightly before reconstruction begins of the second frame
FR1, since otherwise the data for the bottom field FR0-FD1
would be overwritten by data for the second frame FR1.
Such delay is relatively short, however, and could be the
delay associated with a single macroblock of data. Since the
display of the bottom field FR0-FD1 of the first frame FRO
occurs at twice the effective rate of reconstruction of the
second frame FR1, such display and reconstruction proceeds
concurrently between times T4 and T6 without interfering
with each other.
At time T6, display of the bottom field FR0-FD1 of the
first frame FRO completes while reconstruction of the sec­
ond FR1 is approximately half completed, as illustrated at
point H2. Thus, at time T6, the entire first frame FRO has
been displayed, while half of the second frame FR1 has been
reconstructed. Between times T6 and T8, display of the top
field FR1-FD0 of the second frame FR1 is performed, as
illustrated by a dash-dot line 108 between points F and E.
Thus at time T8, reconstruction of the entire second frame
FR1 and display of the top field FR1-FD0 of the second
frame FR1 is completed. Display of the bottom field FR1-
FD1 of the second frame FR1 begins at time T8 and ends at
time T10, as illustrated by a dashed line 110. Reconstruction
and display of a third and subsequent frames proceeds in a
similar manner beginning at time T8.
FIG. 2 is a graph illustrating operation of a decoder
system operating according to prior art while performing 3:2
pull-down. Such a system requires at least three full
framestores of memory. Again, each of the scan lines form­
ing a B frame are referenced along the y-axis and time is
plotted along the x-axis of the graph. Between times TO and
T4, the first B frame is reconstructed as illustrated with a
solid line 200 plotted between the origin (O) and a point A
of the graph. Again, such reconstruction occurs in a pro­
gressive manner. At time T2, after reconstruction of approxi­
mately half of the first frame is completed, the display
device begins retrieving and displaying the top field of the
first frame. A dash-dot line 202 plotted between points B and
A illustrates display of the top field, where such display
completes at approximately time T4.
Beginning approximately at time T4, the display device
begins retrieving and displaying the bottom field of the first
frame. This is illustrated with a dashed line 204 plotted
between points C and D, where display of the bottom field
occurs between times T4 and T6. In contrast to FIG. 1,
reconstruction of the second frame does not begin at time
T4, but instead is stalled until time T6. Reconstruction stall
prevents overwrite of the top field of the first frame, which
is displayed again between times T6 and T8 as illustrated
with a dash-dot line 206 plotted between points E and F.
Reconstruction of the next frame also begins at approxi­
mately time T6, as illustrated with a solid line 208 plotted
between points E and G. It is noted that since reconstruction
of the second frame is half the speed of re-display of the top
field of the first frame, these two events effectively occur
12
simultaneously without prematurely losing the data of the
top field being re-displayed. At time T8, after the top field of
frame 1 is redisplayed and halfway through reconstruction
of the second frame, the display device begins displaying the
top field of the second frame. This is illustrated with a
dash-dot line 210 plotted between points H and G, where
display of the top field of the second frame completes at time
T10. Then, at time T10, display of the bottom field of the
second frame and reconstruction of the third frame both
begin. Display of the bottom field is illustrated with a dashed
line 212 plotted between points J and K, and reconstruction
of the third frame is illustrated with a solid line 214 plotted
between points J and L. It is noted that re-display of the top
field of the second frame is not necessary, so that recon­
struction of the third frame is not stalled. Operation contin­
ues in this fashion where the top field of the third frame is
re-displayed and reconstruction of the fourth frame is
stalled, and so on.
A review of FIGS. 1 and 2 reveals that at least three
framestores of memory is sufficient for reconstructing and
displaying B frames both with and without 3:2 pull-down. It
is conceivable that less memory could be used for B frames,
since after data in a memory area is displayed, that memory
area is available for reconstruction. For practical reasons,
however, this has not been achieved in prior art systems.
Reconstruction occurs progressively on macroblocks of data
at a time, while display is interlaced and is performed in
raster scan format. Thus, reconstruction of a subsequent
frame in the same area of memory would destroy data in the
bottom field of the present frame. This is particularly prob­
lematic for systems requiring 3:2 pull-down, since the top
field data for every other frame must be stored for re-display.
It is conceivable to use less memory for B frames by
reconstructing the data twice as fast, so that reconstruction
is performed twice. Flowever, although the reconstruction
process may be sped up by a certain amount, practical
decoding systems are not presently capable of reconstructing
at twice the present rate. Furthermore, even if double-speed
reconstruction were possible, it would be required to store
the data at twice the present rate. Present DRAM devices
and similar practical memory devices, however, are not
capable of such storage speeds. Thus, reconstruction and
display processes are effectively locked together, so that an
entire framestore of memory was required for B frames. This
was true for decoder systems with or without 3:2 pull-down
capability.
A decoder system according to the present invention
overcomes these limitations and allows reduction of
memory for decoding and displaying B frames during 3:2
pull-down. In general, B frames are not used to predict other
frames, so that once the B frame data is retrieved for display,
it may be discarded. To enable re-use of memory, data from
the top and bottom fields is separated into different segments
of memory. Thus, once retrieval for display begins of the
data in a segment, that segment becomes free for recon­
struction. Flowever, this initial scheme is not completely
sufficient for the top field of every other frame during 3:2
pull-down, since the top field must be somehow made
available again for re-display. Rather than requiring addi­
tional memory to save the top field of every other frame, the
top field is reconstructed again during the period when
reconstruction is typically stalled. In this manner, the
amount of memory required is reduced, and the field requir­
ing re-display for 3:2 pull-down is simply reconstructed
again. It is noted that the present invention is not limited to
immediately displaying the data, so that the term “display”
refers to retrieving data for eventual display. For example,
5
10
15
20
25
30
35
40
45
50
55
60
65
5,835,636
13
although the data from a segment may be displayed imme­
diately upon retrieval, the data might also be retrieved for
transmittal, encoding, storage, etc., and not necessarily for
immediate display.
In the preferred embodiment, a frame is conceptually
divided into four different sections, including the top-upper,
the top-lower, the bottom-upper and the bottom-lower fields.
The memory includes only three segments, where each
segment is the same size as each of the sections. Thus, each
segment of memory is sufficient for storing any one of the
four different sections of the frame. Thus, the portion of the
memory used for B frames is three-fourths the size of a full
framestore. During the reconstruction process, the pixel data
is separated between top and bottom fields and stored in a
respective one of two available segments of memory. In
particular, the data from each macroblock is separated into
two respective fields and stored in a corresponding segment
of the memory, so that each consecutive line in each segment
corresponds to consecutive lines of one field. During the
retrieval for display process, the data within each segment of
memory is retrieved line by line for display until the entire
segment is retrieved. The entire memory segment is avail­
able for reconstruction as soon as retrieval of that segment
is initiated. Since each segment includes a plurality of scan
lines stored consecutively and retrieved consecutively, the
DRAMs typically used to implement the memory are
addressed in a standard manner. Therefore, segmentation of
the memory allows the DRAM page hit rate to be maintained
at the same level, so that storage and retrieval of data from
the memory is not slowed. Furthermore, such segmentation
of the memory keeps the data organized and prevents
undesirable fragmentation of the memory.
For each frame, reconstruction of the upper half of the
frame involves the top-upper and bottom-upper sections.
Thus, the top-upper section is stored in one segment and the
bottom-upper section is stored in another segment. After half
the frame is reconstructed in this manner, retrieval of the top
field of that frame begins. The segment storing the top-upper
section is retrieved first according to interlaced display.
Almost immediately after retrieval of a segment begins, that
segment may be used for reconstruction of another section
of the frame. This is true since retrieval of a segment occurs
faster than its reconstruction. Thus, the third segment and the
segment being retrieved for display are used to store the
top-lower and bottom-lower sections during reconstruction
of the lower half of the frame. Approximately half way
through reconstruction of the lower half of the frame,
retrieval of the segment holding the top-upper section is
completed and retrieval for display of the segment contain­
ing the top-lower section begins, even though its reconstruc­
tion is not yet completed. Nonetheless, reconstruction of
each pixel in the segment is completed before its retrieval for
display.
At this point, two segments contain the data for the
bottom-upper and bottom-lower sections of the frame, so
that retrieval for display of the bottom field begins. One
segment still contains the data for the top-lower section of
the frame, so that re-display of this section is possible for 3:2
pull-down. Flowever, the segment originally containing the
top-upper section has been overwritten and used for the
bottom field. Thus, the segment storing the bottom-upper
section is used to reconstruct the top-upper section while
being retrieved, so that reconstruction is not stalled.
Although reconstruction is typically performed with two
segments, the data for the bottom-upper section is discarded
while reconstructing the top-upper field again. In this
manner, reconstruction of the top-upper section of the frame
14
is reconstructed during retrieval of the bottom field for
display. Then, after the bottom field is retrieved for display,
the two segments of memory containing the top field are
retrieved to re-display the top field. Reconstruction of the
second frame begins during retrieval of the top field of the
first frame for re-display. Operation is similar for the second
frame, except that the top field is not reconstructed twice
since it is not re-displayed for the second frame.
It is noted that the data required to reconstruct the
top-upper field is already stored in a channel buffer of the
decoder. The channel buffer is preferably implemented as a
circular buffer and the decoder ensures that the data in the
channel buffer necessary to reconstruct the top-upper field is
not overwritten until it has been reconstructed the second
time.
Since only three segments are necessary for a frame
typically requiring four segments of memory, the portion of
the memory for B frames is three-fourths or 0.75 the size of
a full framestore. An NTSC frame including 720 pixelsx480
lines using 4:2:0 sampling, where each pixel includes, on the
average, 12 bits, has a full framestore of 4,147,200 bits. The
B frame is thus reduced to 3,110,400 bits, where each of the
segments of memory is preferably 1,036,800 bits. The
present invention, therefore, allows reduction of the picture
buffer memory for NTSC to 11,404,800 bits or approxi­
mately 11.4 Mb. Of course, these specific sizes of memory
correspond to the NTSC format. The present invention is not
limited to any particular video format or framestore size and
may be used to reduce the memory requirements of any
video system using 3:2 pull-down capability. Furthermore,
the present invention is applicable for different types of
video systems using pull-down schemes with different
ratios. Such variations are simply a matter of design choice.
It is noted that a decoder system according to the present
invention may be used to handle still frames. Since the
reconstruction unit is also stalled during still frames, a
processing element may be used to continually reconstruct
the ‘missing’ segment on demand. Furthermore, a decoder
system according to the present invention may be imple­
mented in any type of application including video capability,
where the present invention is not limited by the particular
application. For example, the decoder system could be
incorporated into digital video disk (DVD) applications,
entertainment systems, communications systems, etc. The
decoder system may be implemented in any one or more of
a plurality of system types, such as a set top box, a personal
computer (PC) or any larger computer system, such as
workstations, servers, minicomputers, main frame
computers, super computers, etc. The system may include
single chip modules (SCM), multi-chip modules (MCM),
board level products, box level products, etc.
A decoder system according to the present invention is
preferably incorporated into an MPEG-2 type system.
Flowever, the present invention is also applicable to other
decoder types, such as MPEG-1, Wavelet, FI.261, FI.320,
JPEG, etc. The present invention is preferred for PAL type
display formats, although other display formats are
contemplated, such as NTSC, red, green, blue (RGB)
formats, etc. The present invention may be used with any
type of display device, such as a TV, monitor, liquid crystal
display (LCD), plasma screen, visual projection device,
virtual reality display, etc. The present invention includes a
memory device, which is typically implemented using a
dynamic random access memory (DRAM) for cost consid­
erations. However, other memory types may be used, such
as static RAM (SRAM), video RAM (VRAM), or other
types of RAM systems with adequate performance.
5
10
15
20
25
30
35
40
45
50
55
60
65
5,835,636
15
Referring now to FIG. 3A, a diagram is shown of a B
frame, referred to by the letter F, which is conceptually
divided according to the present invention. Each frame F
includes a top field T and a bottom field B, where the top
field T is displayed first followed by the bottom field B for
interlaced display. Each frame F is conceptually divided into
four sections, including a top-upper section (TU) 302, a
top-lower section (TL) 304, a bottom-upper section (BU)
306 and a bottom-lower section (BL) 308. Thus, the display
order is TU, TL, BU and BL for each frame F for interlaced
display. For 3:2 pull-down, the TU and TL sections are
re-displayed in order following the BL section. FIG. 3B is a
diagram illustrating a memory M used for decoding
(reconstructing) and displaying B frames including 3:2
pull-down. Three segments of the memory M are defined,
including a segment #0 310, a segment #1 312 and a segment
#2 314. Each of the segments #0-#2 has sufficient capacity
for storing any one of the sections TU, TL, BU and BL.
Referring now to FIG. 4, a graph is shown illustrating
operation of a decoder system operating according to the
present invention while performing 3:2 pull-down. In con­
trast to FIGS. 1 and 2, FIG. 4 is demonstrated with respect
to segments of memory rather than fields or sections of a
frame. The three segments #0—#2 of the memory M shown
in FIG. 3B are used to display one or more B frames F
shown in FIG. 3A. Segment #0 is illustrated with a dash-dot
line, segment #1 is illustrated with a dotted line, and segment
#2 is illustrated with a dashed line. Since reconstruction
typically occurs into two segments at a time, solid lines are
shown to illustrate reconstruction. Reconstruction of a sec­
tion is denoted by the letter R, and display is denoted by the
letter D. Thus, reconstruction of the TU section is denoted
by RTU and display of that same section (or the correspond­
ing segment of memory) is denoted Y)TU. As described
above, the data need not be immediately displayed, so that
the term “display” refers to retrieval of data for display.
FIG. 5 is a tabular diagram 500 corresponding to the
graph of FIG. 4 for illustrating operation of a decoder system
according to the present invention. The tabular diagram 500
includes a plurality of columns generally aligned with the
time increments of the graph of FIG. 4. Each column of the
tabular diagram 500 represents access of the segments
#0-#2 of the memory M during each time increment, and
each row corresponds to one of the segments over time, as
denoted to the left of the tabular diagram 500. For example,
an entry of Rrf7 for segment #0 between times TO and T2
denotes that the TU section of the frame is being written into
segment #0 during reconstruction. Also, an entry of for
segment #0 between times T2 and T3 indicates display of
data in segment #0 corresponding to the section TU. FIGS.
4 and 5 should be referenced together for the following
description of operation.
Beginning at time TO, reconstruction of a first frame
begins, which ends at time T4. During the first half of
reconstruction of the first frame between times TO and T2,
segment #0 stores data from the TU section, denoted Rrf7,
while segment #1 stores data from the BU section, denoted
RBf7. It is noted that any two of the segments #0-#2 could
be used to begin the reconstruction process, where segments
#0 and #1 are chosen arbitrarily. Since half of the first frame
is completely reconstructed at time T2, a display device (not
shown) begins retrieving data from segment #0 at time T2 to
display the top-upper section, denoted Y)TU, where display of
the top-upper section is completed at time T3. Between
times T2 and T4, reconstruction of the lower half of the first
frame begins, including the TL and BL sections. Since
segment #0 starts being displayed at time T2, it is available
16
for reconstruction just after time T2. Thus, the data for the
TL section is written to section #2, denoted Rri, while data
from the BL section is written to section #0, denoted RBL.
Meanwhile at time T3, reconstruction of the TL section is at
least halfway completed into segment #2. Thus, the TL
section is displayed from segment #2, denoted T)tl, between
times T3 and T4.
At time T4, reconstruction of the entire first frame, and
display of the top field of the first frame, has been com­
pleted. Also, segment #1 contains data for the BU section
and segment #0 contains data for the BL section of the first
frame. Between times T4 and T5, therefore, display of the
BU from segment #1 is performed and between times T5 and
T6, display of the BL section from segment #0 occurs. The
top field including sections TU, TL of the first frame is to be
re-displayed for 3:2 pull-down. Segment #2 contains data
for the TL section of the first frame. However, the data from
the TU section no longer resides in segment #0 since it was
overwritten with data from the section BL during times T2
to T4. Thus, during display of the BU section from segment
#1 and display of the BL section from segment #0 between
times T4 and T6, reconstruction of the TU section is again
performed, where the data is stored into segment #1.
Although reconstruction typically occurs using two seg­
ments of the memory M, only one segment is used and the
data for the BU section is ignored or otherwise discarded.
At time T6, the TU section of the first frame resides in
segment #1 and the TL section of the first frame resides in
segment #2. Thus, the data of the TU section in segment #1
is re-displayed between times T6 and T7 and the data of the
TL section in segment #2 is re-displayed between times T7
and T8 for pull-down of the top field of the first frame. At
time T8, display of the first frame is completed, including
re-display of the top field.
Meanwhile at time T6, reconstruction of the TU and BU
sections of the second frame are reconstructed into segments
#0 and #1, respectively. Between times T8 and T9, the TU
section of the second frame stored in segment #0 is dis­
played and between times T9 and T10, the TL section of the
second frame stored in segment #2 is displayed. Meanwhile,
between times T8 and T10, the TL and BL sections of the
second frame are reconstructed into segments #2 and #0,
respectively. Another reconstruction of the TU section of the
second frame is not necessary since the top field of the
second frame is not repeated. Thus, the BU section of the
second frame is displayed from segment #1 between times
T10 and Til, while the BU section of the third frame is
reconstructed into that same segment #1 between times T10
and T12. The TU section of the third frame is reconstructed
into segment #2 while the BL section of the second frame is
displayed from segment #0 between times T 11 to T12. At
time T12, the second frame has been completely displayed
and the upper half of the third frame has been reconstructed.
Operation continues in a similar manner for the third frame
as for the first frame, where the TU section of the third frame
is reconstructed again into segment #1 between times T15
and T16. The entire procedure begins repeating at about
frame 5.
Referring now to FIG. 6, a state diagram 600 is shown
illustrating operation of the present invention. The state
diagram 600 illustrates operation of a finite state machine
(FSM) used to control a decoder system for performing 3:2
pull-down, such as an NTSC decoder system. The FSM may
be implemented in hardware, software, or any combination
thereof. Seven states 602, 604, 606, 608, 610, 612 and 614
are defined, each corresponding to one field time of a frame.
The state diagram 600 also corresponds to FIGS. 4 and 5,
5
10
15
20
25
30
35
40
45
50
55
60
65
5,835,636
17
where each state corresponds to two of the time intervals
(Tn). For example, state 602 corresponds to the time interval
T0-T2, state 604 corresponds to the time interval T2-T4,
state 606 corresponds to the time interval T4—T6, etc. The
state diagram 600 may be used to build the tabular diagram
500 of FIG. 5 and the graph shown in FIG. 4. In each state,
an “R” denotes reconstruction, which is followed by two
segment numbers into which data is being written. The first
segment number corresponds to the top field and the second
segment number corresponds to the bottom field. A segment
number “X”, as shown in states 606 and 614, indicates
reconstruction of the TU section of a frame for pull-down
while the data for the BU section is discarded. A “D” in a
state denotes display, which is followed by two segment
numbers indicating the segments being consecutively dis­
played during that state. A decision branch labeled PD
indicates pull-down is performed for the present frame,
whereas PD* indicates pull-down is not performed for that
frame.
Operation begins in state 602, where the top field of a first
frame is reconstructed and the data is written into segments
#0,1. Operation proceeds to state 604, where the top field is
displayed and the bottom field is reconstructed. The data
from reconstruction is written into segments #2, 0 while
segments #0 and #2 are displayed. Since pull-down of the
first frame is required, operation proceeds to state 606,
where the upper half of the top field of the first frame is again
reconstructed while the bottom field is displayed. Operation
proceeds to state 608, where pull-down of the top field
occurs, and reconstruction of the next frame begins. Opera­
tion proceeds back to state 604 from state 608, where the top
field of the second frame is displayed and the bottom field
of the second frame is reconstructed. Since pull-down is not
required for the second frame, operation proceeds to state
610 from state 604, where the bottom field of the second
frame is displayed, and the top field of the third frame is
reconstructed. Operation proceeds to state 612, where,
where the top field of the third frame is displayed and the
bottom field of the third frame is reconstructed. Since
pull-down is required for the third frame, operation proceeds
to state 614 from state 612, where the bottom field of the
third frame is displayed while the top field is reconstructed
again. Operation proceeds to state 610, where pull-down of
the top field of the third frame is performed, while the top
field of the fourth frame is reconstructed. Operation pro­
ceeds to state 612, where the top field of the fourth frame is
displayed and the bottom field of the fourth frame is recon­
structed. Since pull-down is not required for the fourth
frame, operation returns to state 608 from state 612. Opera­
tion continues in a similar manner until all consecutive B
frames have been reconstructed and displayed.
FIG. 7A is a flowchart diagram illustrating a method
according to the present invention, which generally corre­
sponds to the graph of FIG. 4, the tabular diagram 500 of
FIG. 5 and the state diagram 600 of FIG. 6. In a first step
702, the upper half of the first frame is reconstructed into
first and second segments. Operation then proceeds along
two parallel paths for display and reconstruction operations,
respectively. Each step of the display path corresponds to,
and occurs at approximately the same time as, a correspond­
ing step of the reconstruction path. Thus, steps 704 and 706
are performed together, as well as steps 708 and 710, 712
and 714, etc. The display path includes steps 704, 708, 712,
716, 720, 724, 728, 732, 736 and 740, which are performed
sequentially. Likewise, the reconstruction path includes
steps 706, 710, 714, 718, 722, 726, 730, 734, 738 and 742,
which are also performed sequentially. From steps 740 and
742, operation proceeds back to steps 704 and 706, respec­
tively.
18
For the display path, the top and bottom fields are
displayed consecutively, as indicated in steps 704 and 708,
716 and 720, 724 and 728, and 736 and 740. The top field
of every other frame is repeated for pull-down, as indicated
in steps 712 and 732. For the reconstruction path, the upper
half of the frame is reconstructed first, followed by the lower
half, as indicated in steps 702 and 706,714 and 718, 722 and
726, 734 and 738, and 742 and 706, etc. However, if
pull-down is required for a frame, the upper half of the top
field is reconstructed again as indicated in steps 710 and 730.
It is noted that several steps are repeated but using different
segments. For example, reconstruction of the upper half of
the frame occurs in steps 714 and 734, but step 714 uses the
first and second segments whereas step 734 uses the second
and third segments.
FIG. 7B is another flowchart diagram illustrating a
method according to the present invention, which is similar
to the flowchart of FIG. 7B except generalized. A first step
750 is similar to step 702, where the upper half of the first
frame is reconstructed using any two segments. Operation
then splits along parallel paths for display and
reconstruction, respectively. For display, operation proceeds
to step 752 where the top field of the frame is displayed.
Then the bottom field of the frame is displayed in step 754.
If the top field must be pulled-down and re-displayed for the
current frame as determined in step 756, operation proceeds
to step 758 to display the top field again. Operation then
returns back to step 752 to display the top field of the next
frame. If the top field of the current frame does not need to
be re-displayed as determined in step 756, operation pro­
ceeds directly back to step 752 for the next frame. Operation
continues until one or more consecutive B frames have been
displayed.
For reconstruction, operation proceeds from step 750 to
step 760, where the lower half of the current frame is
reconstructed. Step 760 is performed simultaneously with
display step 752. In the first iteration of step 760, one
segment used is the first segment being displayed in step
752, and the other segment is the remaining segment not
used in step 750. Thereafter, the first segment being dis­
played in step 752 is used as one segment in step 760. The
other segment used is the whichever segment of the remain­
ing two that contains data that has already been displayed
and that will not be re-displayed for pull-down. Operation
proceeds to step 762, where it is determined whether the
current frame requires pull-down. If so, operation proceeds
to step 764 where the top-upper section of the current frame
is reconstructed again. Step 764, if performed, occurs at the
same time as step 754, and the segment used in step 764 is
the first segment being used in step 754. From step 764,
operation proceeds to step 766 where the upper half of the
next frame is reconstructed. If the current frame does not
require pull-down as determined in step 762, operation
proceeds directly to step 766. Step 766 is performed during
each iteration and is performed at the same time as step 754,
if pull-down is not performed for the current frame, and is
otherwise performed at the same time as step 758 if pull­
down is performed. Step 766 uses the first segment being
displayed in either step 754 or 758, and uses whichever
segment of the remaining two that contains data that has
already been displayed and that will not be re-displayed for
pull-down. From step 766, operation proceeds back to step
760 until one or more consecutive B frames have been
reconstructed.
Referring now to FIG. 8, a simplified block diagram is
shown of a decoder system 800 according to the present
invention. The decoder system 800 shown in FIG. 8 prima-
5
10
15
20
25
30
35
40
45
50
55
60
65
5,835,636
19
rily illustrates data flow and organization. Encoded video
data in the form of a bitstream is provided to a rate or
channel buffer 804 on a data channel 802 for temporary
storage. The encoded video data typically includes picture
information that is representative of a plurality of single
frames of motion video. Each encoded frame or picture of
motion video is represented in digital form as a sequence of
bits. The structure of this sequence preferably conforms to a
selected video compression standard, such as the MPEG-1
or MPEG-2 standards, for example. The video data within
the channel buffer 804 is provided on another channel 806
to a reconstruction unit 808, which decodes the picture data
to a form appropriate for display.
The reconstruction unit 808 preferably incorporates
decoder functions for translating the encoded video data into
corresponding symbols and/or events, where these symbols
or events are then reconstructed into the original frames. For
example, the reconstruction unit 808 may include an IDCT
pipeline, a motion compensation (MC) pipeline and a merge
and store unit for executing the reconstruction process. The
IDCT pipeline coordinates the reconstruction of each mac­
roblock of a frame, and the MC pipeline processes any
motion compensation information with respect to each mac­
roblock.
The reconstructed frame data from the reconstruction unit
808 is then provided to a picture buffer 812 across a data
channel 810. The picture buffer 812 preferably includes
enough memory for two full anchor frames, referred to as A1
and A2, where the anchor frames are either I or P frames
according to the MPEG standard. The data channel 810 is
bidirectional and enables the reconstruction unit 808 to
retrieve data from either anchor frame A1 or A2 in the
picture buffer 812. For example, the anchor frame A1 may
store an I frame previously reconstructed by the reconstruc­
tion unit 808. The reconstruction unit 808 may retrieve the
I frame and merge data from the channel buffer 804 for
reconstruction of a P frame. The P frame may then be stored
in the picture buffer 812 as the anchor frame A2.
The picture buffer 812 also includes memory for storage
of data for a B frame. As in prior art systems, the anchor
frames A1 and A2 are retrieved by the reconstruction unit
808, for reconstructing a B frame, so that the anchor frames
A1 and A2 must be fully available in decoded form.
However, in prior art systems, the memory size required in
the picture buffer for the B frame was the same size as the
anchor frames A1 and A2. Thus, a picture buffer previously
had to include enough memory for three fall framestores,
including the two anchor frames A1 and A2 and a full B
frame. In the picture buffer 812, however, the B frame is
preferably segmented into three segments SI, S2, and S3,
where the total memory required to store and display the B
frame is less than a full framestore worth of memory.
Preferably, each segment S1-S3 is one-fourth of a
framestore for a total memory size for B frames of 0.75
framestores. The bit size of each of the segments S1-S3
depends upon the total size of the framestore. For NTSC
frames having a total of 4,147,200 bits, the size of the
memory M need only be 3,110,400 bits. When any one of the
segments S1-S3 of the B frame is being displayed, that
segment is freed up for reconstructed data. This is achieved
without affecting the page hit rate of the DRAMs typically
used to implement the picture buffer 812, and without
affecting the display of the B frame. Such reduction of the
amount of memory required for storing the B frame results
in cost savings of the overall decoder system 800. In the
preferred embodiment, the channel buffer 804, the picture
buffer 812 as well as any other memory required for func-
20
tions performed are incorporated into a single memory
device 814 having a maximum size of 16 Mb. This results
in significant cost reduction of the memory device 814.
According to the present invention, the reconstruction
unit 808 provides B frame data on a data channel 816 to a
segmentor 820 of a segmentation unit 818, where the
segmentor 820 divides the data into top and bottom fields
SF1 and SF2. The top and bottom field data is then written
to the B frame portion of the picture buffer 812. In particular,
top field data SF1 is provided into a first segment, such as
segment SI, while bottom field data SF2 is provided to a
second segment, such as segment S2. When these segments
SI and S2 are full, the segmentor 820 chooses another two
segments and provides the separated data to two more
segments. Preferably, the segmentor 820 maintains a seg­
ment pointer table 822 for keeping track of the order of the
segments S1-S3.
The segment pointer table 822 need only include four
pointer locations for storing pointers to the segments. Dur­
ing reconstruction of the first half of the first frame, the
segmentor 820 stores the pointer to the segment containing
the upper half of the top field in the first location and stores
the pointer to the segment containing the upper half of the
bottom field in the third location. During reconstruction of
the second half of the first frame, the segmentor 820 stores
the pointer to the segment containing the lower half of the
top field in the second location and stores the pointer to the
segment containing the lower half of the bottom field in the
fourth location. For purposes of pull-down, the segmentor
820 need only place the pointer to the upper half of the top
field into the first location. For the second frame, the order
of the entries is switched. Thus, during reconstruction of the
first half of the second frame, the segmentor 820 stores the
pointer to the segment containing the upper half of the top
field in the third location and stores the pointer to the
segment containing the upper half of the bottom field in the
first location. During reconstruction of the second half of the
second frame, the segmentor 820 stores the pointer to the
segment containing the lower half of the top field in the
fourth location and stores the pointer to the segment con­
taining the lower half of the bottom field in the second
location. The order remains reversed until the next pull­
down, where the order is once again reversed. A display
controller 824 retrieves the pointers in sequential order of
the pointer locations for interlaced display, which means the
1“, 2nd, 3rd, 4th, and then back to the 1st st pointer, etc. Of
course, the segmentor 820 may place the pointers in sequen­
tial order while the display controller 824 retrieves the
pointers in the appropriate order for interlaced display.
The display controller 824 retrieves data of a frame stored
in the picture buffer 812 across a data channel 826 and
provides the data in the appropriate form for display on a
display device 828. The display device 828 is preferably a
monitor, a TV, or any comparable display device. The
display controller 824 and display device 830 displays the
information in interlaced format. This is performed in a
standard manner for the anchor frames A1 and A2. For B
frames, however, the display controller 824 retrieves data
from each of the segments S1-S3 one at a time. The display
controller 824 preferably accesses the segment pointer table
822 within the segmentation unit 818 to determine which of
the segments S1-S3 to display at any given time. The
pointers to the segments S1-S3 are not necessarily listed in
numerical order in the segment pointer table 822. However,
the segment pointers are preferably retrieved from the table
822 in sequential order. Also, each of the segments are used
and accessed more than once.
5
10
15
20
25
30
35
40
45
50
55
60
65
5,835,636
21
Referring now to FIG. 9, a block diagram is shown of
another decoder system 900 according to the present inven­
tion. The decoder system 900 is preferably an NTSC type
MPEG-2 decoder. It is noted, however, that the present
invention is not limited to MPEG and may be used for any
particular video standard or configuration requiring 3:2
pull-down capability. A bitstream of video data is received
by a bitstream pre-processor 902 coupled to a common
memory bus 904, where the pre-processor 902 extracts the
encoded video data and provides the data on the bus 904. A
memory controller 906 retrieves the data from the bus 904
for temporary storage in a channel buffer portion of a
memory device 908. The memory controller 906 is coupled
between the bus 904 and the memory device 908 for
controlling access to the memory device 908. The memory
device 908 is preferably implemented with DRAMs, and
preferably includes the channel buffer and a picture buffer
for storing pixel data for I, P and B frames.
Video data from the channel buffer is then provided across
the bus 904 to a variable length decoder (VLD) 910, which
converts the data to DCT data. The data is then provided to
an IDCT pipeline 912, which converts the DCT data to
macroblocks of pixel data. The macroblocks are provided to
a merge and store unit 914 coupled to the bus 904, which
provides the data back to the memory device 908. This
procedure is followed for I frames. The procedure is similar
for P frames, except that the data from the IDCT pipeline
912 is merged with I frame data from a motion compensa­
tion (MC) pipeline 916, which retrieves I frame data from
the picture buffer in the memory device 908. The procedure
is similar for B frames, except that data from both I and/or
P anchor frames may be merged together to reconstruct a B
frame.
The VLD 910, the IDCT pipeline 912, the MC pipeline
916 and the merger and store unit 914 collectively perform
reconstruction of frame data. I and P frames are generally
stored within the memory device 908 in a standard manner.
However, B frames are segmented into the memory device
908 in accordance with the present invention as described
previously. In this manner, the memory device 908 need
only include enough memory for less than 3 full framestores,
and preferably 2.75 framestores. The segmentation process
may be performed by the merge and store unit 914 and the
display unit 918 or by the memory controller 906, or by any
particular combination of these devices. The memory con­
troller 906 is a convenient centralized location for storing B
frame data into segments and for providing the data in
appropriate order to the display unit 918. In the preferred
embodiment, however, B frame segmentation according to
the present invention is performed by a combination of the
merge and store unit 914 and the display unit 918. Thus, the
merge and store unit 914 contains circuitry for storing B
frame data in the picture buffer of the memory device 908 in
segments, and the display unit 918 includes circuitry for
retrieving the segments in appropriate order for display.
It is now appreciated that a method and apparatus for
reducing the memory required for decoding bidirectionally
predictive-coded frames during pull-down according to the
present invention substantially reduces the amount of
memory required for a video decoder. Since the amount of
memory is a major cost item in the production of such
decoders, substantial cost savings are achieved. In general,
B frames are not used to predict other frames, so that once
the B frame data is displayed, it may be discarded. To enable
re-use of memory, data from the top and bottom fields is
separated into different segments of memory. Thus, once
display begins of the data in a segment, that segment
22
becomes free for reconstruction. However, this initial
scheme is not completely sufficient for the top field of every
other frame during 3:2 pull-down, since the top field must be
somehow made available again for re-display. Rather than
requiring additional memory to save the top field of every
other frame, the top field is reconstructed again during the
period when reconstruction is typically stalled. In this
manner, the amount of memory required is reduced, and the
field requiring re-display for 3:2 pull-down is simply recon­
structed again.
More particularly, a frame is conceptually divided into
four sections, including the upper and lower halves of the
top field and the upper and lower halves of the bottom field.
The memory for storing and displaying the B frame includes
three segments, each being the same size as each of the four
sections of the frame. Tlie upper half of a first frame,
including the top-upper and bottom-upper sections, is recon­
structed into two segments of memory. The segment con­
taining the top-upper section is then displayed while that
same segment and a third segment are used to reconstruct the
lower half of the first frame, including the top-lower and
bottom-lower sections. The top-lower section is also dis­
played during the reconstruction of the lower half of the first
frame. Then, the segment storing the bottom-upper section
is displayed while that same section is also used to recon­
struct the top-upper section if the top field must be
re-displayed. The reconstruction of the top-upper section
completes while the bottom field is being displayed. In this
manner, two segments of memory store the entire top field
after the first frame has been completely displayed. The top
field is displayed again from these two segments to achieve
pull-down. Operation is similar for the next frame, except
that pull-down is not necessary since only performed for
every other frame.
Since there are only three segments of memory, it is a
relatively straightforward procedure for a decoding or recon­
struction unit or segmentor to determine the next two
available segments, and for a display unit to determine the
appropriate order of segments to display. A segment pointer
table may be used for this purpose, which is a list of pointers
to the segments. The table is updated during reconstruction
to list the segments in display order. Thus, display devices
need only read the pointers from the segment table in
consecutive order to identify the next segment for display.
Also, since only three segments are required to store and
display B frames including 3:2 pull-down, where each
segment is one-fourth the size of the frame, the framestore
for B frames is only 0.75 times a full framestore. Thus, the
size of the picture buffer is reduced to 2.75 framestores for
anchor and B frames.
Although a system and method according to the present
invention has been described in connection with the pre­
ferred embodiment, it is not intended to be limited to the
specific form set forth herein, but on the contrary, it is
intended to cover such alternatives, modifications, and
equivalents, as can be reasonably included within the spirit
and scope of the invention as defined by the appended
claims.
I claim:
1. A decoder system for efficient decoding and retrieving
bidirectionally predictive-coded (B) frames of pixel data for
display including pull-down, wherein each B frame includes
four sections corresponding to upper and lower portions of
first and second fields of the frame, respectively, the decoder
system comprising:
a reconstruction unit for reconstructing B frames into
pixel data, wherein said reconstruction unit recon­
structs the upper portion of the first field of every other
frame twice;
5
10
15
20
25
30
35
40
45
50
55
60
65
5,835,636
23
a memory including three segments for storing pixel data,
wherein each segment is sized to store any one of the
frame sections;
a segmentor coupled to said reconstruction unit and said
memory for receiving and separating said B frame pixel
data according to the first and second fields for each
section of each frame, wherein said segmentor stores
pixel data from the first field into one segment and
stores pixel data from the second field into another
segment of the memory, and wherein said segmentor
initially selects any two of said three segments, and
then selects a segment being retrieved for display and
the remaining segment, and thereafter selects the seg­
ment being retrieved for display and whichever seg­
ment contains pixel data that has already been retrieved
for display and that will not be re-displayed, and further
selects the segment being retrieved for display for
receiving pixel data for pull-down; and
retrieval circuitry coupled to said memory for retrieving
pixel data from one of said three segments at a time for
interlaced display including pull-down.
2. The decoder system of claim 1, wherein 3:2 pull-down
is performed.
3. The decoder system of claim 2, wherein the decoder
operates according to the National Television Systems Com­
mittee (NTSC) format.
4. The decoder system of claim 3, wherein each B frame
comprises a plurality of macroblock rows, and wherein each
of said plurality of macroblock rows includes data for two of
the four sections of each frame.
5. The decoder system of claim 3, wherein each frame
includes 720 pixels by 480 lines.
6. The decoder system of claim 5, wherein each frame
includes twelve bits per pixel on the average, so that each of
said three segments of said memory includes no more than
1,036,800 bits.
7. The decoder system of claim 6 wherein said memory
further includes two full framestores for storing two anchor
frames.
8. The decoder system of claim 1, wherein said memory
is implemented using dynamic random access memory.
9. The decoder system of claim 1, wherein said retrieval
circuitry begins retrieving pixel data from said plurality of
segments of said memory after said memory stores one-half
of a decoded B frame.
10. The decoder system of claim 1, further comprising:
a segment pointer table comprising a list of pointers to
said segments; and
wherein said segmentor maintains said segment pointer
table by placing said pointers in an appropriate order
for interlaced display of said B frame pixel data.
11. The decoder system of claim 10, wherein said retrieval
circuitry retrieves each of said pointers from said segment
pointer table in sequential order for accessing each of said
segments.
24
12. A method of reconstructing, storing and retrieving
pixel data for display of bidirectionally predictive-coded (B)
frames using pull-down conversion using three segments of
memory, wherein each segment is one-fourth the size of a
framestore, the method comprising the steps of:
reconstructing the upper half of each frame into pixel
data;
reconstructing the lower half of each frame into pixel
data;
for every other frame, reconstructing the upper half of the
frame into pixel data again;
during each of said reconstruction steps, separating the
pixel data according to first and second fields of the
frame;
selecting two available segments for said step of recon­
structing the upper half of the frame, selecting another
two available segments for said step of reconstructing
the lower half of the frame and further selecting one
available segment for said step of reconstructing the
upper half again;
storing first field pixel data into one selected segment and
storing second field data into the other selected segment
during said steps of reconstructing the upper half of the
frame and reconstructing the lower half of the frame,
and further storing first field pixel data into the selected
segment for said step of reconstructing the upper half of
the frame again; and
retrieving for interlaced display the stored pixel data from
the segments of memory.
13. The method of claim 12, wherein said step of selecting
segments comprises the steps of:
selecting any two segments for the upper half of a first
frame;
selecting a segment being retrieved and the third segment
for the lower half of a first frame;
during either of said steps of reconstructing the upper half
of the frame and reconstructing the lower half of the
frame, selecting the segment being retrieved and
whichever segment contains pixel data that has already
been retrieved and that will not be retrieved for
re-display; and
during said step of reconstructing the upper half of the
frame into pixel data again, selecting the segment being
retrieved for receiving first field pixel data.
14. The method of claim 12, further comprising the steps
of:
updating a segment table including a list of pointers to
segments according to interlaced display; and
said retrieving step further comprising the step of retriev­
ing pointers from the segment table.
5
10
15
20
25
30
35
40
45
50