Apple Inc.

Patent Trials and Appeals Board

Dec 7, 2021

2020002774 (P.T.A.B. Dec. 7, 2021)

UNITED STATES PATENT AND TRADEMARK OFFICE
UNITED STATES DEPARTMENT OF COMMERCE
United States Patent and Trademark Office
Address: COMMISSIONER FOR PATENTS
P.O. Box 1450
Alexandria, Virginia 22313-1450
www.uspto.gov
APPLICATION NO. FILING DATE FIRST NAMED INVENTOR ATTORNEY DOCKET NO. CONFIRMATION NO.
15/455,890 03/10/2017 Jae Hoon Kim 082438.310200 1060
139020 7590 12/07/2021
BakerHostetler / Apple Inc.
Washington Square, Suite 1100,
1050 Connecticut Ave, NW
Washington, DC 20036-5304
EXAMINER
ZENATI, AMAL S
ART UNIT PAPER NUMBER
2651
NOTIFICATION DATE DELIVERY MODE
12/07/2021 ELECTRONIC
Please find below and/or attached an Office communication concerning this application or proceeding.
The time period for reply, if any, is set in the attached communication.
Notice of the Office communication was sent electronically on above-indicated "Notification Date" to the
following e-mail address(es):
eofficemonitor@bakerlaw.com
PTOL-90A (Rev. 04/07)
UNITED STATES PATENT AND TRADEMARK OFFICE
_______________

BEFORE THE PATENT TRIAL AND APPEAL BOARD
_______________

Ex parte JAE HOON KIM, CHRIS Y. CHUNG, DAZHONG ZHANG,
HANG YUAN, HIS-JUNG WU, XIAOSONG ZHOU, and JIEFU ZHAI
_______________

Appeal 2020-002774
Application 15/455,890
Technology Center 2600
_______________

Before JOHN A. JEFFERY, JOHNNY A. KUMAR, and
JAMES W. DEJMEK, Administrative Patent Judges.

DEJMEK, Administrative Patent Judge.

DECISION ON APPEAL
Appellant1 appeals under 35 U.S.C. § 134(a) from a Final Rejection of
claims 1–34. We have jurisdiction over the pending claims under 35 U.S.C.
§ 6(b).
We reverse.

1 Throughout this Decision, we use the word “Appellant” to refer to
“applicant” as defined in 37 C.F.R. § 1.42 (2018). Appellant identifies
Apple Inc. as the real party in interest. Appeal Br. 1.
Appeal 2020-002774
Application 15/455,890
2
STATEMENT OF THE CASE
Introduction
Appellant’s disclosed and claimed invention generally relates to
“video processing and, more particularly, perspective shifting in a video
conferencing session.” Spec. ¶ 1. More specifically, Appellant describes an
embodiment in which an image analyzer parses video data into one or more
foreground elements and one or more background elements. Spec. ¶ 19,
Fig. 2A. To achieve a perspective shift, in response to an input from a
recipient device, the image analyzer may shift the identified foreground
elements with respect to the identified background elements when the video
data is reconstructed prior to transmission to the recipient device. Spec.
¶¶ 19, 28.
Claim 1 is illustrative of the subject matter on appeal and is
reproduced below with the disputed limitations emphasized in italics:
1. A method comprising:
generating, by a first terminal, a video stream;
distinguishing, by an analyzer, foreground pixels of a
frame of the video stream representing a foreground element
from background pixels of the frame representing a background
element;
responsive to information representing a viewing
condition at a second terminal, creating, by the analyzer, a
modified video frame by compositing the foreground pixels and
the background pixels including a shift in the relative positions
of the foreground pixels and the background pixels; and
transmitting the modified video frame to the second
terminal.

Appeal 2020-002774
Application 15/455,890
3
The Examiner’s Rejections
1. Claims 1–19, 27, and 32–34 stand rejected under 35 U.S.C.
§ 103 as being unpatentable over El-Saban et al. (US 2012/0327172 A1;
Dec. 27, 2012) (“El-Saban”) or Lee (US 9,258,520 B2; Feb. 9, 2016) in view
of Didow et al. (US 2007/0064120 A1; Mar. 22, 2007) (“Didow”) or Knee et
al. (EP 2077525 A1; July 8, 2009) (“Knee”). Final Act. 2–5.
2. Claims 20–26 and 28–31 stand rejected under 35 U.S.C. § 103
as being unpatentable over El-Saban or Lin et al. (US 2015/0029294 A1;
Jan. 29, 2015) (Lin”) in view of Didow. Final Act. 6–7.
ANALYSIS2
Each of the independent claims (i.e., claims 1, 14, 20, 27, 28, 33, and
34) generally recite, inter alia, distinguishing foreground and background
pixels of the same frame of a video stream and, responsive to a viewing
condition input at a second (i.e., recipient) device, creating a modified video
frame by compositing the foreground and background pixels (from the same
frame) in a manner to achieve a shift in the relative position of the
foreground pixels and background pixels. The Examiner finds, alternatively,
that either El-Saban, Lee, Didow, Knee, or Lin teaches this aspect of the
claimed invention. See Final Act. 2–7; Ans. 4–8.
We begin our analysis with a brief review of the prior art.
Although El-Saban describes various foreground/background
separation techniques (see, e.g., El-Saban ¶¶ 8–9, 43–45), El-Saban

2 Throughout this Decision, we have considered the Appeal Brief, filed
September 17, 2019 (“Appeal Br.”); the Reply Brief, filed February 27, 2020
(“Reply Br.”); the Examiner’s Answer, mailed January 10, 2020 (“Ans.”);
and the Final Office Action, mailed March 22, 2019 (“Final Act.”), from
which this Appeal is taken.
Appeal 2020-002774
Application 15/455,890
4
describes allowing a user to define a template on the display of a video
capture device (e.g., a camera). El-Saban ¶ 66. El-Saban further describes
the template is superimposed on the display and the device is moved to
position the object within the superimposed template. El-Saban ¶¶ 67–69,
Figs. 3A, 3B. Contrary to the Examiner’s findings (see Final Act. 2–6;
Ans. 4–6), El-Saban does not create a modified frame by shifting the
foreground pixels and background pixels from a single frame relative to each
other, but instead combines foreground regions and background regions
from different images.
Moreover, Lee teaches a more intuitive approach to allow a user to
center the subject of a video stream (captured by a first device) on the
display of a second device (i.e., a receiving device). Lee, col. 1, ll. 31–39.
In particular, Lee teaches presenting a mirror image of the captured image to
a user of the first device such that if the subject appears too far left or right
from center, the user can intuitively move the subject or the camera in the
opposite direction to better align the subject. Lee, col. 6, ll. 21–45. Thus,
Lee does not teach creating a modified frame by shifting the foreground
pixels relative to the background pixels of the same frame.
Didow relates to “a method of creating event still photos or videos
using chroma-key technology.” Didow ¶ 2. Didow discloses that in such
techniques, a subject stands (or is placed) in front of a “green screen” (or
“blue screen”) and a substitute background is inserted into a resulting
composite photograph. Didow ¶¶ 5, 25. In other words, rather than creating
a modified frame by shifting the foreground pixels relative to the
background pixels of the same frame, Didow’s resulting composite image
uses the background from a separate image along with the foreground
subject of a different image.
Appeal 2020-002774
Application 15/455,890
5
Knee generally relates to “automatically reframing a moving image
sequence where the framing is adjusted in dependence upon foreground and
background segment weights derived for pixels in images in the sequence.”
Knee ¶ 4. However, as relied on by the Examiner, Knee does not describe
the reframing as shifting the foreground pixels relative to the background
pixels of the same frame. Rather, it would appear that the relative
positioning of the foreground and background pixels remains constant.
Finally, Lin generally relates to “integrating user personas with other
display content during video conferencing.” Lin ¶ 1. Lin describes
embedding an extracted video persona “into a background feed, which may
be, e.g., a background video, a desktop, slides, images, or any application
window.” Lin ¶ 3. Similar to Didow, rather than creating a modified frame
by shifting the foreground pixels relative to the background pixels of the
same frame, Lin’s resulting composite image uses the background from a
separate image along with the foreground subject of a different image.
Appellant disputes the Examiner’s findings that any of the identified
references teach creating a modified frame of a video stream by compositing
the foreground pixels and background pixels of the same frame with a
relative shift. Appeal Br. 7–12; Reply Br. 2–4.3 In particular, Appellant
asserts that rather than describing compositing foreground pixels and
background pixels from the same frame to achieve a perspective shift, El-
Saban describes capturing a new picture of a new scene to create a desired
picture. Appeal Br. 8–9 (citing El-Saban ¶¶ 68–69, Figs. 3A, 3B); Reply
Br. 2. Appellant further asserts that Lee merely describes moving an object

3 The Appeal Brief does not include page numbers. For reference herein, we
assume the first page of the Appeal Brief is page 1 and the subsequent pages
are consecutively numbered.
Appeal 2020-002774
Application 15/455,890
6
(e.g., the subject of the video frame) or a camera to center the object in the
video frame. Appeal Br. 9–10 (citing Lee, col. 6, ll. 21–45, Figs. 5A, 5B);
Reply Br. 2–3. Appellant argues Didow describes a traditional “green
screen” technique to combine an image of an object in front of a green
background with a second image of a scene by replacing the green
background in a first image with the scene captured in a second image.
Appeal Br. 10–11 (citing Didow ¶¶ 55–56); Reply Br. 3. Further, Appellant
argues that Knee merely describes “reframing an image without any
compositing or shifting of foreground relative to background.” Appeal
Br. 11–12 (citing Knee ¶ 7); Reply Br. 2. In addition, Appellant argues that
the combination of either El-Saban or Lee in view of either Didow or Knee
fails to teach compositing foreground and background pixels from the same
frame, including a shift in the relative positions of the foreground and
background pixels. Appeal Br. 12.
Based on our review of the prior art (discussed above), we agree with
Appellant that, as identified by the Examiner, the prior art (alone or in
combination) fails to teach distinguishing foreground pixels and background
pixels in a frame of a video stream and, in response to a request for an
altered viewing condition at a second terminal, creating a modified video
frame by compositing the foreground and background pixels including a
shift in the relative positions of the foreground pixels and background pixels,
as set forth in independent claims 1, 14, 20, 27, and 28.
For the reasons discussed supra, we are persuaded of Examiner error.
Accordingly, we do not sustain the Examiner’s rejection of independent
claim 1. For similar reasons, we do not sustain the Examiner’s rejections of
independent claims 14, 20, 27, and 28, which recite limitations
commensurate in scope to the disputed limitations of independent claim 1.
Appeal 2020-002774
Application 15/455,890
7
In addition, we do not sustain the Examiner’s rejections of claims 2–13, 15–
19, 21–26, and 29–32, which depend directly or indirectly therefrom.
In addition, the prior art similarly fails to teach identifying a
foreground portion and a background portion from a single digital image in
a video stream and, in response to a request for an altered viewing condition
at a destination terminal, creating a modified video frame by compositing
the foreground and background portions including a shift in the relative
positions of the foreground portion and background portion, as recited in
independent claim 33 and 34. As discussed above, the cited references
generally composite a modified image using foreground and background
portions taken from different images. Accordingly, we do not sustain the
Examiner’s rejection of independent claims 33 and 34.

CONCLUSION
We reverse the Examiner’s decision rejecting claims 1–34 under
35 U.S.C. § 103.
DECISION SUMMARY
Claim(s)
Rejected
35 U.S.C.
§
Reference(s)/Basis Affirmed Reversed
1–19, 27,
32–34
103 El-Saban, Lee,
Didow, Knee
1–19, 27,
32–34
20–26,
28–31
103 El-Saban, Lin,
Didow
20–26,
28–31
Overall
Outcome
1–34

REVERSED