Microsoft Technology Licensing, LLC

Patent Trials and Appeals Board

Dec 20, 2021

2021000191 (P.T.A.B. Dec. 20, 2021)

UNITED STATES PATENT AND TRADEMARK OFFICE
UNITED STATES DEPARTMENT OF COMMERCE
United States Patent and Trademark Office
Address: COMMISSIONER FOR PATENTS
P.O. Box 1450
Alexandria, Virginia 22313-1450
www.uspto.gov
APPLICATION NO. FILING DATE FIRST NAMED INVENTOR ATTORNEY DOCKET NO. CONFIRMATION NO.
14/723,065 05/27/2015 Tom Salter 355835-US-NP 9090
69316 7590 12/20/2021
MICROSOFT CORPORATION
ONE MICROSOFT WAY
REDMOND, WA 98052
EXAMINER
LIU, GORDON G
ART UNIT PAPER NUMBER
2612
NOTIFICATION DATE DELIVERY MODE
12/20/2021 ELECTRONIC
Please find below and/or attached an Office communication concerning this application or proceeding.
The time period for reply, if any, is set in the attached communication.
Notice of the Office communication was sent electronically on above-indicated "Notification Date" to the
following e-mail address(es):
chriochs@microsoft.com
ljohnson@microsoft.com
usdocket@microsoft.com
PTOL-90A (Rev. 04/07)
UNITED STATES PATENT AND TRADEMARK OFFICE
____________

BEFORE THE PATENT TRIAL AND APPEAL BOARD
____________

Ex parte TOM SALTER, BEN SUGDEN, LAURA MASSEY,
and BRIAN KEANE
____________

Appeal 2021-000191
Application 14/723,065
Technology Center 2600
____________

Before JOSEPH L. DIXON, NORMAN H. BEAMER,
and MICHAEL T. CYGAN, Administrative Patent Judges.

BEAMER, Administrative Patent Judge.

DECISION ON APPEAL
Appellant1 appeals under 35 U.S.C. § 134(a) from the Examiner’s
Non-Final Rejection of claims 1–8 and 10–19. Claims 9 and 20 are
cancelled. We have jurisdiction over the pending rejected claims under
35 U.S.C. § 6(b).
We affirm.

1 We use the word “Appellant” to refer to “applicant” as defined in
37 C.F.R. § 1.42 (2019). Appellant identifies Microsoft Technology
Licensing, LLC as the real party in interest. (Appeal Br. 4.)

Appeal 2021-000191
Application 14/723,065

2
THE INVENTION
Appellant’s disclosed and claimed invention is directed to a head
mounted display (HMD) device operating in a real world physical
environment with a visibly rendered projected gaze ray showing what a user
is looking at in the environment. (Abstr.)
Independent claim 1, reproduced below, is illustrative of the subject
matter on appeal:
1. One or more computer readable memories storing
computer-executable instructions which, when executed by one
or more processors in a local head mounted display (HMD)
device located in a physical environment, perform:
using data from a sensor package incorporated into the
HMD device to dynamically perform head tracking of a user
within the physical environment;
responsively to the head tracking, determining a field of
view of a mixed reality or virtual reality environment that is
renderable by the local HMD device, the field of view being
variable depending at least in part on a pose of the user’s head
in the physical environment;
receiving data that is transmitted from a remote HMD
device utilized by a remote user, the data including coordinates
of an origin and an intercept of a virtual gaze ray that is visibly
rendered and is projected from the origin at a view position of
the remote HMD device and terminates at the intercept at a
point of intersection between the virtual gaze ray and the mixed
reality or virtual reality environment, wherein the visibly
rendered virtual gaze ray with the coordinates of its origin and
intercept enable the user to see where the remote user is
looking; and
visibly rendering the virtual gaze ray and an avatar
representing the remote user on the local HMD device using the
received coordinates from the data within the field of view,
wherein the rendered virtual gaze ray projects from a location
Appeal 2021-000191
Application 14/723,065

3
on the rendered avatar corresponding to the view position of the
remote HMD device and extends to the intercept.
(Appeal Br. 17 (Claims App.).)
REJECTIONS
The Examiner rejected claims 1, 3–7, 10–12, and 14–19 under
35 U.S.C. § 103 as being unpatentable over Mount, et al. (US 20130141419
A1, pub. June 6, 2013), Maciocci, et al. (US 20120249741 A1, pub. Oct. 4,
2012), Abe, et al. (US 20150356788 A1, pub. Dec. 10, 2015), Tanguay, et
al. (US 20060092178 A1, pub. May 4, 2006), and Liu, et al. (US
20130050258 A1, pub. Feb. 28, 2013). (Non-Final Act. 5–35.)
The Examiner rejected claim 2 under 35 U.S.C. § 103 as being
unpatentable over Mount, Maciocci, Abe, Tanguay, Liu, and Wilairat, et al.
(US 20150261293 A1, pub. Sep. 17, 2015). (Non-Final Act. 35–36.)
The Examiner rejected claim 8 under 35 U.S.C. § 103 as being
unpatentable over Mount, Maciocci, Abe, Tanguay, Liu, and Gemmell et al.,
“Gaze Awareness for Videoconferencing: A Software Approach” (2000).
(Non-Final Act. 36–38.)
The Examiner rejected claim 13 under 35 U.S.C. § 103 as being
unpatentable over Mount, Maciocci, Abe, Tanguay, Liu, and
Raghoebardayal, et al. (US 20140132715 A1, pub. May 15, 2014). (Non-
Final Act. 38–39.)
ISSUE ON APPEAL
Appellant’s arguments present the following issue:2

2 Rather than reiterate the arguments of Appellant and the positions of the
Examiner, we refer to the Appeal Brief (filed Oct. 2, 2018) (Appeal Br.); the
Appeal 2021-000191
Application 14/723,065

4
Whether the Examiner erred in finding the combination of Mount,
Maciocci, Abe, Tanguay, and Liu would have taught or suggested the
limitations of independent claims 1, 12, and 16. (Appeal Br. 6–13.)
ANALYSIS
We have reviewed the Examiner’s rejection in light of Appellant’s
arguments. Arguments Appellant could have made, but chose not to make
are deemed to be waived. See 37 C.F.R. § 41.37(c)(1)(iv). We disagree
with Appellant’s arguments, and we adopt as our own: (1) the pertinent
findings and reasons set forth by the Examiner in the Action from which this
appeal is taken (Final Act. 5–39); and (2) the corresponding reasons set forth
by the Examiner in the Examiner’s Answer in response to Appellant’s
Appeal Brief. (Ans. 3–7.) We concur with the applicable conclusions
reached by the Examiner and emphasize the following.
In rejecting the independent claims, the Examiner relies on the
disclosure in Mount of head-mounted displays that visually augment an
observed physical space by superimposing virtual objects on the view, where
different users can view the same space from different perspectives,
including viewing the same virtual objects but each from their own
perspective. (Final Act. 5–6; Mount Figs.1A–C, ¶¶ 16, 25.) The head-
mounted displays have the capability of tracking the user’s head movements
to determine the field of view in the virtual reality environment. (Final Act.
6; Mount ¶¶ 25, 67.)

Non-Final Office Action (mailed May 2, 2018) (Non-Final Act.); and the
Examiner’s Answer (mailed Dec. 11, 2018) (Ans.) for the respective details.
Appeal 2021-000191
Application 14/723,065

5
The Examiner further relies on the disclosure in Maciocci of visibly
rendering a gaze ray (which Maciocci refers to as a “virtual input device”)
on a head-mounted display using received coordinates based on a vector
pointing from the display to a portion of a virtual object being pointed to by
the user. (Final Act. 7–8; Maciocci Fig. 37, ¶¶ 69, 302.) Although Maciocci
discloses multiple users with multiple head-mounted displays viewing the
same environment, it does not explicitly disclose that one user can view the
virtual input device generated by a different user. (Final Act. 8; Maciocci
Fig. 2, ¶ 87.) The Examiner finds that this aspect of the claims is obvious in
view of Maciocci in combination with Abe, given the disclosure in Abe of
multiple users sharing virtual objects viewed from head-mounted displays,
including avatars of the users. (Final Act. 8–10; Maciocci ¶ 3; Abe Figs. 2,
15, ¶¶ 104, 177.)
In addition, the Examiner relies on the disclosure of Tanguay of a
“virtual laser pointer,” in which the projected light of light of a laser pointer
on a surface is tracked and reproduced in the virtual reality environment
viewed by users via head-mounted displays, which the Examiner finds
teaches the claim requirement of transmitting coordinate data in order to see
where a remote user is looking. (Final Act. 10–12; Tanguay ¶¶ 33, 101–
104.) For this aspect of the claims the Examiner also relies on the disclosure
of Liu of a head-mounted display that tracks a user’s gaze direction and
highlights an object in the view. (Final Act. 13–14; Liu Figs. 15A, 16B,
¶¶ 136–137, 164.)
Appellant argues that Tanguay does not disclose transmitting the
coordinates of a virtual gaze ray, because it is limited to displaying the mark
that the laser light makes on a two-dimensional surface. (Appeal Br. 7–8.)
Appeal 2021-000191
Application 14/723,065

6
Appellant also argues that Liu is limited to a single user, and therefore
cannot satisfy the claim requirement of showing where a remote user is
looking. (Appeal Br. 8–9.) In addition, Appellant argues that Abe only
shares virtual objects, but does not share virtual gaze rays. (Appeal Br. 9–
10.) In regard to the virtual input device of Maciocci (the asserted visual
gaze ray), Appellant argues that it is solely disclosed as limited to the view
of the single user that is doing the pointing. (Appeal Br. 11–12.) Also for
independent claims 12 and 16, Appellant argues that Maciocci cannot
control the visual appearance of the gaze ray, and that the shared virtual
objects of Maciocci are not equivalent to gaze rays. (Appeal Br. 13–15.)
In the Answer, the Examiner reiterates the finding that the
combination of cited references teaches or suggests the requirements of the
independent claims:
[T]he combination of the prior arts on record, Maciocci, etc.
(US 20120249741 Al), ABE, etc. (US 20150356788 Al),
Tanguay, etc. (US 20060092178 Al), and Liu, etc. (US
20130050258 Al) teach these claimed limitations in such a way
that Maciocci teaches that the virtual gaze ray may be rendered
visible on the local HMD (See Maciocci: Fig. 37, and [0302]),
and ABE teaches that the local HMD and remote HMD may
share virtual/real objects (See ABE: Figs.12-15, and [0177]),
while Tanguay teaches how to transmit and receive data
between local and remote HMDs (See Tanguay: [0033] and
[0104]), and Liu teaches the details of the gaze ray related data
like interception coordinates (See Liu: Fig. 15, and [0136]), etc.
Thus, the remote gaze ray of Maciocci may be a virtual object
shared with (received by) the local HMD and rendered visible
in the local HMD, and this combination may be corresponding
to the limitations mentioned above.
(Ans. 4.) The Examiner also responds that Abe teaches that two head-
mounted displays may share assisted reality space, including, by implication,
Appeal 2021-000191
Application 14/723,065

7
transmitting data between displays, and that in addition Tanguay explicitly
teaches such data transmission. (Ans. 4–5.) The Examiner further
emphasizes that Maciocci and Abe teach the rendering and sharing of a
virtual gaze ray. (Ans. 5.)
Appellant does not reply to the Examiner’s Answer, and we are not
persuaded that the Examiner’s findings are incorrect. Appellant’s arguments
are unpersuasive as focusing on references individually, whereas the
Examiner relies on the combination of references. Non-obviousness cannot
be established by attacking references individually where, as here, the
ground of unpatentability is based upon the teachings of a combination of
references. In re Merck & Co., Inc., 800 F.2d 1091, 1097 (Fed. Cir. 1986).
Rather, the test for obviousness is whether the combination of references,
taken as a whole, would have suggested the patentee’s invention to a person
having ordinary skill in the art. In re Keller, 642 F.2d 413, 425 (CCPA
1981).
Therefore, we sustain the rejections of independent claims 1, 12, and
16 over Mount, Maciocci, Abe, Tanguay, and Liu. We also sustain the
rejections of claims 2–8, 10, 11, 13–15, and 17–19, which are not argued
separately with particularity. (Appeal Br. 13, 15.)

Appeal 2021-000191
Application 14/723,065

8
DECISION SUMMARY
In summary:
Claim(s)
Rejected
35
U.S.C. §
Reference(s)/Basis Affirmed Reversed
1, 3–7, 10–
12, 14–19
103 Mount, Maciocci,
Abe, Tanguay, Liu
1, 3–7, 10–
12, 14–19

2 103 Mount, Maciocci,
Abe, Tanguay, Liu,
Wilairat
2
8 103 Mount, Maciocci,
Abe, Tanguay, Liu,
Gemmell
8
13 103 Mount, Maciocci,
Abe, Tanguay, Liu,
Raghoebardayal
13
Overall
Outcome
1–8, 10–19

TIME PERIOD FOR RESPONSE
No time period for taking any subsequent action in connection with
this appeal may be extended under 37 C.F.R. § 1.136(a)(1)(iv).
AFFIRMED