Inclusion Criteria:

          -  Newly diagnosed locally advanced rectal cancer with pathology confirmation with plans
             to proceed with neoadjuvant short course radiation and chemotherapy as confirmed by
             treating physician

          -  At least 18 years of age.

          -  ECOG performance status ≤ 1

          -  Normal bone marrow and organ function as defined below:

               -  Absolute neutrophil count ≥ 1,500/mcl

               -  Platelets ≥ 100,000/mcl

               -  Hemoglobin > 9 g/dL

               -  Total bilirubin ≤ IULN

               -  AST(SGOT)/ALT(SGPT) ≤ 2.5 x IULN

               -  Serum creatinine < 1.5 x IULN OR measured or calculated creatinine clearance ≥ 50
                  mL/min/1.73 m2

               -  INR or PT ≤ 1.5 x IULN unless patient is receiving anticoagulant therapy as long
                  as INR or PTT is within therapeutic range of intended use of anticoagulants

               -  aPTT ≤ 1.5 x IULN unless patient is receiving anticoagulant therapy as long as
                  INR or PTT is within therapeutic range of intended use of anticoagulants

          -  Willing to undergo study-related biopsies subject to accessibility of tumor,
             appropriateness of biopsy (not contraindicated), and continued subject consent.

          -  Women of childbearing potential and men must agree to contraceptive methods as
             described in protocol prior to study entry, for the duration of study participation,
             and for 120 days after the last dose of study treatment. Should a woman become
             pregnant or suspect she is pregnant while participating in this study, she must inform
             her treating physician immediately.

          -  Able to understand and willing to sign an IRB approved written informed consent
             document (or that of legally authorized representative, if applicable).

        Exclusion Criteria:

          -  Received prior anti-cancer therapy for rectal cancer.

          -  Prior treatment with an anti-PD-1, anti-PD-L1, anti-PD-L2, anti-CD137, or
             anti-cytotoxic T-lymphocyte-associated antigen 4 (CTLA-4) antibody (including
             ipilimumab or any other antibody or drug specifically targeting T-cell co-stimulation
             or checkpoint pathways) or other agents targeting IDO pathway (including indoximod)

          -  Previous radiotherapy in the pelvic region or previous rectal surgery (e.g. TEM) or
             any investigational treatment for rectal cancer within the past month.

          -  A history of prior malignancy active within the previous 3 years except for locally
             curable cancers that have been apparently cured, including, but not limited to, basal
             or squamous cell skin cancer, superficial bladder cancer, or carcinoma in situ of the
             prostate, cervix, or breast.

          -  Currently receiving any other investigational agents.

          -  Extensive growth into cranial part of the sacrum (above S3) or the lumbosacral nerve
             roots indicating that surgery will never be possible even if substantial tumor
             downsizing is seen.

          -  Presence of metastatic disease or recurrent rectal tumor.

          -  Diagnosis of Familial Adenomatosis Polyposis coli (FAP), Hereditary Non-Polyposis
             Colorectal Cancer (HNPCC), active Crohn's disease, or active ulcerative colitis.

          -  A history of allergic reactions attributed to compounds of similar chemical or
             biologic composition to epacadostat, pembrolizumab, 5-FU, oxaliplatin, or other agents
             used in the study.

          -  Has an active infection requiring systemic therapy.

          -  Warfarin (Coumadin): patients currently on warfarin are excluded. Patients who go off
             warfarin and have INR within normal limits have no washout period.

          -  Any history of serotonin syndrome (SS) after receiving serotonergic drugs. This
             syndrome has been most closely associated with the use of MAOIs, meriperidine,
             linezolid, or methylene blue; all of these agents are prohibited during the study

          -  Uncontrolled intercurrent illness including, but not limited to symptomatic congestive
             heart failure, unstable angina pectoris, or cardiac arrhythmia.

          -  Has an active or inactive autoimmune disease or syndrome (i.e. rheumatoid arthritis,
             moderate or severe psoriasis, multiple sclerosis, inflammatory bowel disease) that has
             required systemic treatment in the past 2 years or is receiving systemic therapy for
             an autoimmune or inflammatory disease (i.e. with use of modifying agents,
             corticosteroids, or immunosuppressive drugs). Exceptions include subjects with
             vitiligo or resolved childhood asthma/atopy, hypothyroidism stable on hormone
             replacement, controlled asthma, Type I diabetes, Graves' disease, or Hashimoto's
             disease.

          -  An abnormal screening ECG that, in the investigator's opinion, is clinically
             meaningful.

          -  Presence of a gastrointestinal condition that may affect drug absorption.

          -  Receipt of live attenuated vaccine within 30 days before the first dose of study
             treatment. Examples of live vaccines include, but are not limited to, the following:
             measles, mumps, rubella, chicken pox, yellow fever, rabies, Bacillus Calmette-Guérin
             (BCG), and typhoid vaccine. Seasonal influenza vaccines for injection are generally
             killed virus vaccines and are allowed; however, intranasal influenza vaccines (e.g.
             FluMist) are live attenuated vaccines and are not allowed.

          -  Pregnant and/or breastfeeding. Women of childbearing potential must have a negative
             pregnancy test within 72 hours of registration

          -  Evidence of interstitial lung disease or active, non-infectious pneumonitis including
             symptomatic and/or pneumonitis requiring treatment

          -  Known presence of active TB.

          -  Known active hepatitis B (e.g. HBsAg reactive or HBV DNA detected) or hepatitis C
             (e.g. HCV RNA [qualitative] is detected) infection. Testing at screening is required.