Inclusion Criteria:

          -  For dose escalation: Patients must have A) histologically or cytologically confirmed
             low lying (up to 6 cm of anal verge) rectal adenocarcinoma eligible for radiation
             therapy to rectal tumor, B) if the treatment is palliative in the metastatic setting,
             no additional requirements for tumor size or nodal involvement is needed; C) if the
             treatment is in the neoadjuvant setting, the tumor must ALSO be high-risk locally
             advanced rectal cancer defined as T3-4, N+, and/or at risk for a positive radial
             margin (as determined by the surgeon)

          -  For dose expansion: Patients must have A) histologically or cytologically confirmed
             rectal adenocarcinoma eligible for radiation therapy to rectal tumor irrespective of
             location from anal verge, B) if the treatment is palliative in the metastatic setting,
             no additional requirements for tumor size or nodal involvement is needed; C) if the
             treatment is in the neoadjuvant setting, the tumor must ALSO be high-risk locally
             advanced rectal cancer defined as T3-4, N+, and/or at risk for a positive radial
             margin (as determined by the surgeon)

          -  Age >= 18 years. Because no dosing or adverse event data are currently available on
             the use of talimogene laherparepvec in combination with chemotherapy in patients < 18
             years of age

          -  Eastern Cooperative Oncology Group (ECOG) performance status =< 2 (Karnofsky >= 60%)

          -  Absolute neutrophil count (ANC) >= 1.5 x 10^9 /L

          -  Hemoglobin >= 9 g/dL

          -  Platelets >= 100,000 x 10^9 /L

          -  Serum bilirubin =< 1.5 x institutional upper limit of normal (ULN) (except patients
             with Gilbert's syndrome, who can have total bilirubin < 3 mg/DL)

          -  Aspartate aminotransferase (AST) and alanine aminotransferase (ALT) =< 2.5 x
             institutional ULN

          -  Serum creatinine =< 1.5 mg/dL OR calculated creatinine clearance (Cockcroft-Gault
             formula) >= 50 mL/min OR 24-hour urine creatinine clearance >= 50 mL/min

          -  Prothrombin time (PT)/international normalized ratio (INR) and partial thromboplastin
             time (PTT) =< 1.5 x institutional unless the subject is on anticoagulant therapy (if
             the subject is receiving anticoagulant therapy, PT, and activated partial
             thromboplastin time [aPTT] must be within therapeutic range of intended use of
             anticoagulants)

          -  Patients must have signed informed consent indicating that they are aware of the
             investigational nature of the study, and are aware that participation is voluntary;
             patients must be made aware of their other treatment options

          -  Talimogene laherparepvec, as well as other therapeutic agents used in this trial
             including radiation and capecitabine, may cause fetal harm when administered to a
             pregnant woman; women of childbearing potential (WOCBP) is defined as any female who
             has experienced menarche and who has not undergone surgical sterilization
             (hysterectomy or bilateral oophorectomy) or who is not postmenopausal; menopause is
             defined clinically as 12 months of amenorrhea in a woman over 45 in the absence of
             other biological or physiological causes; WOCBP and men must agree to use adequate
             contraception (hormonal or barrier method of birth control; abstinence) prior to study
             entry, during the study participation, and for four months after the last dose of the
             drug; WOCBP must have a negative serum pregnancy test within 72 hours prior to
             enrollment and agree to use effective contraception throughout the treatment period
             and for 4 months after the last dose of study treatment; should a woman become
             pregnant or suspect she is pregnant while she or her partner is participating in this
             study, she should inform her treating physician immediately

          -  Ability to understand and the willingness to sign a written informed consent document

        Exclusion Criteria:

          -  Patients who have had radiotherapy within < 4 weeks are ineligible

          -  Patients who have not recovered from adverse events due to prior anti-cancer therapy
             (i.e., have residual toxicities > grade 1) except alopecia are ineligible

          -  Use of other investigational, chemotherapeutic or targeted drugs within 28 days (or
             five half-lives, whichever is shorter; with a minimum of 14 days from the last dose)
             preceding the first dose of talimogene laherparepvec and during the study are
             ineligible

          -  Patients who have previously been treated with talimogene laherparepvec, any other
             oncolytic virus or pelvic radiation are ineligible

          -  Patients with known active central nervous system (CNS) metastases are ineligible;
             subjects with previously treated brain metastases may participate provided they are
             stable (without evidence of progression by imaging for at least four weeks prior to
             the first dose of trial treatment and any neurologic symptoms have returned to
             baseline), have no evidence of new or enlarging brain metastases

          -  Patients with a known immediate or delayed hypersensitivity reaction or idiosyncrasy
             to talimogene laherparepvec or any of its components, capecitabine, fluorouracil
             (5-FU) and / or oxaliplatin are ineligible

          -  Patients with a history or evidence of active autoimmune disease (e.g., pneumonitis,
             glomerulonephritis, vasculitis, or other); or history of active autoimmune disease
             that has required systemic treatment (i.e., use of corticosteroids, immunosuppressive
             drugs or biological agents used for treatment of autoimmune diseases) within 2 months
             of enrollment are ineligible; (replacement therapy [e.g., thyroxine for
             hypothyroidism, insulin for diabetes or physiologic corticosteroid replacement therapy
             for adrenal or pituitary insufficiency] is not considered a form of systemic treatment
             for autoimmune disease)

          -  Patients with evidence of clinically significant immunosuppression such as the
             following are ineligible:

               -  Primary immunodeficiency state such as severe combined immunodeficiency disease

               -  Concurrent opportunistic infection

               -  Receiving systemic immunosuppressive therapy (> 2 weeks) including oral steroid
                  doses > 10 mg/day of prednisone or equivalent within 2 months prior to enrollment

          -  Patients with active herpetic skin lesions or prior complications of herpetic
             infection (e.g., herpetic keratitis or encephalitis) are ineligible

          -  Patients with viral infections requiring intermittent or chronic systemic (intravenous
             or oral) treatment with an antiherpetic drug (e.g., acyclovir), other than
             intermittent topical use are ineligible

          -  Patients with other viral infections are ineligible:

               -  Known to have acute or chronic active hepatitis B or hepatitis C infection

               -  Known to have human immunodeficiency virus (HIV) infection

               -  Prior therapy with viral-based tumor vaccine

               -  Received live vaccine within 28 days prior to enrollment

          -  Subject who is unwilling to minimize exposure with his/her blood or other body fluids
             to individuals who are at higher risks for herpes simplex virus (HSV)-1 induced
             complications such as immunosuppressed individuals, individuals known to have HIV
             infection, pregnant women, or children under the age of 1 year, during talimogene
             laherparepvec treatment and through 30 days after the last dose of talimogene
             laherparepvec are ineligible

          -  Patients with uncontrolled intercurrent illness including, but not limited to, active,
             non-colorectal malignancies requiring systemic therapy, ongoing or active infection,
             symptomatic congestive heart failure, unstable angina pectoris, cardiac arrhythmia, or
             psychiatric illness/social situations that would limit compliance with study
             requirements are ineligible

          -  Although no effects on embryo-fetal development have been observed in animal studies,
             adequate and well-controlled studies with talimogene laherparepvec have not been
             conducted in pregnant women; in addition, given the high risk of detrimental effects
             of other study agents including capecitabine, 5-FU, oxaliplatin and radiation on
             embryo-fetal development, the study treatment must be excluded in the following
             patients:

               -  Female subject is pregnant or breast-feeding, or planning to become pregnant
                  during study treatment and through 4 months after the last dose of talimogene
                  laherparepvec

               -  Female subject of childbearing potential who is unwilling to use acceptable
                  method(s) of effective contraception during study treatment and through 4 months
                  after the last dose of talimogene laherparepvec

               -  Sexually active subjects and their partners unwilling to use male or female latex
                  condom to avoid potential viral transmission during sexual contact while on
                  treatment and within 30 days after treatment with talimogene laherparepvec

          -  Patients that are unable to swallow oral medications are ineligible

          -  Patients with a major surgical procedure, open biopsy, or significant traumatic injury
             within 28 days prior to initiation of therapy, or anticipation of need for major
             surgical procedure during the course of the study other than that defined by protocol
             are ineligible

          -  If patients require anticoagulation while on protocol, they should be switched from
             warfarin to another alternative anticoagulant such as low molecular weight heparin or
             oral direct factor Xa inhibitors as deemed appropriate by the treating physician for
             the duration they are on capecitabine (must be switched at least 1 week prior to
             starting capecitabine) to avoid drug-drug interaction of warfarin with capecitabine

          -  Patients with a known dihydropyrimidine dehydrogenase (DPD) deficiency are ineligible

          -  Patients who are unable to get MRIs due to any reason including pacemakers or
             automatic implantable cardioverter-defibrillator (AICD) are ineligible