Inclusion Criteria:

          -  Patients must have histologically or cytologically confirmed non-small cell lung
             cancer (cohort 1) or colorectal cancer (cohort 2)

          -  Patients must have measurable disease, defined as at least one lesion that can be
             accurately measured in at least one dimension (longest diameter to be recorded for
             non-nodal lesions and short axis for nodal lesions) as >= 20 mm (>= 2 cm) with
             conventional techniques or as >= 10 mm (>= 1.5 cm) with spiral computed tomography
             (CT) scan, magnetic resonance imaging (MRI), or calipers by clinical exam

          -  Patients in both cohorts must have progressive disease following prior therapy;
             specifically:

               -  Cohort 1 (NSCLC): Patients must have evidence of radiologic or clinical disease
                  progression during previous treatment with systemic PD-1 directed therapy and/or
                  have been deemed not to derive clinical benefit from PD-1 directed treatment;
                  this includes patients who demonstrated an initial response and subsequent
                  progression; no prior treatment with chemotherapy or targeted agents are
                  required; intervening therapy is allowed between previous PD-1 directed treatment
                  and there is no required interval from prior PD-1 treatment required; PD-1
                  directed treatment includes treatment with antibodies targeting the PD-1 receptor
                  such as pembrolizumab or nivolumab, as well as PD-L1 targeted antibodies such as
                  MEDI4736 (durvalumab), atezolizumab and avelumab; these agents may have been
                  administered as part of a clinical trial

               -  Cohort 2 (colorectal cancer): Patients must have progressed on >= one-line
                  chemotherapy

          -  At least 21 days must have elapsed from prior systemic therapy (chemotherapy or
             radiation)

          -  Eastern Cooperative Oncology Group (ECOG) performance status =< 1 (Karnofsky >= 60%)
             and life expectancy greater than 6 months; furthermore, enrollment of patients with
             greater than 10 measurable lesions is discouraged

          -  Patients must have normal organ and marrow function independent of transfusion for at
             least 7 days prior to screening and independent of growth factor support for at least
             14 days prior to screening

          -  Hemoglobin (Hgb) >= 9 g/dl

          -  Absolute neutrophil count >= 1,500/mcL

          -  Platelets >= 100,000/mcL

          -  Total bilirubin =< 1.5 x normal institutional limits; this will not apply to patients
             with confirmed Gilbert's syndrome (persistent or recurrent hyperbilirubinemia
             [predominantly unconjugated bilirubin] in the absence of evidence of hemolysis or
             hepatic pathology), who will be allowed in consultation with their physician

          -  Aspartate aminotransferase (AST) (serum glutamic-oxaloacetic transaminase
             [SGOT])/alanine aminotransferase (ALT) (serum glutamate pyruvate transaminase [SGPT])
             = < 2.5 x institutional upper limit of normal; for patients with hepatic metastases,
             ALT and AST =< 5 x ULT

          -  Measured creatinine clearance (CL) > 40 mL/min OR calculated creatinine clearance (CL)
             > 40 mL/min as determined by Cockcroft-Gault (using actual body weight)

          -  Patients must have at least one lesion that has not previously been irradiated (and is
             not within a previously radiated field) and for which palliative radiation is
             potentially indicated and could be safely delivered at the radiation doses specified
             in this protocol; this lesion must not be the only measurable lesion so that it is
             still possible to determine the response rate outside of the radiation treatment
             field; this lesion must not be within the central nervous system (CNS) (brain or
             spinal cord) or requiring urgent or emergent palliative radiation given the timing of
             radiation specified on this protocol; furthermore, this lesion:

               -  For cohort 1 (NSCLC cohort) - the lesion to be irradiated must be in the lung,
                  lymph nodes, adrenal gland or liver

               -  For cohort 2 (colorectal cohort) - the lesion to be irradiated must be in the
                  liver

          -  The effects of MEDI4736 and tremelimumab on the developing human fetus are unknown;
             for this reason and because radiation is known to be teratogenic, evidence of
             post-menopausal status or negative urinary or serum pregnancy test for female
             pre-menopausal patients is required; women will be considered post-menopausal if they
             have been amenorrheic for 12 months without an alternative medical cause; the
             following age-specific requirements apply:

               -  Women < 50 years of age would be considered post-menopausal if they have been
                  amenorrheic for 12 months or more following cessation of exogenous hormonal
                  treatments and if they have luteinizing hormone and follicle stimulating hormone
                  levels in the post-menopausal range for the institution or underwent surgical
                  sterilization (bilateral oophorectomy or hysterectomy)

               -  Women >= 50 years of age would be considered post-menopausal if they have been
                  amenorrheic for 12 months or more following cessation of all exogenous hormonal
                  treatments, had radiation-induced oophorectomy with last menses > 1 year ago, had
                  chemotherapy-induced menopause with > 1 year interval since last menses, or
                  underwent surgical sterilization (bilateral oophorectomy or hysterectomy)

          -  Females of childbearing potential who are sexually active with a non sterilized male
             partner must use at least 1 highly effective method of contraception from the time of
             screening and must agree to continue using such precautions for 180 days after the
             last dose of durvalumab + tremelimumab combination therapy or 90 days after the last
             dose of durvalumab monotherapy; non-sterilized male partners of a female patient must
             use male condom plus spermicide throughout this period; cessation of birth control
             after this point should be discussed with a responsible physician; not engaging in
             sexual activity for the total duration of the drug treatment and the drug washout
             period is an acceptable practice; however, periodic abstinence, the rhythm method, and
             the withdrawal method are not acceptable methods of birth control; female patients
             should also refrain from breastfeeding throughout this period

          -  Non-sterilized males who are sexually active with a female partner of childbearing
             potential must use a male condom plus spermicide from screening through 180 days after
             receipt of the final dose of durvalumab + tremelimumab combination therapy or 90 days
             after receipt of the final dose of durvalumab monotherapy; not engaging in sexual
             activity is an acceptable practice; however, occasional abstinence, the rhythm method,
             and the withdrawal method are not acceptable methods of contraception; male patients
             should refrain from sperm donation throughout this period

          -  Female partners (of childbearing potential) of male patients must also use a highly
             effective method of contraception throughout this period

          -  Highly effective methods of contraception, defined as one that results in a low
             failure rate (ie, less than 1% per year) when used consistently and correctly are
             described in the table below; note that some contraception methods are not considered
             highly effective (e.g. male or female condom with or without spermicide; female cap,
             diaphragm, or sponge with or without spermicide; non-copper containing intrauterine
             device; progestogen-only oral hormonal contraceptive pills where inhibition of
             ovulation is not the primary mode of action [excluding Cerazette/desogestrel which is
             considered highly effective]; and triphasic combined oral contraceptive pills)

               -  Should a woman become pregnant or suspect she is pregnant while she or her
                  partner is participating in this study, she should inform her treating physician
                  immediately

          -  Ability of a patient or a Legally Authorized Representative (LAR) to understand and
             the willingness to sign a written informed consent document

          -  Body weight > 30 kg

          -  Must have a life expectancy of at least 12 weeks

          -  Cohort 1 (NSCLC cohort)

               -  Ability to undergo a fresh tumor biopsy for the purpose of screening for this
                  clinical trial (including able and willing to give valid written consent) to
                  ability or to provide an available archival tumor sample taken less than 3 months
                  prior to study enrollment (and not obtained prior to progression on a PD-1/PD-L1
                  inhibitor) if a fresh tumor biopsy is not feasible with an acceptable clinical
                  risk; tumor lesions used for fresh biopsies should be the same lesions to be
                  irradiated when possible and should not be the same lesions used as Response
                  Evaluation Criteria in Solid Tumors (RECIST) target lesions, unless there are no
                  other lesions accessible; additional, optional archival tumor tissue is also
                  requested from before the prior PD-1 directed therapy

          -  Cohort 2 (colorectal cohort)

               -  Ability to undergo a fresh tumor biopsy for the purpose of screening for this
                  clinical trial (including able and willing to give valid written consent) to
                  ability or to provide an available archival tumor sample taken less than 3 months
                  prior to study enrollment if a fresh tumor biopsy is not feasible with an
                  acceptable clinical risk; tumor lesions used for fresh biopsies should be the
                  same lesions to be irradiated when possible and should not be the same lesions
                  used as RECIST target lesions, unless there are no other lesions accessible

               -  Microsatellite stable (MSS) tumor as documented by either:

                    -  Immunohistochemistry (IHC) testing that does not suggest loss of MLH-1,
                       MSH-2, PMS2 or MSH6

                    -  Polymerase chain reaction (PCR) testing that does not suggest microsatellite
                       instability (MSI)

        Exclusion Criteria:

          -  Patients who have had systemic (chemotherapy, biologic therapy or radiotherapy) within
             3 weeks (6 weeks for nitrosoureas or mitomycin C) prior to entering the study

          -  Receipt of prior radiotherapy or condition for any reason that would contribute
             radiation dose that would exceed tolerance of normal tissues, at the discretion of the
             treating physician

          -  Patients who have not recovered from adverse events due to prior anti-cancer therapy
             (i.e., have residual toxicities > grade 1)

          -  Patients who are receiving any other investigational agents

          -  Patients with untreated brain metastases, spinal cord compression, or leptomeningeal
             carcinomatosis should be excluded from this clinical trial; patients whose brain
             metastases have been treated may participate provided they show radiographic stability
             (defined as 2 brain images, both of which are obtained after treatment to the brain
             metastases; these imaging scans should both be obtained at least four weeks apart and
             show no evidence of intracranial progression); in addition, any neurologic symptoms
             that developed either as a result of the brain metastases or their treatment must have
             resolved or be stable either, without the use of steroids, or are stable on a steroid
             dose of =< 10 mg/day of prednisone or its equivalent (and anti-convulsants) for at
             least 14 days prior to the start of treatment

          -  History of allergic reactions attributed to compounds of similar chemical or biologic
             composition to tremelimumab and MEDI4736 or previous toxicity attributed to MEDI4736
             or other PD-1 or PD-L1 directed therapy that led to drug discontinuation

          -  Prior exposure to immune-mediated therapy, including durvalumab and tremelimumab,
             except for anti-PD-1 or anti-PD-L1 therapy (including durvalumab) in NSCLC patients;
             this includes anti-CTLA-4 agents (prior treatment with these agents is NOT allowed in
             either cohort) and, excludes therapeutic anticancer vaccines, excluding therapeutic
             anticancer vaccines; exposure to other investigational agents may be permitted after
             discussion with the study principal investigator (PI)

          -  Uncontrolled intercurrent illness including, but not limited to, ongoing or active
             infection, symptomatic congestive heart failure, unstable angina pectoris, cardiac
             arrhythmia, or psychiatric illness/social situations that would limit compliance with
             study requirements

          -  Pregnant women are excluded from this study because MEDI4736 (durvalumab),
             tremelimumab are immune checkpoint inhibitors with the potential for teratogenic or
             abortifacient effects, as is radiation therapy; because there is an unknown but
             potential risk for adverse events in nursing infants secondary to treatment of the
             mother with MEDI4736(durvalumab), tremelimumab and radiation, breastfeeding should be
             discontinued if the mother is treated with MEDI4736(durvalumab), tremelimumab and
             radiation

          -  Female patients who are pregnant or breastfeeding or male or female patients of
             reproductive potential who are not willing to employ effective birth control from
             screening to 90 days after the last dose of durvalumab monotherapy or 180 days after
             the last dose of durvalumab + tremelimumab combination therapy, whichever is later

          -  Human immunodeficiency virus (HIV)-positive patients are ineligible; appropriate
             studies will be undertaken in patients receiving combination antiretroviral therapy
             when indicated

          -  Any concurrent chemotherapy, immune therapy, biologic, hormonal therapy for cancer
             treatment

          -  Current or prior use of immunosuppressive medication within 14 days before the first
             dose of their assigned IP; the following are exceptions to this criterion:

               -  Intranasal, inhaled, topical steroids, or local steroid injections (e.g.,
                  intra-articular injection)

               -  Systemic corticosteroids at physiologic doses not to exceed 10 mg/day of
                  prednisone or its equivalent

               -  Steroids as pre-medication for hypersensitivity reactions (e.g., CT scan
                  pre-medication)

          -  Major surgical procedure (as defined by the investigator) within 28 days prior to the
             first dose of IP; Note: local surgery of isolated lesions for palliative intent is
             acceptable

          -  History of allogeneic organ transplantation

          -  Active or prior documented autoimmune or inflammatory disorders (including
             inflammatory bowel disease, diverticulitis [with the exception of diverticulosis];
             sarcoidosis syndrome, or other serious gastrointestinal [GI] chronic conditions
             associated with diarrhea; systemic lupus erythematosus; Wegener syndrome
             [granulomatosis with polyangiitis]; myasthenia gravis; Graves disease; rheumatoid
             arthritis; hypophysitis; uveitis, sarcoidosis syndrome, etc.) within the past 3 years
             prior to the start of treatment; the following are exceptions to this criterion:

               -  Patients with vitiligo or alopecia

               -  Patients with hypothyroidism (e.g., following Hashimoto syndrome) stable on
                  hormone replacement or psoriasis not requiring systemic treatment

               -  Any chronic skin condition that does not require systemic therapy

               -  Patients without active disease in the last 5 years may be included but only
                  after consultation with the study physician

               -  Patients with celiac disease controlled by diet alone

          -  History of another primary malignancy except for

               -  Malignancy treated with curative intent and with no known active disease >= 5
                  years before the first dose of study drug and of low potential risk for
                  recurrence

               -  Adequately treated non-melanoma skin cancer or lentigo maligna without evidence
                  of disease

               -  Adequately treated carcinoma in situ without evidence of disease (e.