Inclusion Criteria:

          -  Histologic confirmation of adenocarcinoma of the prostate

          -  The patient must have received definitive local therapy for prostate cancer,
             consisting of either radiation therapy and/or prostatectomy (salvage or adjuvant
             radiation post-prostatectomy is not exclusionary)

          -  PSA must be >= 2 ng/ml if received only prior definitive radiation (no PSA threshold
             required if prior prostatectomy was performed) with a PSA doubling time (PSADT) =< 10
             months:

               -  PSADT calculation must include all recorded PSA values > 0.2 ng/ml over the past
                  6 months prior to randomization, with a minimum of 3 values spaced at least 2
                  weeks apart, with each included value preferably measured at the same laboratory.
                  PSA values obtained prior to localized therapy will be excluded

               -  The calculation of PSADT is based on the natural log of PSA

          -  Prior salvage radiation or not a candidate for localized salvage radiation due to
             subject preference or clinical assessment based upon disease characteristics and/or
             subject co-morbidities

          -  Prior hormonal therapy (i.e. androgen deprivation therapy) when given as
             neoadjuvant/concurrent/adjuvant therapy along with definitive radiation is allowed,
             provided this was stopped >= 6 months prior to starting treatment per protocol AND
             testosterone is >= 150 ng/dl

          -  No evidence of metastatic disease on imaging by whole body bone scan and computed
             tomography (CT) or magnetic resonance imaging (MRI) of the chest/abdomen/pelvis within
             6 weeks before study therapy start day. PSMA positron emission tomography (PET) or
             fluciclovine scan within 6 weeks of start day may substitute other imaging studies.

               -  Patients with oligometastatic disease (i.e. =< 3 sites) detectable on advanced
                  imaging only (e.g. PSMA or fluciclovine PET) are eligible

               -  Abdominal or pelvic lymph nodes measuring =< 2 cm in short axis are allowed

          -  Biomarker positive:

               -  Biallelic CDK12 inactivating mutations as documented using a clinical grade
                  sequencing assay performed in a Clinical Laboratory Improvement Act
                  (CLIA)/College of American Pathologists (CAP) certified laboratory or

               -  MMRd/MSI-high as documented using a clinical grade assay performed in a CLIA/CAP
                  certified laboratory or

               -  Loss of function mutations in homologous recombination genes (i.e. homologous
                  recombination deficiency; HRD) as documented using a clinical grade sequencing
                  assay performed in a CLIA/CAP certified laboratory. Homologous recombination
                  genes include, but not limited to BRCA1, BRCA2, ATM, CHEK2, PALB2, RAD51D, NBN,
                  GEN1, RAD51C, MRE11A, BRIP11A, FAM175A.

          -  Capable of giving signed informed consent which includes compliance with the
             requirements and restrictions listed in the informed consent form (ICF) and in this
             protocol. Written informed consent and any locally required authorization (e.g.,
             Health Insurance Portability and Accountability Act) obtained from the patient/legal
             representative prior to performing any protocol-related procedures, including
             screening evaluations

          -  Eastern Cooperative Oncology Group (ECOG) performance status of =< 1

          -  Hemoglobin >= 10.0 g/dL with no blood transfusion in the past 28 days (within 28 days
             prior to administration of study treatment)

          -  Absolute neutrophil count (ANC) be >= 1.5 x 10^9/L (within 28 days prior to
             administration of study treatment)

          -  Platelet count >= 100 x 10^9/L (within 28 days prior to administration of study
             treatment)

          -  Serum bilirubin =< 1.5 x institutional upper limit of normal (ULN) (within 28 days
             prior to administration of study treatment). This will not apply to patients with
             confirmed Gilbert's syndrome (persistent or recurrent hyperbilirubinemia that is
             predominantly unconjugated in the absence of hemolysis or hepatic pathology), who will
             be allowed only in consultation with their physician

          -  Aspartate aminotransferase (AST) (serum glutamic-oxaloacetic transaminase
             [SGOT])/alanine aminotransferase (ALT) (serum glutamate pyruvate transaminase [SGPT])
             =< 2.5 x institutional upper limit of normal unless liver metastases are present, in
             which case it must be =< 5 x ULN (within 28 days prior to administration of study
             treatment)

          -  Patient must have creatinine clearance (CL) >= 51 mL/min by the Cockcroft-Gault
             formula (Cockcroft and Gault 1976) or by 24-hour urine collection for determination of
             creatinine clearance (within 28 days prior to administration of study treatment)

          -  Patient is willing and able to comply with the protocol for the duration of the study
             including undergoing treatment and scheduled visits and examinations including follow
             up

          -  Patients must have a life expectancy >= 16 weeks

          -  Body weight > 30 kg

          -  Male patients must use a condom during treatment and for 3 months after the last dose
             of olaparib when having sexual intercourse with a pregnant woman or with a woman of
             childbearing potential. Female partners of male patients should also use a highly
             effective form of contraception if they are of childbearing potential

        Exclusion Criteria:

          -  Prior chemotherapy for prostate cancer, unless done in the neoadjuvant setting, and if
             the last dose was > 6 months prior to enrollment

          -  Any prior treatment with a PD1 or PD-L1 inhibitor, including durvalumab

          -  Any prior treatment with a PARP inhibitor, including olaparib

          -  History of another primary malignancy except for

               -  Malignancy treated with curative intent and with no known active disease >= 3
                  years before the first dose of durvalumab and of low potential risk for
                  recurrence

               -  Adequately treated non-melanoma skin cancer or lentigo maligna without evidence
                  of disease

               -  Adequately treated carcinoma in situ without evidence of disease

          -  History of leptomeningeal carcinomatosis

          -  Resting electrocardiogram (ECG) indicating uncontrolled, potentially reversible
             cardiac conditions, as judged by the treating physician (e.g., unstable ischemia,
             uncontrolled symptomatic arrhythmia, congestive heart failure, corrected QT interval
             by Fridericia [QTcF] prolongation > 470 ms, electrolyte disturbances, etc.), or
             patients with congenital long QT syndrome

          -  Any unresolved toxicity National Cancer Institute (NCI) Common Terminology Criteria
             for Adverse Events (CTCAE) grade >= 2 from previous anticancer therapy with the
             exception of alopecia, vitiligo, and the laboratory values defined in the inclusion
             criteria

               -  Patients with grade >= 2 neuropathy will be evaluated on a case-by-case basis
                  after consultation with the study physician

               -  Patients with irreversible toxicity not reasonably expected to be exacerbated by
                  treatment with durvalumab or olaparib may be included only after consultation
                  with the study physician

          -  Patients with myelodysplastic syndrome/acute myeloid leukemia or with features
             suggestive of MDS/AML

          -  Patients considered a poor medical risk due to a serious, uncontrolled medical
             disorder, non-malignant systemic disease or active, uncontrolled infection. Examples
             include, but are not limited to, extensive bilateral lung disease on high resolution
             computed tomography (HRCT) scan, uncontrolled ventricular arrhythmia, recent (within 3
             months) myocardial infarction, uncontrolled major seizure disorder, unstable spinal
             cord compression, superior vena cava syndrome, interstitial lung disease, serious
             chronic gastrointestinal conditions associated with diarrhea or any psychiatric
             disorder that prohibits obtaining informed consent

          -  Patients unable to swallow orally administered medication and patients with
             gastrointestinal disorders likely to interfere with absorption of the study medication

          -  Immunocompromised patients, e.g., patients with uncontrolled human immunodeficiency
             virus (HIV). HIV+ patients will be allowed on the study if on highly active
             antiretroviral therapy (HAART) and disease is controlled: CD4 >= 350 cell/mcl,
             undetectable viral load, and no prophylactic (PPX) antibiotics

               -  Note: HIV screening is not required to be eligible for this study

          -  Active infection including tuberculosis (TB testing only performed if deemed necessary
             per standard clinical practice), hepatitis B (known positive hepatitis B virus [HBV]
             surface antigen [HBsAg] result), hepatitis C. Patients with a past or resolved HBV
             infection (defined as the presence of hepatitis B core antibody [anti-HBc] and absence
             of HBsAg) are eligible. Patients positive for hepatitis C virus (HCV) antibody are
             eligible only if polymerase chain reaction is negative for HCV ribonucleic acid (RNA)

          -  Active or prior documented autoimmune or inflammatory disorders (including
             inflammatory bowel disease [e.g., colitis or Crohn's disease], diverticulitis [with
             the exception of diverticulosis], systemic lupus erythematosus, Sarcoidosis syndrome,
             or Wegener syndrome [granulomatosis with polyangiitis, Graves' disease, rheumatoid
             arthritis, hypophysitis, uveitis, etc]). The following are exceptions to this
             criterion:

               -  Patients with vitiligo or alopecia

               -  Patients with hypothyroidism (e.g., following Hashimoto syndrome) stable on
                  hormone replacement

               -  Any chronic skin condition that does not require systemic therapy

               -  Patients without active disease in the last 5 years may be included but only
                  after consultation with the sponsor investigator

               -  Patients with celiac disease controlled by diet alone

          -  Current or prior use of immunosuppressive medication within 14 days before the first
             dose of durvalumab. The following are exceptions to this criterion:

               -  Intranasal, inhaled, topical steroids, or local steroid injections (e.g., intra
                  articular injection)

               -  Systemic corticosteroids at physiologic doses not to exceed 10 mg/day of
                  prednisone or its equivalent

               -  Steroids as premedication for hypersensitivity reactions (e.g., CT scan
                  premedication)

          -  Receipt of live attenuated vaccine within 30 days prior to the first dose of
             durvalumab. Note: Patients, if enrolled, should not receive live vaccine whilst
             receiving durvalumab and up to 30 days after the last dose of durvalumab

          -  Patients receiving any chemotherapy, immunotherapy, biologic, radiotherapy or hormonal
             therapy for cancer treatment concurrently or within 3 weeks of study treatment.
             Concurrent use of hormonal therapy for non-cancer related conditions (e.g., hormone
             replacement therapy) is acceptable

          -  Radiotherapy treatment to more than 30% of the bone marrow or with a wide field of
             radiation within 4 weeks of the first dose of study drug

          -  Concomitant use of known strong CYP3A inhibitors (e.g. itraconazole, telithromycin,
             clarithromycin, protease inhibitors boosted with ritonavir or cobicistat, indinavir,
             saquinavir, nelfinavir, boceprevir, telaprevir) or moderate CYP3A inhibitors (e.g.
             ciprofloxacin, erythromycin, diltiazem, fluconazole, verapamil). The required washout
             period prior to starting study treatment is 2 weeks

          -  Concomitant use of known strong (e.g. phenobarbital, enzalutamide, phenytoin,
             rifampicin, rifabutin, rifapentine, carbamazepine, nevirapine and St John's wort) or
             moderate CYP3A inducers (e.g. bosentan, efavirenz, modafinil). The required washout
             period prior to starting study treatment is 5 weeks for enzalutamide or phenobarbital
             and 3 weeks for other agents

          -  Major surgery within 2 weeks of starting study treatment and patients must have
             recovered from any effects of any major surgery

          -  Previous allogenic bone marrow transplant or double umbilical cord blood
             transplantation (dUCBT)

          -  Participation in another clinical study with an investigational product administered
             in the last 3 months

          -  Concurrent enrollment in another clinical study, unless it is an observational
             (non-interventional) or supportive care, while on the clinical study or during the
             follow-up period of this interventional study

          -  Patients with a known hypersensitivity to olaparib or durvalumab or any of the
             excipients of the product

          -  Involvement in the planning and/or conduct of the study