Inclusion Criteria:

          -  Written informed consent provided.

          -  Males >=18 years of age (at the time written consent is obtained for screening).

          -  Histologically confirmed adenocarcinoma of the prostate: screening and on-treatment
             biopsy is mandatory. If adequate number of paired biopsy samples are collected (>=20
             paired samples for each dose level in each Arm, unless an Arm is closed early), then
             further biopsy sampling will be considered based on available data; screening biopsy
             can be waived if participant had a recent biopsy after failure of the most recent
             therapy (within 30 days) and the biopsy sample is secured to be sent as screening
             biopsy for this study.

          -  Surgically or medically castrated, with testosterone levels of less than or equal to
             (<=)50 nanograms per deciliter (ng/dL) (<2.0 nanometer [nM]). If the participant is
             being treated with luteinizing hormone-releasing hormone (LHRH) agonists/antagonists
             (participant who have not undergone orchiectomy) this therapy must have been initiated
             at least 4 weeks prior to Week 1 Day 1 and must be continued throughout the study.

          -  Participants must have failed prior therapy with abiraterone, enzalutamide, or both:

               1. Has completed at least 12 weeks of prior continuous therapy with abiraterone or
                  enzalutamide in any prior line.

               2. Lead-in dosing period for enzalutamide only will be required under the following
                  circumstance:

             (i) If the participant has enzalutamide discontinuation for >7 days prior to dosing
             start with GSK525762 plus enzalutamide on trial, then a enzalutamide only lead-in
             dosing of 28 days is required.

        (ii) If the participant has enzalutamide discontinuation for <=7 days prior to dosing start
        with GSK525762 plus enzalutamide on trial, then an enzalutamide only lead-in dosing of 14
        days is required.

        (iii) If the participant is on continuous dosing with enzalutamide prior to dosing start
        with GSK525762 plus enzalutamide on trial, then participant can start on combined dosing at
        end of screening period.

        (c) Lead-in dosing period for abiraterone only will be required: if the participant has
        abiraterone discontinuation for more than 3 days prior to dosing start with GSK525762 plus
        abiraterone on trial, then abiraterone only lead-in dosing of 7 days is required.

          -  One to two line(s) of prior taxane-based chemotherapy allowed. If docetaxel
             chemotherapy is used more than once, this will be considered as one regimen.
             Participants who have not received prior chemotherapy in any setting will qualify for
             study if they are ineligible for or refuse chemotherapy.

          -  Documented prostate cancer progression as assessed by the investigator with one of the
             following:

               1. PSA progression defined by a minimum of 3 rising PSA levels with an interval of
                  >=1 week between each determination. The PSA value at screening must be >=5
                  microgram (µg)/Liter (L) (5 ng/mL) if PSA is the only indication of progression;
                  participants on systemic glucocorticoids for control of symptoms must have
                  documented PSA progression by PCWG3 while on systemic glucocorticoids prior to
                  commencing Week 1 Day 1 treatment.

               2. Radiographic progression of soft tissue disease by PCWG3-modified RECIST 1.1
                  criteria or bone metastasis with 2 or more documented new bone lesions on a bone
                  scan with or without PSA progression.

          -  ECOG performance status of 0 or 1.

          -  Life expectancy >12 weeks.

          -  Able to swallow and retain orally administered medication.

          -  Must have adequate organ function.

          -  Male participants are eligible to participate if they agree to use contraceptive
             methods.

        Exclusion Criteria:

          -  Surgery or local prostatic intervention (excluding a prostatic biopsy) less than 28
             days of Week 1 Day 1.

          -  Participants with neuroendocrine and/or small cell CRPC.

          -  Recent prior therapy, defined as:

               1. Any investigational or approved non-biologic anti-cancer drug within 14 days
                  prior to the first dose of GSK525762 and abiraterone/enzalutamide.

               2. Any nitrosoureas or mitomycin C within 42 days prior to the first dose of
                  GSK525762 and abiraterone/enzalutamide.

               3. Any anti-cancer biologic agents within five half-lives prior to the first dose of
                  GSK525762 and abiraterone/enzalutamide.

               4. If the participant received radiotherapy <90 days prior to study treatment, the
                  irradiated lesion cannot be the only lesion used for evaluating response.

               5. Any major surgery within 28 days prior to the first dose of GSK525762 and
                  abiraterone/enzalutamide.

          -  Evidence of severe or uncontrolled systemic diseases (e.g., unstable or uncompensated
             respiratory, hepatic, renal, cardiac disease, or clinically significant bleeding
             episodes). Any serious and/or unstable pre-existing medical (aside from malignancy),
             psychiatric disorder, or other conditions that could interfere with participant's
             safety, obtaining informed consent or compliance to the study procedures, in the
             opinion of the Investigator; systolic blood pressure higher than 150 millimeter of
             mercury (mmHg) or diastolic blood pressure higher than 90 mmHg found on 2 separate
             occasions separated by 1 week, despite adequate therapy, will be defined as
             uncontrolled hypertension; uncontrolled diabetes mellitus (despite therapeutic,
             compliance intervention) as defined by a hemoglobin A1c (HbA1c) level more than 8%
             and/or occurrence of more than 2 episodes of ketoacidosis in the 12 months prior to
             the first dose of study drug.

          -  Cardiac abnormalities as evidenced by any of the following:

               1. Baseline QT interval corrected for heart rate by Fridericia's formula (QTcF)
                  interval >=480 milliseconds (msec).

               2. Clinically significant conduction abnormalities or arrhythmias, such as
                  participants with second degree (Type II) or third degree atrio-ventricular
                  block.

               3. History or evidence of current >=Class II congestive heart failure as defined by
                  New York Heart Association (NYHA).

               4. History of acute coronary syndromes (including unstable angina and myocardial
                  infarction), coronary angioplasty, or stenting within the past 3 months.
                  Participants with a history of stent placement requiring ongoing anti-coagulant
                  therapy (e.g., clopidogrel, prasugrel) will not be permitted to enroll.

               5. Known cardiac metastasis.

          -  Participants with history of known bleeding disorder(s) or history of clinically
             significant hemorrhage (e.g., gastrointestinal, neurologic), within the past 6 months.

          -  Therapeutic-dose anticoagulation (e.g., warfarin, low-molecular weight heparin [LMWH],
             or novel oral anticoagulants) must be discontinued and coagulation parameters must be
             normalized prior to the first dose of GSK525762 and abiraterone/enzalutimide.
             Prophylactic anticoagulation, with low doses (per standard practice) of agents such as
             LMWH, direct thrombin inhibitors, or factor Xa inhibitors is permitted.

          -  Concurrent use of high dose aspirin (doses up to 81 milligrams (mg) oral dose daily
             allowed, or 100 mg, as per country standards) and non-steroidal anti-inflammatory
             drugs (NSAIDS), except for where NSAIDs provide documented benefit over other
             analgesics, and then to be used with caution including concomitant use of proton pump
             inhibitors).

          -  Any acute toxicities due to prior chemotherapy and / or radiotherapy that have not
             resolved to a National Cancer Institute-Common Terminology Criteria for Adverse Events
             (NCI-CTCAE) version 4.0 grade <=1 with the exception of chemotherapy induced alopecia
             and grade 2 peripheral neuropathy.

          -  The participant has an active second malignancy other than curatively resected basal
             cell or squamous cell carcinoma of the skin, in situ carcinoma of the bladder, or
             other cancers for which they are treated with curative intent with no active disease
             in the 3 years prior to enrollment.

          -  Participants with known symptomatic brain metastasis are not suitable for enrolment.
             Participants with asymptomatic, stable, treated brain metastases are eligible for
             study entry.

          -  History of seizure within 6 months of study treatment initiation or any condition that
             may predispose participant to seizure (e.g., prior cortical stroke or significant
             brain trauma) or who are currently being treated with cytochrome P450 enzyme inducing
             anti-epileptic drugs for seizures (use of anti-epileptic drugs to control pain is
             allowed in participants not suffering from seizures unless drug is excluded due to
             Cytochrome (CY)P3A4 induction - phenytoin, carbamazepine, phenobarbital).

          -  History of loss of consciousness or transient ischemic attack within 12 months prior
             to enrollment.

          -  Participants with symptomatic or impending cord compression unless appropriately
             treated beforehand and clinically stable and asymptomatic.

          -  Current use of a prohibited medication or planned use of any forbidden medications
             during treatment with GSK525762 and abiraterone/enzalutamide. This includes
             medications that are potent inducers or inhibitors of CYP3A4 enzymes or strong
             inhibitors of CYP2C8.

          -  Participants with gastrointestinal disorders likely to interfere with absorption of
             the study medication.

          -  Participants with known bleeding diathesis.

          -  Current active liver or biliary disease (with the exception of Gilbert's syndrome or
             asymptomatic gallstones, liver metastases or otherwise stable chronic liver disease
             per investigator assessment).

          -  Initiating bisphosphonate or denosumab therapy or adjusting dose/regimen within 3
             months prior to Week 1 Day 1. Participants on a stable bisphosphonate or denosumab
             therapy are eligible and may continue.

          -  Any serious known immediate or delayed hypersensitivity reaction to GSK525762 or
             idiosyncrasy to drugs chemically related to the investigational drugs. Additionally,
             any known hypersensitivity to either enzalutamide, abiraterone or any excipients would
             be excluded.

          -  Known history of human immunodeficiency virus (HIV).

          -  Presence of hepatitis B surface antigen (HBsAg) or positive hepatitis C antibody test
             result at screening.