The phase transitions of (1-x)Pb$(Yb_{1/2}Ta_{1/2})O_3-xPb(Yb_{1/2}Nb_{1/2})O_3$ solid solution systems have been investigated by measuring dielectric properties, E-P hysteresis loops, X-ray diffraction patterns (XRD) and its transmission electron microscopy (TEM) images. PYT is a highly ordered complex perovskite oxide, which shows two successive phase transitions; The first one is ferroelectric transition with frequency dispersion near 186℃ and the other is the first order phase transition which undergoes from a paraelectric to antiferroelectric at 310℃. The structure of PYT has not been determined exactly yet. The XRD and TEM results of PYT show that its crystal structure is similar to PYN just below the first phase temperature. But two phases - antiferroelectric and ferroelectric phase coexist below the second transition. Below the second phase transition temperature the ferroelectric phase has been identified by the presence of E-P hysteresis loop. XRD diffraction lines due to Pb-antiparallel shift, TEM spots, however, indicated the antiferroelectric phase below the first transition temperature continue to exist below the second transition as well. As the composition of PYN increases, it has been found that the maximum dielectric constant attributed to the second transition decreases gradually such that the antiferroelectric phase is stabilized. But no apparent changes in XRD line positions are observed on PYN substitution. From the selected area diffraction pattern (SADP), an abnormal phase which is thought to be originated from incommensulate in the Nb-substituted solid solution. This may be interpreted as the local strain change due to the ionic size differences of $Ta^{5+}$ and $Nb^{5+}$ ions makes the incommensulate phase that restrains the ferroelectric phase.