Recently, a great amount of information has been generated and stored through computer, TV, Internet, DVD, surveillance systems and digital libraries. This information has been transformed into image-based forms, which require a number of the storage devices. In such a situation, people have difficulties in finding what they want to see and collect. Especially, people are interested in meaningful events such as sports highlights or specific events captured in surveillance cameras. So the automatic video indexing and retrieval is an essential technique in video applications.
A video can be hierarchically abstracted according to its syntactic features and semantics. There exist video semantics such as event or scenario in the highest level. They are represented by syntactic features like editing effects (e.g. cut and dissolve) and key frames (or shots) in the lower level. The lowest level consists of scene features (motion, color, shape, and texture) of key frames or partitioned shots. Although extensive research has been performed on the video hierarchy, the work related to each level was studied independently one another. In other words, shot boundaries, key frames, and their features was not unified to describe higher semantic level. In the specific case of sports videos like golf, a drive shot (full-swing) of a player tends to be captured into two different scenes because one camera cannot cover the whole volume of space and time made by the fast moving ball. Moreover, the different camera motion tracking the ball provides an important cue to represent the event. So, the key frames generated by discontinuous camera motion could efficiently summarize the event, and also scene features at those key frames can be used to index the event. In this paper, we address the abstraction of video events, especially sports (golf) highlights. The key frames together with shot boundaries provide temporal interest points (TIP) for event representation.
In the meanwhile, the detection of shot boundaries (such as dissolves) and key-frames is commonly suffering from temporal scalability. In other words, the length of dissolve transition is not unknown and different sequence to sequence. The key frames are differently identified by camera motion discontinuity according to the velocity and acceleration of motion. Considering the facts mentioned above and providing the efficient (and temporal-scalable) abstraction of events, we propose the motion tolerant shot boundary (or length) detection, the temporal-scalable detection of key-frames, and feature-based event descriptor to cluster similar events.
As a prior step for event abstraction, a video has to be decomposed into smaller segments called shots. However, camera and object motion would result in false or miss detection of shot boundaries, which is still a challenging problem. Moreover the length of dissolve transition is not known and different sequence to sequence. In this paper, the motion-tolerant shot boundary (or length) detection for cut and dissolve is proposed using regression analysis. The luminance difference ?between image blocks (8x8) reveals different scattering (distribution) for dissolve, cut, and motion frames, respectively, which results in distinct regression characteristics. The length of dissolve can be known from the proposed correlation coefficient used as a measure for dissolve detection.
To efficiently summarize an event, we extract temporally scalable key-frames (referred as TIP hereinafter) corresponding to motion discontinuous instants which human usually gives attention to. The scalability deals with the velocity and acceleration change of motion. In the temporal neighborhood determined by the scale, the scene features (motion, color, and edge) are used to define event descriptors. By applying k-Nearest Neighborhood algorithm to the feature vectors, we retrieve similar golf highlights (putting, drive and bunker shots) that correspond to a query event. In addition to golf highlight detection, we also apply the same framework (using human hand trajectory, not camera motion) to human actions occurring in an office environment. We show the proposed event abstraction is suitable for sequences which are depicting events by camera motion or object trajectory.
방대한 분량을 가진 비디오 데이터를 효과적으로 검색하기 위해 비디오가 가진 특징량 및 의미 있는 내용을 추출하여 색인화 하는게 매우 중요하다. 본 논문에서는 동영상이 연속적인 부분과 불연속적인 부분으로 구성되어 있다는 가정하에, 불연속적인 부분을 색인화하여 비디오 요약및 의미 있는 사건에 대한 추출에 사용한다. 드라마와 같은 정적인 동영상의 경우 특징량 (움직임, 색상)이 가장 불연속적인 부분은 장면이 전환되는 시점이고, 한 장면 내에서는 그 특징량은 크게 변하지 않고 연속성을 띠고 있다. 따라서 각 장면의 대표 프레임 (예를 들면 경계 프레임) 하나가 장면의 내용을 충분히 기술할 수 있다. 한편, 골프와 같은 스포츠 동영상의 경우에 있어, 장면 전환 시점이 가장 불연속적인 부분이지만, 많은 장면이 카메라 움직임에 의한 강한 특징량의 변화를 가지고 있다. 따라서 하나의 프레임으로 한 장면을 효과적으로 담아낼 수 없으며, 여러 개의 대표 프레임이 필요하다. 이를 위해 본 논문에서는 시간상의 움직임의 속도 변화에 따른 대표 프레임 (Temporal Interest Points, TIP)을 움직임 정보 (motion curvature)를 이용하여 추출하고, 이 대표 프레임 주변의 특징량 (motion, color, edge)을 장면의 이벤트를 기술하는 데 사용한다. 본 논문에서는 동영상의 가장 불연속적인 부분인 장면 전환(dissolve 와 cut)을 강인하게 (motion-tolerant) 찾아내고, 분리된 장면을 TIP에 의해 효과적으로 요약할 수 있음을 보인다. 또한 TIP에 의한 이벤트 색인 및 검색을 골프 비디오에 적용하여 드라이브 샷과 펴팅 샷을 구분할 수 있음을 보인다. 특징량 중에 움직임과 경계 정보는 골프 하이라이를 특징지우는데 큰 역할을 하고 있으나, 색상 정보는 날씨에 따라 변하기 때문에 때로는 성능을 떨어뜨리고 있다. 본 논문은 카메라의 움직임이나 객체의 자취에 의해 이벤트가 기술되는 데에 적합하며, 비관리 학습에 의해 이벤트 검색을 하는 특징을 가지고 있다.
