Cognitive radio (CR) technology with dynamic spectrum management capabilities is widely advocated for utilizing effectively the unused spectrum resources. The main idea behind CR technology is to trigger secondary communications to utilize the unused spectral resources. However, CR technology heavily relies on spectrum sensing techniques which are applied to estimate the presence of primary user (PU) signals. This paper mostly focuses on novel analysis filter bank (AFB) based cooperative spectrum sensing (CSS) algorithms to detect the spectral holes in the interesting part of the radio spectrum. To counteract the practical wireless channel effects, collaborative subband based approaches of PU signal sensing are studied. CSS has the capability to eliminate the problems of both hidden nodes and fading multipath channels. FFT and AFB based receiver side sensing methods are applied for OFDM waveform and filter bank based multicarrier (FBMC) waveform, respectively. Subband energies are then applied for enhanced energy detection (ED) based CSS methods. Our special focus is on sensing potential spectral gaps close to relatively strong primary users, considering also the effects of spectral regrowth due to power amplifier nonlinearities. The study shows that AFB based CSS with FBMC waveform improves the performance significantly.