The study, by market research firm Kline & Company, said that the market situation has provoked a move towards customer loyalty programs and intense selling techniques.
Sales of cosmetics and toiletries rose by only 3 percent in North America last year, and with online cosmetics retailing growing at the rate of 26 percent a year in the US, beauty retailers are fighting for fewer and fewer in-store sales.
Retailers such as CVS, The Body Shop and Ulta are offering customer loyalty cards and purchase rewards to retain customers and attract new ones in an overcrowded market.
The big stores are reaching out to customers in what the Kline study calls a 'multi-channel' approach so that a trip to a store is no longer necessary when buying beauty products.
Instead of just the traditional in-store approach, online, direct mail and telephone contact are all being used by stores to persuade people to buy more cosmetic goods.
In order to lure in the consumers retailers are personalizing their selling tactics.
This may involve aggressive selling techniques such as the hosting of 'members-only' events and personal phone invitations from in-store beauty consultants, or purely information based services that will feed sales indirectly.
"Beauty product retailers are using more personalized approaches and more frequent, targeted communications to develop tighter bonds with their customer base," said Carrie Mellage from Kline's market research division.
She pointed to NordstromBeauty.com as an example of the personalized service that customers will come to expect.
The site offers a toll-free beauty hotline that customers can call to speak to consultants about product lines.
These new approaches are not expected to ignite an upward burst in sales but individual retailers may have to adopt them in order to survive.
The Kline study anticipates annual growth in beauty product sales in department stores to be 3 percent over the next few years.
Mellage added: "In a market where only modest growth is expected, retailers are learning that price is not the only motivating factor for customers.
They want service, rewards, atmosphere in exchange for their beauty dollars."
The findings of the Kline study support the conclusions of a report published in April on using the internet to sell cosmetic goods by the market research firm Forrester.
Given that female customers are using the internet to exchange information and views about cosmetic products, the report implored beauty retailers to develop their internet presence.
The report suggested that beauty retailers 'initiate a conversation' with customers by e-mailing them with the latest product news and giving buyers the opportunity to chat on-line with beauty consultants.