Changes to benefits payments have led to difficulties letting affordable homes and may stop hundreds more being built every year, research suggests.
Community Housing Cymru (CHC), which represents housing associations, reports problems letting 700 homes.
It is claimed a cut in the spare room subsidy from housing benefit means some tenants do not want larger properties.
The UK government said it could not afford "to pay for people to live in properties larger than they need".
Ministers reduced entitlement to housing benefit for working age tenants in April.
Those with one extra bedroom had a reduction of 14% to their eligible rent and those with two or more extra bedrooms, a 25% cut.
A CHC survey of its member housing associations aimed to assess the impact of the spare room subsidy, called a "bedroom tax" by some critics.
CHC estimated the cost to its members in the first six months since the changes was more than £1m, which it said would impact on future investment.
It said the majority of its members had seen an increase in rent arrears as some tenants had struggled to pay.
Housing associations report having 700 houses that have become difficult to rent as prospective tenants do not want larger properties because their benefits will not cover them.
The research also found 51% of tenants were paying the shortfall in rent while 49% were paying only part or none of it.
The lost income from empty properties and the arrears in rent will impact on housing association investments.
If the £1.1m that has been run up in six months was to continue at the same rate, it would mean 400 fewer affordable homes being built a year because rent income is used by housing associations to borrow money and invest in house building.
Housing associations warned it would have a wider impact on businesses in Wales, including the construction industry.
Nick Bennett, CHC chief executive, called for welfare to be devolved to ministers in Wales.
"We note what's been happening around Wales with various protests against the 'bedroom tax', with landlords getting the blame," he said.
"Tenant groups and some politicians have campaigned for a 'no bedroom tax evictions' policy, but landlords can't continue to subsidise the extra costs brought about by welfare reform which would impact on the potential reduction in services for other tenants who pay their rent.
"We need to stand back and realise that the only solution is to take power closer to the people."
UK ministers rejected the criticisms of the policy.
A spokesperson for the Department of Work and Pensions said: "Our reforms restore fairness to a system that was allowed to spiral out of control.
"Even after the removal of the spare room subsidy we still pay the majority of most claimants' rent, but the taxpayer can no longer afford to pay for people to live in properties larger than they need.
"It is right that people contribute to these costs, just as private renters do.
"We have made £6.2m available to Welsh councils to support vulnerable people, with an additional £880,000 available to help those in rural parts of Wales."