But the housing minister, Brandon Lewis, said the government had "got the country building again" and was "delivering the homes that hard-working people rightly deserve".
Shelter's concern arose from official housing statistics for England which showed the number of foundations laid down fell to 33,280 in the three months to June, a fall of 14% compared with the previous quarter.
The annual figure, comparing the latest 12 months to the year before, was 1% lower at 136,320.
However, the figures also show that the number of new homes actually completed over the latest 12 month period rose by 15% to 131,060.
The National Housing Federation, which represents housing associations, said the news was encouraging.
Before the election David Cameron predicted that we would be building 200,000 homes a year by 2017.