The so-called "extra-budgetary funds" include revenues to local governments from land deals.
The governments should release information about all public spending, spending on infrastructure, the administrative expenses budget and its implementation, Wen said in the speech.
Early this year, some ministries and central government departments published their budgets from last year and implementation reports of the budgets.
"We have received good feedback and many suggestions (about budget transparency). We should keep improving our work," Wen said.
He promised that all government information, which is required to be published by law, should be made public promptly, precisely and in detail.
He said transparency was key to lawful governance.
Wen said the task of building a lawful government was an essential aspect of the reform of the country's political system.
"Our country's economic and social development has entered a new phase. More challenges have emerged as domestic and overseas environments become more complicated. We should deepen reform while promoting administration in accordance with the law in order to solve new problems," he said.
City and county governments, in particular, should improve their awareness of lawful governance as they dealt with the public on the daily basis, he said.