Sources in Nepal, the current Chair of the South Asian Association for Regional Cooperation, confirmed having received messages from the four countries conveying their inability to attend the summit due in Islamabad on November 9-10.
The sources said that the Himalayan nation is yet to take a decision on calling off the summit, which is over a month away.
With the Saarc process based on consensus, even if one member does not attend the summit is automatically postponed or cancelled.
However, a final decision rests on the current Saarc Secretary General Arjun Bahadur Thapa, who is currently in New York and will return in two days.
With half of the eight-nation regional bloc deciding to skip the event, there is every possibility of the event being postponed, in a big diplomatic snub to Pakistan.
Sri Lanka has already said that the Saarc Summit would not be possible without India's participation.
India on Tuesday announced that it was pulling out of the Islamabad summit in the wake of the September 18 cross-border terror attack on an Indian Army base in Uri town of Jammu and Kashmir, which New Delhi has blamed on terrorists based in Pakistan.
The Uri attack came amid the dragging unrest and violence in Kashmir Valley that has left around 90 people dead since the killing of Hizbul Mujahideen terrorist Burhan Wani on July 8.
In a statement on Tuesday, India said increasing cross-border terrorist attacks in the region and growing interference in the internal affairs of Saarc member-states "by one country" have created an environment that was not conducive to the successful holding of the 19th Saarc summit.
Ministry of External Affairs spokesperson Vikas Swarup said that India has conveyed to Nepal its decision not to attend the summit, for which Prime Minister Narendra Modi was scheduled to go.
"India remains steadfast in its commitment to regional cooperation, connectivity and contacts but believes that these can only go forward in an atmosphere free of terror.
"In the prevailing circumstances, the Government of India is unable to participate in the proposed summit in Islamabad," he said in the statement.
Pakistan has termed the boycott decision as "unfortunate".
In its message to Nepal on Tuesday, Afghanistan said that because of "increased level of violence and fighting as a result of imposed terrorism on Afghanistan, President of Afghanistan Mohammad Ashraf Ghani with his responsibilities as the Commander in Chief will be fully engaged, and will not be able to attend the summit".
In a similar message to Nepal, Bangladesh said: "The growing interference in the internal affairs of Bangladesh by one country has created an environment which is not conducive to the successful hosting of the 19th Saarc Summit in Islamabad in November 2016."
"Bangladesh, as the initiator of the Saarc process, remains steadfast in its commitment to regional cooperation, connectivity and contacts but believes that these can only go forward in a more congenial atmosphere," the message said.
"In view of the above, Bangladesh is unable to participate in the proposed summit in Islamabad."
Bhutan, in its message, cited "the recent escalation of terrorism in the region" as the reason for its non-participation.
