Rory Stewart spent yesterday afternoon talking to people in Chrisp Street Market in Poplar, designed by Frederick Gibberd and built as part of the Festival of Britain in 1951.

Canary Wharf is four minutes away on the Docklands Light Railway, and huge office blocks bearing the words “BARCLAYS” and “HSBC” are visible in the middle distance.

When I arrived, Stewart was already there, courteous, smiling, unhurried, listening carefully to people, responding to their civility with his civility, and all the while being filmed so that clips of these encounters could be disseminated via social media.

Rather amazingly, he met a distant cousin of his who has worked at an accountant in Poplar for the last 33 years. Their common ancestor, if I followed the conversation right, was James Wilson, Minister of the Church of Scotland in Montrose.

An old lady came up to him and with a secret smile whispered something in his ear. I asked her what she had said, and she whispered in my ear: “Sling Boris out!”

A woman who had come from Greenwich to meet him said: “I’ve yet to hear anyone ask you a question you didn’t answer.”

She told ConHome: “I didn’t vote Conservative last time. I would only vote Tory if Rory Stewart is Prime Minister.

“I’m appalled by Conservatives who say he’s not one of us. He’s absolutely Conservative. He’s fiscally Conservative.”

Stewart was by now on his knees, making friends with two shaggy dogs, Schnauzers, who live at Ivy’s Cafe, opened in 1951.

A woman said to him: “The issue of legalising cannabis – the issue is if you’re in social housing it can be very disturbing.”

She has friends with a small baby who have to put wet towels along the bottom of their door to stop the cannabis smoke coming in.

In a grave tone, Stewart discussed the “muddled logic” of drugs policy, and the problem of heroin addiction in prisons.

A common denominator soon emerged among the people who approached him. Many had voted for several different political parties within the last few years. A man aged only 22 had already supported Labour, the Conservatives, the Lib Dems and the Greens in various national and local elections.

These floating voters are deeply interested in politics, have found no anchorage and believe Stewart might be worthy of their trust.

A Cypriot man aged 75, who came to England in 1960 and worked as a nurse, said his son, who has taken a degree in classics at Magdalen College Oxford, had urged him to come and see Stewart.

With evident pleasure, Stewart asked him about a Greek Orthodox Patriarch, to whom he asked his good wishes to be conveyed.

“It’s been a bit of a desperate few years,” two young civil servants told Stewart. They had been Labour members, but had left the party.

A tremendously indignant lady in a headscarf came up and told Stewart “all the young men are jacking up”, injecting themselves with heroin, in the public toilets, all the toilet doors are broken and they use the ladies’ toilets because they’re less likely to get caught there.

“Do you want to show me?” Stewart said, for he likes to go and see things for himself.

We walked the short distance to the toilets, which were indeed in a filthy and dilapidated state, impossible to get clean unless they were renovated. Stewart said later: “It makes me ashamed to be British. I want to be able to take my son to the loo.”

Craig Cross, a banker by profession, who became a fan of Stewart in 2014 during the campaign to preserve the Union of Scotland and England, asked me: “How do you convince other Tory MPs that he has such cross-party support?”

Cross has sent emails to 310 Tory MPs asking them to support Stewart. He has had replies from 40 to 45 of them, none of whom has said that he or she will vote for Stewart.

“The debates are going to be great on Sunday,” he said. “I think it will be fantastic. It will show up Boris Johnson if he doesn’t turn up.”

A small boy arrived and leapt into Stewart’s arms: his son, Sasha, aged four, brought along by his wife, Shoshana. All three looked at ease in Poplar. They have a second son, Ivo, aged two, who was being looked after by Shoshana’s mother.

Stewart said of this form of campaigning, when he goes out and talks to people: “I really, really like it. Because I suppose one of the reasons why I’m a Tory is I’m an empiricist, not a rationalist. I believe in real places, real things, going to look at the toilets.

“I’m fascinated by the incongruity of things. Nothing is quite as it seems.”

He described his last-minute efforts to canvas in the Committee Corridor on Thursday, before the first ballot, in which he got 19 votes, just enough to stay in the race.

Stewart looked MPs in the eye: “You keep telling me you want me to be Prime Minister, your daughter would vote for me, and then there’s always a ‘but’ at the end.”

If Stewart were to get into the final two – something which seems mathematically impossible at the moment – he would be a formidable performer on the hustings to be held in front of party members.

“There is a potential for unlocking a Disraeli Young England conservatism,” he said.

“Young England was a rather aristocratic movement,” I remarked.

He accepted this, but went on: “They can relate to me in a way they can’t with the others.”

Some tipsy men outside the Festival Inn spotted Stewart and shouted in a joky way: “Mum, I’m on TV!”

And then a man wanted to take Stewart to see something. As we walked down the street, the man began telling a horrifying story about the murder five years ago of his brother.

A policeman ran across the end of the street. A few seconds later, a police car sped in the same direction.

We reached a patch of grass, heard a loud engine and as we came round the corner of a block of flats saw a red air ambulance with its rotors turning, seemingly about to take off.

A hundred yards further on a police tape barred the way, and a small white tent had been erected.

A bystander told Stewart a man had been stabbed: “It took them 30 minutes to come. He was still breathing.”

The man had since died, and the helicopter did not take off for some time.

“I hope the Conservatives will do something, man,” the bystander said. “It’s the middle of the day, and we feel unsafe with our kids and our children.”

A young Bengali woman in a headscarf said: “Murder has become such a normal thing. We don’t feel safe at all.

“Little schoolboys carrying knives. It’s like a fashion, killing each other.”

She listed several recent murders: “There should be more security. More police around. If you call them they don’t come in time.

“The man that died in Ramadan was only 22 years old, the only son of his mother. How painful that was for her.

“The police should search everybody. It’s not about racism. It’s about security. I have white neighbours, black neighbours, I’m friends with them.”

She has been married for four years but does not want to have children: “I don’t want to give birth in this kind of area.”

As we walked back to Chrisp Street Market, Stewart said: “The cliché that the basic function of the state is to keep people safe is absolutely true.”

He added that a visit like this “means that my whole view of priorities changes”.

We met a middle-class man who had just picked up an Amazon parcel, and has lived in the area for three years. He said “it’s really quiet at times”, and wished Stewart well.

Three lads with bikes lounged on a concrete table-tennis table as they ate ice creams. They wanted us to feel vaguely intimidated by them, and were not on for a chat.

When we were past them, Stewart described them as “mini-gangsters”, and observed that crime did not figure in the life of the man with the Amazon parcel.

We went into Eastenders Pie ‘n’ Mash shop, where his team had arranged for him to be filmed answering questions live on Twitter and Facebook.

“It’s actually been a slightly disturbing afternoon,” he began.