Hadith: He came to the Black Stone and kissed it and said: Verily, I know that you are a stone that can neither benefit nor harm. Had I not seen the Prophet (may Allah's peace and blessings be upon him) kissing you, I would not have kissed you.

عن عمر بن الخطاب رضي الله عنه «أنَّه جَاء إِلى الحَجَر الأَسوَدِ، فَقَبَّلَه، وقال: إِنِّي لَأَعلَم أَنَّك حَجَرٌ، لا تَضُرُّ ولا تَنفَعُ، ولَولاَ أَنِّي رَأَيتُ النبيَّ -صلَّى الله عليه وسلَّم- يُقَبِّلُك مَا قَبَّلتُك».
[صحيح] - [متفق عليه]

‘Umar ibn al-Khattāb (may Allah be pleased with him) reported that he came to the Black Stone and kissed it and said: "Verily, I know that you are a stone that can neither benefit nor harm. Had I not seen the Prophet (may Allah's peace and blessings be upon him) kissing you, I would not have kissed you."
[Authentic hadith] - [Narrated by Bukhari & Muslim]

Explanation

Places, times, and other things are not considered holy or glorified for the worship of Allah for their own sake, rather they are considered holy by virtue of religious legislation. Hence, ‘Umar ibn al-Khattāb (may Allah be pleased with him) went to the Black Stone and kissed it amongst the pilgrims, who had recently abandoned the worship and glorification of idols and embraced Islam, and he clarified that he did not kiss or glorify this Stone out of his own accord or because the stone could benefit or harm him. Rather, it was an act of worship that he learned from the Prophet (may Allah's peace and blessings be upon him) as he saw him kiss it, so he did it out of following his example, not out of his personal opinion or as an innovation.