Science is absolutely dependent on luck

"As people dredge the unknown, they are engaging in a highly creative act. What an inventor “finds” is always an expression of him- or herself. Martin Chalfie, who won a Nobel Prize for his work connected with green fluorescent protein — the stuff that makes jellyfish glow green — told me that he and several other Nobel Prize winners benefited from a chain of accidents and chance encounters on the way to their revelations. Some scientists even embrace a kind of “free jazz” method, he said, improvising as they go along: “I’ve heard of people getting good results after accidentally dropping their experimental preparations on the floor, picking them up, and working on them nonetheless,” he added."

Observational science is rather different from the hypothesis->experiement->verify->refine/disprove cycle taught in schools. It makes sense to have some idea you want the data to test, but planning anything that's more than vaguely defined is (as a rule of thumb) foolish. Too often the data will just show something you didn't expect at all. Better to approach the data with as few preconceptions as possible and try and determinine what it's telling you without bias. You could try speculating about as many different scenarios as possible but it's usually a waste of effort that ends up being castles in the air. So in that sense, observational astronomy is a great example of an inherently serendipitous process.

Of course, there are exceptions. If you're building a new instrument you need to have some idea as to what it should detect - you can't go spending millions of currency units on some new-fangled telescope unless you're pretty sure it's going to detect something. But more often than not, once you've established that you'll detect something, you let the data tell you about the universe, rather than trying to test a specific hypothesis. Hypothesis testing is useful and has its place, but it is not the be-all and end-all of science.

Which is a very roundabout way of saying, "I don't like writing telescope proposals, they're silly".

EDIT : I was also going to add that with regards to fostering an environment that promotes serendipitous discoveries, I've already written about that here : http://astrorhysy.blogspot.cz/2015/11/when-worlds-collide-science-in-society.html. Relevant quote :
These similarities mean that sometimes the process of doing both science and the arts can be very similar. Both require large amounts of time to do nothing but thinking (and in the case of science at least, an awful lot of background reading). Inspiration can't be forced - you cannot make people have new ideas. You can, however, encourage them. Science and art are both sometimes highly elaborate forms of play, to explore the question, "what if I did it this way...", or better yet, "what does this button do ?" Such thinking intrinsically demands a liberal, reasonably informal atmosphere. Insisting that people are at their desks during some particular set of hours and only talk to each other during scheduled meetings makes absolutely no damn sense whatsoever.

Except that I have a nice personal counter-example. I recently came up with a way to view volumetric data in spherical polar coordinates directly in Blender with no need for Cartesian gridding... during a long and extremely boring meeting to which I was not paying any attention. So perhaps seemingly pointless meetings have their uses after all.

Via Kazimierz Kurz.
http://www.nytimes.com/2016/01/03/opinion/how-to-cultivate-the-art-of-serendipity.html?_r=0