> Instead, it should crawl a whitelist of domains, or “tier 1” domains. These would be the limited mainly to authoritative or high-quality sources for their respective specializations, and would be weighed upwards in search results.
Not a big fan of this conclusion. Who chooses the white list, and why should I trust them? Is it democratically chosen? Just because a site is popular very clear does not mean it's trustworthy. Does it get vetted? by whom? Also, who's definition of trustworthy are we trusting?
If I want my blog to show up on your search engine, do I have to get it linked by one of those sites, or can I register with you? Will I be tier 1, or
> If I want my blog to show up on your search engine, do I have to get it linked by one of those sites, or can I register with you? Will I be tier 1, or
I think what I'd say in defense is that we've misunderstood what search engines are useful for. They're really bad at helping us discover new things. Your blog might be awesome, but it's not going to be easy for a search engine to tell that it's awesome. It's going to have to compete with other blogs that also want views, some of whom are going to be better than yours at SEO, and so on.
What a search engine might be able to tell is that it's useful. Because what search engines are at least potentially good at is answering questions. You do that by having a list of known good sites to answer specific types of questions, and looking at the sites they link to. It's when you try to do both (index everything on the web and provide accurate answers to specific questions) that you end up failing to do either. For example this is the #2 result for "python f strings" on DDG[1]. It's total garbage, and, quoting the blog, "we can do better". (This result is also on page 1 for the same query on Google.)
What I believe ddevault is suggesting is that we make a search engine that does the only thing search engines are really good at, answering questions. You throw away the idea of indexing everything on the web, and therefore the possibility of "discovery". What that means is that in 2020 you need some other mechanism for discovering new sites, bloggers, and so on. Fortunately we do have some alternatives in that space.
To be clear, I don't know if I 100% buy this argument, but I think it's the general idea behind what's being suggested in this blog post.
As an experiment, I searched “tech news aggregator” on both google and DDG. Neither listed Hacker News. Instead, apart from a few actual sites, most of the links were articles saying “top ten tech news sites” or links to quora q&a threads.
It definitely seems that search engines can’t find new websites for people. Now they are just aggregating Q&A.
Yeah - this is sort of my thinking. For better or worse if you enter "discovery" terms, what you will get back is not the results of a "discovery" search, but rather an answer to the question "what are some websites that will help me discover X".
> For example this is the #2 result for "python f strings" on DDG[1]. It's total garbage, and, quoting the blog, "we can do better". (This result is also on page 1 for the same query on Google.)
There are other options that may be better, but in general, very few people are looking for them.
I tried this out real quick just to see for myself. First it was slow for me, if I wasn’t wanting to try it out I may have closed the tab, but I did a search for a question I recently googled (that was an example of an annoying google search.) I searched “Python arcade sprite opacity“ and while the first three results were the same as google the fourth was a github link to the project itself and brought me to the answer (which wasn’t on page 1 of google for me, although it was on page two.)
So you need to speed it up, but it does look good for searching documentation at least.
> You do that by having a list of known good sites to answer specific types of questions, and looking at the sites they link to.
I mean, that's basically the core of original Google pagerank, right? A "good" site linking to another site is what makes that other site some amount of "good" too, links from better sites carry more 'juice'. "good" is of course not just binary, but a quantitative weight.
I don't know to what extent that's still at the core of their relevancy rankings. I don't know how all those annoying spammy recipe blogs or content farms get to the top of the results either. I don't think it's because Google's engineers believe they are "good" results.
Relevancy ranking on web search is clearly a hard problem, mainly because so many authors are trying to game it, it's a feedback loop.
If Google can only do as google does despite pouring a whole lot of money into it, I don't see a reason to bet that better will be what's basically an over-simplified description of how Google started out doing it (and then evolved it because it wasn't good enough).
That was exactly my thought on reading that part of the article: that the author was describing Pagerank, but substituting Pagerank's design objective of algorithmically outsourcing quality judgments to the global community of website publishers with the programmer's prejudices instead.
I hope the project of improving on DuckDuckGo is successful though, and some of the other proposals in the article sound promising to me.
In the future I would like to see an open source search engine 'paid for' using some combination of homomorphic encryption, blockchain and Tor-style technologies to trade bandwidth and processing power with its userbase in exchange for search results and other services, but I don't have the expertise to assess how feasible that might be.
the money really is the crux of the issue, too. if you take as a given that consumers won't pay for a service they can get for free, then you're kind of in a bind creating an unopinionated search - if you're truly objective, and reflecting the underlying value of each link faithfully, who do you charge?
This is not about a single search engine instance replacing all your search engine needs.
There could be a community of Software developers running one instance of the OS search engine that focuses on programmers needs: documentation, vcs hosters, dev blogs, tech news websites and on topic blogs.
Great if you need to search for how to solve a software issue, terrible if you need to figure out how long to cook spaghetti.
The lists of crawled pages i guess would be visible/searchable too, at least on instances that you could trust.
If you don't like the list, and the maintainers for whatever reason don't want to change it to your liking, feel free to use another instance or set one up yourself.
Things actually sort of ran that way once. The DMOZ directory system was a cannonical, list of sites by subject listed top-down. It was maintained by a community of volunteers in the fashion of Wikipeda. I believe it was used as one reference for Google and other search engines at one time. I don't know if such an "objective" system could be rebuilt, however.
Still, it's good to remember that it was once uncertain whether people should access the web using something a table of content (portal/directory) or something like an index (search engine). It seems the search engines won.
Great idea, but why build a search engine at all in this case? You can use DDG + your filter and see only the results from your whitelist.
Could easily be implemented for any current search engine.
To a large extent, this is what you already do when you view a page of search results. Filter them based on your understand of what sites / results hold value.
On a public scale, you could make an argument for tighter integration/better privacy with the lists. For example:
Browser -----Request-to-SE-----> Search Engine
^ |
| Unfiltered Results (In YAML/JSON)
| |
| V
|--Desired Results------ Local Filtering/Rendering
On a private scale, if you are only crawling sites on the allow list than you have the possibility of being able to better maintain a local database of sites to show up on the search.
Edit: Possibly this could be easier to use to set up distributed search as well, as each node could index a given list, and then distribute that list similarly to DNS. Don't really know how well that would work though, just an idea.
Aside from this a big reason to build this is it seems a lot simpler than writing a giant web crawler ala google and thus is a good target for an open source solution. Which is the biggest problem with duck duck go.
I would think everyone would run their own “crawler” but maybe you could use a ledger and delegate sites to different workers. If you’re whitelisting sites you could maybe only crawl a link or two deep.
It’d take a lot of cycles while small but if you get a network growing you could even have sub-networks with their own whitelist additions (and every user has a blacklist.)
That’s what directory sites offered once upon a time. It was a pretty good way to discover new content back then. I spent a lot of time on dmoz when I wanted to find information about various topics.
But the email is already like this. It's the inbox providers who choose what domain is legit and new domains start from negative rating. Treating the web the same way doesn't sound too unnatural.
It would be bad if those in the positions profit by "authorizing" who is good though.
I'm not sure why email should be an example of the correct way to do it. And with email I can check my spam folder and see exactly what has been rejected. So unless the search engine has a list of sites that aren't deemed worthy included with every search (which probably wouldn't happen), I think this solution has some pretty big flaws. It should be noted that the current system also has these flaws, as Google and DDG can show you whatever they want base don whatever criteria they see fit.
I like this idea! Have the usual official results... then have an option to go to level 2, level 3, level 4 etc (lvl 1 is not included in lvl 2)
You can have really biased technically terrible filters that for example put a site on level 4 because it is to new, to small and any number of other dumb SEO nonsense arguments. (The topic was not in the url! There was poor choice of text color!)
I think wikipedia has a lot of research to offer on what to do but also what not to do. Try getting to tier 2 edits on a popular article? It would take days to sort out the edits and construct by hand a tier 2 article.
Per your 2nd para Google used to have some options to tailor the results more, like allinurl or inurl or title or link (IIRC the word had to be in a link pointing to that page) or whatever.
I expected that to evolve to get more specificity but things went completely the otherway and we can't even specify a term is on a page reliably with Google now.
Similarly, I was all in on xhtml and semantics (like microformats) where you'd be able to search for "address: high street AND item:beer with price:<2" to find a cheap drink.
I imagine for a FOSS solution we would have to make configurable every separable ranking algo and the option to toggle them in groups as well as build cli like queries around them (with a gui)
I'm starting to see a picture now. In stead of wondering how to build a search engine we should just build things that are compatible. A bit like The output of your database is the input of my filter.
Take site search, it is easy to write specs for with tons of optional features and can easily outperform any crawler. Meta site search can produce similar output. Distributed diy cralwers can provide similar data.
Arguably top websites should not be indexed at all. They should provide their own search api.
The end user puts in a query and gets a bunch of results. It goes into a table with a colum for each unique property. The properties show up in the side bar to refine results (sorted by howmany results have the property) Clicking on one/filling out the field/setting a min max displays the results and sends out a new more specific query looking for those specific properties. New properties are obtained that way.
Yes, I was thinking along similar lines IIUC of a sort of federated search using common db schemas and search apis so that I could crawl pages and they could be dragged in to your SERP by a meta-search engine. I think the main thing you lose is popularity and ranking from other people's past searches - that could be built in but it relies on trusting the individual indices, which would be distributed and so could be modified to fake popularity or return results that were not wanted (though then one could just cut off that part of one's search network).
This is exactly where he lost me. I don't think it is hard at all to find results in "tier 1" domains with DDG. I would argue the we have the opposite problem almost entirely. Besides blogspam / internet cancer and t1 sites you hardly get any results. It's incapable of finding the actually useful communities or blogs for your query.
> Who chooses the white list, and why should I trust them?
That's part of the point. There could be different search engines that run the same code, but have different sets of tier 1 domains that cater to different audiences. And if you have the resources, you could set up your own engine with a set of tier-1 domains that you chose.
Instead of having many bots do inefficient crawling, web sites should publish their own index. Intermediate parties can combine indexes of the sites. Sites that do not provide indexes get less visitors.
That's the best way to fill your search engine with spam. There needs to be a third-party that verifies that the site-provided index is inline with the actual content. At which point said third-party can be a crawler.
Not a big fan of this conclusion. Who chooses the white list, and why should I trust them? Is it democratically chosen? Just because a site is popular very clear does not mean it's trustworthy. Does it get vetted? by whom? Also, who's definition of trustworthy are we trusting?
If I want my blog to show up on your search engine, do I have to get it linked by one of those sites, or can I register with you? Will I be tier 1, or