Kalki. The smallest search engine in the world. Update: Dec 29th. New Orange Black Color scheme.

This is easy to code but the issue is that it would be very cost intensive to manually add and update all these search results.

I have been thinking hard about this. You are completely right it could be expensive. Partial solutions:

- You don't need to index the whole web. Most people look at the first page of results only.
- You could laser focus on vegetarian food, alternative health, conspiracies, meditation ... all the stuff Google is de-indexing.
- You could also localize by region. i.e. make a good local search engine for Taiwan.

*

I have been researching. You could pull results from a mix of APIS. Google, Twitter, Bing and others would only cost you 400USD per month. All the big APIs also have free layers.

[link to blog.api.rakuten.net (secure)]

If I had a search engine with 1000 customers I would be the happiest man in the universe.

You could get volunteers to help with the adding and updating. A mix of human help, commercial APIs, and niche-indexing might work.

Last Edited by Daozen on 07/01/2020 07:11 AM

If you want to set it up as an open source project with volunteers i don't see any problem with it. As long as you get enough people to help you.

Yes, open source with volunteers, that could be fun. And remember, Ecosia run a thriving business with only a browser/iOS/Android app. All their results come straight from Bing IIRC.

*

I was thinking, what if you got a volunteer sysadmin type guy to index a tiny niche on his home server? i.e. buddhist vegetarian restaurants in Danshui, Taiwan. There are literally 50-100 buddhist restaurants in this city.

What would be the issues when you tried to retrieve that query? Would it take too long? I guess speed, uptime and servers crashing are the big issues here.

I'm thinking of a few casual volunteers running a distributed search engine.

A distributed search engine is a search engine where there is no central server. Unlike traditional centralized search engines, work such as crawling, data mining, indexing, and query processing is distributed among several peers in a decentralized manner where there is no single point of control.

[link to en.wikipedia.org (secure)]

Can Distributed Search Engines Challenge Google’s Dominance?

[link to hackernoon.com (secure)]

The biggest problem would be the human factor. One day you wake up and the first page is all swastikas because some neo-Nazi subgroup thought it would be funny. The next day all results are Rick-rolls or affiliate links.

Any website that accepts user input faces these challenges. That's why we have weird word filters and strong mods here on GLP.

Haha. I can see how that could easily happen. Thanks for bringing that up.

You could have search rankings manually approved before going live... That could be done on 2-3 hourly basis.

You could have keyword filters, peer oversight and strong mods just like GLP.

Still thinking about this. Looking at the Yandex XML API right now.

The issue with that is: "how to scale?". One "trusted" person could probably approve a few thousand per day. You could use machine learning to approve, but you would need to train the models.

To me, the better choice seems to be more localized, more distributed. Groups of local users that mostly know each other maintain their own indexes. And then users could belong to multiple groups, group approved indexes could be shared across networks, etc.

Much more thought needed on it, but these are the types of things I have thought about over the years.

They used to have stuff like this in the early 1990s but then the big search engines came along.

Finally, google came and started offering people free stuff and took over the marketplace and now you can't get rid of them.

Fuck, I thought that the title said full ground and I got a chubby.

I've been thinking about scaling too. Here are my thoughts.

- Go local at first. Market it as a TW only search engine, for example.
- Make sure the app works as a standalone browser app. Don't worry about servers for now.
- Draw results from Bing, Google and other search APIs at first. Gradually mix in the hand curated results over time.
- More localized, more distributed, is a good mantra.
- Give ourselves a generous approval time for index approval. Try and hit index approval within 48 hours as a target... but publicly ask for up to a week.
- Don't worry about speed, focus on quality.

OK, but look at it this way.

MacDonalds is a fast food behemoth, but there's still plenty of room for Subway and Mosburger.

Subway and Mosburger are relatively huge, but there's still room for one off indy restaurants like Kooks, Danshui.

The search market is so staggeringly huge, that even a tiny little bit of it could make a viable business. You're not really looking to challenge Google, just make an excellent "Mom and Pop" search engine for 10,000 happy customers.

You don't need a huge corp to quietly thrive. I work for a company with about 50 staff. We're never gonna be featured in Techcrutch... and no, I don't like all my colleagues, but I daresay we're happier than most big companies.

There's a few interesting alternative search engines here:

[link to kinsta.com (secure)]

The first 5-10 are fairly well known to search-tards, but it gets interesting later.

*

Infinity search works, and is open source:

EDIT: Infinity search is really good. Their image search CSS is lush.

Working site here:

[link to infinitysearch.co (secure)]

[link to gitlab.com (secure)]

They have an API and a good business model:

[link to infinitysearch.co (secure)]

[link to rapidapi.com (secure)]

Last Edited by Daozen on 12/03/2020 03:09 PM

Meta-search engines are probably the way forward. They used to be too slow, but the web is getting quicker. More than one company has made a business just making a secure, private portal for Google.

[link to en.wikipedia.org (secure)]

[link to www.startpage.com (secure)]

[link to searchengineland.com (secure)]

[link to searchengineland.com (secure)]


Google’s most serious challenge until now, in my opinion, really came on January 1, 2011. In the weeks before, there had been grumblings that “content farms” were somehow walking all over Google’s results, serving up lightweight content to answer common questions. On New Year’s Day 2011, Vivek Wadhwa published a column about why a better Google was “desperately” needed.

The column was an over-the-top condemnation of Google’s search quality backed by no actual metrics. Google was clearly serving hundreds of millions of searches successfully per day, or its actual users would have been abandoning it in droves. They were not. But Wadhwa’s column resonated with tech bloggers who for various reasons just felt in their gut that Google had a problem.

Google again went to red-alert status. Within two months, it launched what was called the Panda Update, a change especially meant to go after content farms and low-quality content. The normals relaxed, assuming all was fixed. Meanwhile, search marketers watched as Google rolled out nearly 30 subsequent updates over a four-year period to get a handle on the issue.

One thing to keep in mind, if you are actually trying to build something that allows people to escape censorship (can't believe I have to say that in the U.S. now, but I do), going the metasearch route would seem fine at first but once you get any kind of notoriety or market share, the upstream would cut you off without hesitation.

Startpage, DDG, etc., all have either deals in place with Google, et al, and/or they have enough connections and power to keep the big guys in check and not allow them to pull out the rug just yet.

If you just want to make money, sure, it's worth a try, but there is already some good competition out there.

My input is farming out to existing search engines as a supplemental, for the searches that you cannot yet service, that's fine. And over time, in theory, the surface area of what you service vs. what you need to farm out continues to grow.

Many of these fast food restaurants are actually all owned by just a few companies, even though they appear to be different. The goal is of course control, which is what google wants.

You need someone that can program the algorithms with censorship command keywords via a pretty hefty AI, you simply cant regulate it with mods as there is literally billions of searches each day, youd need a "clear deck" on the ship as trinity terms it, and a good portion of self control as to not skew and bias the AI crap filter.

Lots of wisdom here. I'll take in what you say. I know that hand modding large volumes of searches is unworkable. But I still think there will be an element of hand/manual curation.

So there'll be a split: 80% AI, 20% human... or even 98% AI, 2% human. There'll be some happy ratio somewhere.

One thing to remember: The incumbents (Google etc) are not particularly good any more, as they have been memory-holing huge chunks of data like crazy.

So yeah, it'll be chaotic and messy. But you just have to wade in.

*

In other news, I've just discovered RAPIDAPI. That looks like a lot of fun.

Last Edited by Daozen on 12/03/2020 03:43 PM

QUOTE: once you get any kind of notoriety or market share, the upstream would cut you off without hesitation.


Do you mean the APIs I'll be using will no longer make their data available? I can see that happening. It's all part of the fun, I guess.

I'm really more into making a beautifully designed front end, and then make it mission focused. I'm going to promote useful local projects.

I plan to innovate mainly on the CSS and product side. That doesn't take so much programming knowledge, just some brainstorming and market testing.

Alright, got an API key from Bing/Microsoft Azure.

It's returning a 401:

Error
HTTP Status 401 PermissionDenied

error: [object Object]

[link to codepen.io (secure)]

Anyone have any idea why it's not working?

DMOZ is a hand curated directory/search engine. Or was, not sure if it still exists.

Hand curated was essentially the yahoo! model in the early days.

Im going to have a combination of hand/crowd curation and AI.

Assuming page one has 20 search results:

I'll get the bottom 15 results from Bing or someone else. That's the filler.

Top 5 will be hand curated or staff-picks.

So it'll be a mix.

*

For all their algorithms and machine learning, Google hand curate a LOT. They've manually deranked thousands of sites with good info.

Last Edited by Daozen on 12/06/2020 12:38 PM

They're not around any more... I'll check them out. Thanks for the tip.

I saw a medium article on crowd curated image search results. They really improved the results.

Last Edited by Daozen on 12/06/2020 12:38 PM

Asking on Stack Overflow. Maybe they've updated their API.

[link to stackoverflow.com (secure)]

Sorry OP, but "curating" of the search results is exactly the reason Google sucks.

Here's the recipe for a proper search engine that is actually useful for people:

1. Index everything.

2. Find the thing the user is looking for

3. Show the thing to the user

Any form of "curating" means you insert a layer of control between the user and the thing he or she is looking for. There are no honorable reasons to want this kind of control, and plenty of dishonorable reasons to want it.

Build a "curated" search engine, and it will soon be infiltrated and taken over by power-hungry psychopaths, 100 % money back guaranteed. You might as well save the effort and just use Google.

I'm more of a power-hungry narcissist. Humanity will be much better in my hands.

You're right that you should try and be as objective as possible. But at some point, even the most honest person in the world has to make choices. Like, what do you show as the first result for a user search of "Paris, Restaurant" ?

Who gets to the top of that query?

You come off as idealistic, but I don't think you're factoring in the day to day choices that search engines have to make.

Re: "curation", you can ask the crowd to help sort out badly indexed image searches, for example. There's nothing insidious about this.

I hope I can do better than Google, anyway.

Last Edited by Daozen on 12/06/2020 01:55 PM

My position has nothing to do with idealism. In fact, it is THE OPPOSITE of idealism, since it is idealism that leads to people wanting to "curate" search results, and I want none of that.

The thing is, everything in this world comes down to Capitalism vs. Communism.

Capitalism = decentralize power to the consumer

Communism = centralize power to the controlling elite

It is my firm belief that ALL attempts to build "Utopian" solutions based on centralized power are doomed to fail miserably and collapse in a ruinous heap from rampant corruption and destructive fascist insanity (all of known history serves as evidence to this point!).

The only way to build a stable and efficient solution to anything, is to decentralize it and let "the market" (people) sort it out as it may.

Curating search results = centralized power over information. A group of people will then enforce their ideals and opinions (and other selfish interests), and as I tried to explain, it is always the worst of humanity that are drawn to such power. Good people like to live and let live, only psychopaths seek power over others.

The paradox here, is that Google used to do the right thing! They used to run their search engine AS A BUSINESS!

This means, they attempt to provide users with as helpful results as possible (giving them what they're looking for!), and they also sold the top result spots to whoever paid for this particular search result.

This worked!

If a business finds it worthwhile to advertise their existence by paying to be listed on top when someone types "Paris, Restaurant", then that business is very likely to be a restaurant in Paris, and it's even likely to be one of the biggest and most most relevant restaurants. Meaning, the user is very likely to want to see this link!

Having businesses pay to be listed at the top is actually fair, because any business can do so. There is no censorship or "curating" involved, no clique of humans decides, THE PEOPLE decide by voting with their wallets (if people don't like the top link the business that paid for it will have paid for nothing, and then another business can outcompete them if they don't stop wasting money.).

And the rest of the results (under the paid ones) were Google's automated attempt to find the most relevant restaurants in Paris (the ones people would most likely want to be told about). They did a pretty good job giving as good results as possible, because once again, if a capitalist business doesn't serve its customers well, they'll go somewhere else.

So Google was actually a very good search engine, and that's the reason people flocked to it in the first place!

This only changed when they started to "curate" search results. That's when Google turned into a fascist censorship shithole that forces communist propaganda down your throat whether you want it or not.

And so people are looking for alternatives! Simply because Google isn't serving them good search results anymore.

If you wish to be one of the alternatives people prefer over Google, you'd better offer uncurated search results that deliver what people are looking for, not what a group of controllers wish to feed them. Otherwise your new search engine serves no purpose in the market - there already exists a Google, corrupt search is already cornered!