No OneTemporary
Actions

Size

14 KB

Referenced Files

None

Subscribers

None

View Options

	diff --git a/docs/configuration/search.md b/docs/configuration/search.md
	index a785a18ad..82217e5ee 100644
	--- a/docs/configuration/search.md
	+++ b/docs/configuration/search.md
	@@ -1,121 +1,121 @@
	# Configuring search

	{! backend/administration/CLI_tasks/general_cli_task_info.include !}

	## Built-in search

	To use built-in search that has no external dependencies, set the search module to `Pleroma.Activity`:

	> config :pleroma, Pleroma.Search, module: Pleroma.Search.DatabaseSearch

	While it has no external dependencies, it has problems with performance and relevancy.

	## Meilisearch

	Note that it's quite a bit more memory hungry than PostgreSQL (around 4-5G for ~1.2 million
	posts while idle and up to 7G while indexing initially). The disk usage for this additional index is also
	around 4 gigabytes. Like [RUM](./cheatsheet.md#rum-indexing-for-full-text-search) indexes, it offers considerably
	higher performance and ordering by timestamp in a reasonable amount of time.
	Additionally, the search results seem to be more accurate.

	Due to high memory usage, it may be best to set it up on a different machine, if running pleroma on a low-resource
	computer, and use private key authentication to secure the remote search instance.

	To use [meilisearch](https://www.meilisearch.com/), set the search module to `Pleroma.Search.Meilisearch`:

	> config :pleroma, Pleroma.Search, module: Pleroma.Search.Meilisearch

	You then need to set the address of the meilisearch instance, and optionally the private key for authentication. You might
	also want to change the `initial_indexing_chunk_size` to be smaller if you're server is not very powerful, but not higher than `100_000`,
	because meilisearch will refuse to process it if it's too big. However, in general you want this to be as big as possible, because meilisearch
	indexes faster when it can process many posts in a single batch.

	> config :pleroma, Pleroma.Search.Meilisearch,
	> url: "http://127.0.0.1:7700/",
	> private_key: "private key",
	> initial_indexing_chunk_size: 100_000

	Information about setting up meilisearch can be found in the
	[official documentation](https://docs.meilisearch.com/learn/getting_started/installation.html).
	You probably want to start it with `MEILI_NO_ANALYTICS=true` and `MEILI_NO_CENTRY=true` environment variables,
	to disable analytics.

	### Private key authentication (optional)

	To set the private key, use the `MEILI_MASTER_KEY` environment variable when starting. After setting the _master key_,
	you have to get the _private key_, which is actually used for authentication.

	=== "OTP"
	```sh
	- ./bin/pleroma_ctl search.meilisearch show-private-key <your master key here>
	+ ./bin/pleroma_ctl search.meilisearch show-keys <your master key here>
	```

	=== "From Source"
	```sh
	- mix pleroma.search.meilisearch show-private-key <your master key here>
	+ mix pleroma.search.meilisearch show-keys <your master key here>
	```

	-This is the key you actually put into your configuration file.
	+You will see a "Default Admin API Key", this is the key you actually put into your configuration file.

	### Initial indexing

	After setting up the configuration, you'll want to index all of your already existsing posts. Only public posts are indexed. You'll only
	have to do it one time, but it might take a while, depending on the amount of posts your instance has seen. This is also a fairly RAM
	consuming process for `meilisearch`, and it will take a lot of RAM when running if you have a lot of posts (seems to be around 5G for ~1.2
	million posts while idle and up to 7G while indexing initially, but your experience may be different).

	The sequence of actions is as follows:

	1. First, change the configuration to use `Pleroma.Search.Meilisearch` as the search backend
	2. Restart your instance, at this point it can be used while the search indexing is running, though search won't return anything
	3. Start the initial indexing process (as described below with `index`),
	and wait until the task says it sent everything from the database to index
	4. Wait until everything is actually indexed (by checking with `stats` as described below),
	at this point you don't have to do anything, just wait a while.

	To start the initial indexing, run the `index` command:

	=== "OTP"
	```sh
	./bin/pleroma_ctl search.meilisearch index
	```

	=== "From Source"
	```sh
	mix pleroma.search.meilisearch index
	```

	This will show you the total amount of posts to index, and then show you the amount of posts indexed currently, until the numbers eventually
	become the same. The posts are indexed in big batches and meilisearch will take some time to actually index them, even after you have
	inserted all the posts into it. Depending on the amount of posts, this may be as long as several hours. To get information about the status
	of indexing and how many posts have actually been indexed, use the `stats` command:

	=== "OTP"
	```sh
	./bin/pleroma_ctl search.meilisearch stats
	```

	=== "From Source"
	```sh
	mix pleroma.search.meilisearch stats
	```

	### Clearing the index

	In case you need to clear the index (for example, to re-index from scratch, if that needs to happen for some reason), you can
	use the `clear` command:

	=== "OTP"
	```sh
	./bin/pleroma_ctl search.meilisearch clear
	```

	=== "From Source"
	```sh
	mix pleroma.search.meilisearch clear
	```

	This will clear all the posts from the search index. Note, that deleted posts are also removed from index by the instance itself, so
	there is no need to actually clear the whole index, unless you want all of it gone. That said, the index does not hold any information
	that cannot be re-created from the database, it should also generally be a lot smaller than the size of your database. Still, the size
	depends on the amount of text in posts.
	diff --git a/lib/mix/tasks/pleroma/search/meilisearch.ex b/lib/mix/tasks/pleroma/search/meilisearch.ex
	index 5098668ad..db56876fa 100644
	--- a/lib/mix/tasks/pleroma/search/meilisearch.ex
	+++ b/lib/mix/tasks/pleroma/search/meilisearch.ex
	@@ -1,142 +1,144 @@
	# Pleroma: A lightweight social networking server
	# Copyright © 2017-2021 Pleroma Authors <https://pleroma.social/>
	# SPDX-License-Identifier: AGPL-3.0-only

	defmodule Mix.Tasks.Pleroma.Search.Meilisearch do
	require Pleroma.Constants

	import Mix.Pleroma
	import Ecto.Query

	import Pleroma.Search.Meilisearch,
	only: [meili_post: 2, meili_put: 2, meili_get: 1, meili_delete!: 1]

	def run(["index"]) do
	start_pleroma()

	meili_version =
	(
	{:ok, result} = meili_get("/version")

	result["pkgVersion"]
	)

	# The ranking rule syntax was changed but nothing about that is mentioned in the changelog
	- if not Version.match?(meili_version, ">= 0.24.0") do
	+ if not Version.match?(meili_version, ">= 0.25.0") do
	raise "Meilisearch <0.24.0 not supported"
	end

	{:ok, _} =
	meili_post(
	"/indexes/objects/settings/ranking-rules",
	[
	"published:desc",
	"words",
	"exactness",
	"proximity",
	"typo",
	"attribute",
	"sort"
	]
	)

	{:ok, _} =
	meili_post(
	"/indexes/objects/settings/searchable-attributes",
	[
	"content"
	]
	)

	IO.puts("Created indices. Starting to insert posts.")

	chunk_size = Pleroma.Config.get([Pleroma.Search.Meilisearch, :initial_indexing_chunk_size])

	Pleroma.Repo.transaction(
	fn ->
	query =
	from(Pleroma.Object,
	# Only index public and unlisted posts which are notes and have some text
	where:
	fragment("data->>'type' = 'Note'") and
	(fragment("data->'to' \\? ?", ^Pleroma.Constants.as_public()) or
	fragment("data->'cc' \\? ?", ^Pleroma.Constants.as_public())),
	order_by: [desc: fragment("data->'published'")]
	)

	count = query \|> Pleroma.Repo.aggregate(:count, :data)
	IO.puts("Entries to index: #{count}")

	Pleroma.Repo.stream(
	query,
	timeout: :infinity
	)
	\|> Stream.map(&Pleroma.Search.Meilisearch.object_to_search_data/1)
	\|> Stream.filter(fn o -> not is_nil(o) end)
	\|> Stream.chunk_every(chunk_size)
	\|> Stream.transform(0, fn objects, acc ->
	new_acc = acc + Enum.count(objects)

	# Reset to the beginning of the line and rewrite it
	IO.write("\r")
	IO.write("Indexed #{new_acc} entries")

	{[objects], new_acc}
	end)
	\|> Stream.each(fn objects ->
	result =
	meili_put(
	"/indexes/objects/documents",
	objects
	)

	with {:ok, res} <- result do
	if not Map.has_key?(res, "updateId") do
	IO.puts("\nFailed to index: #{inspect(result)}")
	end
	else
	e -> IO.puts("\nFailed to index due to network error: #{inspect(e)}")
	end
	end)
	\|> Stream.run()
	end,
	timeout: :infinity
	)

	IO.write("\n")
	end

	def run(["clear"]) do
	start_pleroma()

	meili_delete!("/indexes/objects/documents")
	end

	- def run(["show-private-key", master_key]) do
	+ def run(["show-keys", master_key]) do
	start_pleroma()

	endpoint = Pleroma.Config.get([Pleroma.Search.Meilisearch, :url])

	{:ok, result} =
	Pleroma.HTTP.get(
	Path.join(endpoint, "/keys"),
	- [{"X-Meili-API-Key", master_key}]
	+ [{"Authorization", "Bearer #{master_key}"}]
	)

	decoded = Jason.decode!(result.body)

	- if decoded["private"] do
	- IO.puts(decoded["private"])
	+ if decoded["results"] do
	+ Enum.each(decoded["results"], fn %{"description" => desc, "key" => key} ->
	+ IO.puts("#{desc}: #{key}")
	+ end)
	else
	- IO.puts("Error fetching the key, check the master key is correct: #{inspect(decoded)}")
	+ IO.puts("Error fetching the keys, check the master key is correct: #{inspect(decoded)}")
	end
	end

	def run(["stats"]) do
	start_pleroma()

	{:ok, result} = meili_get("/indexes/objects/stats")
	IO.puts("Number of entries: #{result["numberOfDocuments"]}")
	IO.puts("Indexing? #{result["isIndexing"]}")
	end
	end
	diff --git a/lib/pleroma/search/meilisearch.ex b/lib/pleroma/search/meilisearch.ex
	index 33bbf8392..0f9182ffc 100644
	--- a/lib/pleroma/search/meilisearch.ex
	+++ b/lib/pleroma/search/meilisearch.ex
	@@ -1,169 +1,169 @@
	defmodule Pleroma.Search.Meilisearch do
	require Logger
	require Pleroma.Constants

	alias Pleroma.Activity

	import Pleroma.Search.DatabaseSearch
	import Ecto.Query

	@behaviour Pleroma.Search.SearchBackend

	defp meili_headers do
	private_key = Pleroma.Config.get([Pleroma.Search.Meilisearch, :private_key])

	[{"Content-Type", "application/json"}] ++
	- if is_nil(private_key), do: [], else: [{"X-Meili-API-Key", private_key}]
	+ if is_nil(private_key), do: [], else: [{"Authorization", "Bearer #{private_key}"}]
	end

	def meili_get(path) do
	endpoint = Pleroma.Config.get([Pleroma.Search.Meilisearch, :url])

	result =
	Pleroma.HTTP.get(
	Path.join(endpoint, path),
	meili_headers()
	)

	with {:ok, res} <- result do
	{:ok, Jason.decode!(res.body)}
	end
	end

	def meili_post(path, params) do
	endpoint = Pleroma.Config.get([Pleroma.Search.Meilisearch, :url])

	result =
	Pleroma.HTTP.post(
	Path.join(endpoint, path),
	Jason.encode!(params),
	meili_headers()
	)

	with {:ok, res} <- result do
	{:ok, Jason.decode!(res.body)}
	end
	end

	def meili_put(path, params) do
	endpoint = Pleroma.Config.get([Pleroma.Search.Meilisearch, :url])

	result =
	Pleroma.HTTP.request(
	:put,
	Path.join(endpoint, path),
	Jason.encode!(params),
	meili_headers(),
	[]
	)

	with {:ok, res} <- result do
	{:ok, Jason.decode!(res.body)}
	end
	end

	def meili_delete!(path) do
	endpoint = Pleroma.Config.get([Pleroma.Search.Meilisearch, :url])

	{:ok, _} =
	Pleroma.HTTP.request(
	:delete,
	Path.join(endpoint, path),
	"",
	meili_headers(),
	[]
	)
	end

	def search(user, query, options \\ []) do
	limit = Enum.min([Keyword.get(options, :limit), 40])
	offset = Keyword.get(options, :offset, 0)
	author = Keyword.get(options, :author)

	res =
	meili_post(
	"/indexes/objects/search",
	%{q: query, offset: offset, limit: limit}
	)

	with {:ok, result} <- res do
	hits = result["hits"] \|> Enum.map(& &1["ap"])

	try do
	hits
	\|> Activity.create_by_object_ap_id()
	\|> Activity.with_preloaded_object()
	\|> Activity.with_preloaded_object()
	\|> Activity.restrict_deactivated_users()
	\|> maybe_restrict_local(user)
	\|> maybe_restrict_author(author)
	\|> maybe_restrict_blocked(user)
	\|> maybe_fetch(user, query)
	\|> order_by([object: obj], desc: obj.data["published"])
	\|> Pleroma.Repo.all()
	rescue
	_ -> maybe_fetch([], user, query)
	end
	end
	end

	def object_to_search_data(object) do
	# Only index public or unlisted Notes
	if not is_nil(object) and object.data["type"] == "Note" and
	not is_nil(object.data["content"]) and
	(Pleroma.Constants.as_public() in object.data["to"] or
	Pleroma.Constants.as_public() in object.data["cc"]) and
	String.length(object.data["content"]) > 1 do
	data = object.data

	content_str =
	case data["content"] do
	[nil \| rest] -> to_string(rest)
	str -> str
	end

	content =
	with {:ok, scrubbed} <- FastSanitize.strip_tags(content_str),
	trimmed <- String.trim(scrubbed) do
	trimmed
	end

	if String.length(content) > 1 do
	{:ok, published, _} = DateTime.from_iso8601(data["published"])

	%{
	id: object.id,
	content: content,
	ap: data["id"],
	published: published \|> DateTime.to_unix()
	}
	end
	end
	end

	@impl true
	def add_to_index(activity) do
	maybe_search_data = object_to_search_data(activity.object)

	if activity.data["type"] == "Create" and maybe_search_data do
	result =
	meili_put(
	"/indexes/objects/documents",
	[maybe_search_data]
	)

	with {:ok, res} <- result,
	true <- Map.has_key?(res, "updateId") do
	# Do nothing
	else
	_ ->
	Logger.error("Failed to add activity #{activity.id} to index: #{inspect(result)}")
	end
	end
	end

	@impl true
	def remove_from_index(object) do
	meili_delete!("/indexes/objects/documents/#{object.id}")
	end
	end

File Metadata

Mime Type: text/x-diff
Expires: Sat, Aug 8, 4:49 PM (1 d, 22 h)
Storage Engine: blob
Storage Format: Raw Data
Storage Handle: 1723723
Default Alt Text: (14 KB)

No OneTemporaryActions

View Options

File Metadata

Event Timeline

No OneTemporary
Actions