Page MenuHomePhorge
Files F85627969

No OneTemporary
Actions

Size
15 KB
Referenced Files
None
Subscribers
None

View Options

diff --git a/docs/administration/CLI_tasks/database.md b/docs/administration/CLI_tasks/database.md
index 647f6f274..64dd66c0c 100644
--- a/docs/administration/CLI_tasks/database.md
+++ b/docs/administration/CLI_tasks/database.md
@@ -1,100 +1,110 @@
# Database maintenance tasks
{! backend/administration/CLI_tasks/general_cli_task_info.include !}
!!! danger
These mix tasks can take a long time to complete. Many of them were written to address specific database issues that happened because of bugs in migrations or other specific scenarios. Do not run these tasks "just in case" if everything is fine your instance.
## Replace embedded objects with their references
Replaces embedded objects with references to them in the `objects` table. Only needs to be ran once if the instance was created before Pleroma 1.0.5. The reason why this is not a migration is because it could significantly increase the database size after being ran, however after this `VACUUM FULL` will be able to reclaim about 20% (really depends on what is in the database, your mileage may vary) of the db size before the migration.
```sh tab="OTP"
./bin/pleroma_ctl database remove_embedded_objects [option ...]
```
```sh tab="From Source"
mix pleroma.database remove_embedded_objects [option ...]
```
### Options
- `--vacuum` - run `VACUUM FULL` after the embedded objects are replaced with their references
## Prune old remote posts from the database
This will prune remote posts older than 90 days (configurable with [`config :pleroma, :instance, remote_post_retention_days`](../../configuration/cheatsheet.md#instance)) from the database, they will be refetched from source when accessed.
!!! danger
The disk space will only be reclaimed after `VACUUM FULL`. You may run out of disk space during the execution of the task or vacuuming if you don't have about 1/3rds of the database size free.
```sh tab="OTP"
./bin/pleroma_ctl database prune_objects [option ...]
```
```sh tab="From Source"
mix pleroma.database prune_objects [option ...]
```
### Options
- `--vacuum` - run `VACUUM FULL` after the objects are pruned
## Create a conversation for all existing DMs
Can be safely re-run
```sh tab="OTP"
./bin/pleroma_ctl database bump_all_conversations
```
```sh tab="From Source"
mix pleroma.database bump_all_conversations
```
## Remove duplicated items from following and update followers count for all users
```sh tab="OTP"
./bin/pleroma_ctl database update_users_following_followers_counts
```
```sh tab="From Source"
mix pleroma.database update_users_following_followers_counts
```
## Fix the pre-existing "likes" collections for all objects
```sh tab="OTP"
./bin/pleroma_ctl database fix_likes_collections
```
```sh tab="From Source"
mix pleroma.database fix_likes_collections
```
## Vacuum the database
### Analyze
Running an `analyze` vacuum job can improve performance by updating statistics used by the query planner. **It is safe to cancel this.**
```sh tab="OTP"
./bin/pleroma_ctl database vacuum analyze
```
```sh tab="From Source"
mix pleroma.database vacuum analyze
```
### Full
Running a `full` vacuum job rebuilds your entire database by reading all of the data and rewriting it into smaller
and more compact files with an optimized layout. This process will take a long time and use additional disk space as
it builds the files side-by-side the existing database files. It can make your database faster and use less disk space,
but should only be run if necessary. **It is safe to cancel this.**
```sh tab="OTP"
./bin/pleroma_ctl database vacuum full
```
```sh tab="From Source"
mix pleroma.database vacuum full
-```
\ No newline at end of file
+```
+
+## Add expiration to all local statuses
+
+```sh tab="OTP"
+./bin/pleroma_ctl database ensure_expiration
+```
+
+```sh tab="From Source"
+mix pleroma.database ensure_expiration
+```
diff --git a/lib/mix/tasks/pleroma/database.ex b/lib/mix/tasks/pleroma/database.ex
index 82e2abdcb..d57e59b11 100644
--- a/lib/mix/tasks/pleroma/database.ex
+++ b/lib/mix/tasks/pleroma/database.ex
@@ -1,133 +1,149 @@
# Pleroma: A lightweight social networking server
# Copyright © 2017-2020 Pleroma Authors <https://pleroma.social/>
# SPDX-License-Identifier: AGPL-3.0-only
defmodule Mix.Tasks.Pleroma.Database do
alias Pleroma.Conversation
alias Pleroma.Maintenance
alias Pleroma.Object
alias Pleroma.Repo
alias Pleroma.User
require Logger
require Pleroma.Constants
+ import Ecto.Query
import Mix.Pleroma
use Mix.Task
@shortdoc "A collection of database related tasks"
@moduledoc File.read!("docs/administration/CLI_tasks/database.md")
def run(["remove_embedded_objects" | args]) do
{options, [], []} =
OptionParser.parse(
args,
strict: [
vacuum: :boolean
]
)
start_pleroma()
Logger.info("Removing embedded objects")
Repo.query!(
"update activities set data = safe_jsonb_set(data, '{object}'::text[], data->'object'->'id') where data->'object'->>'id' is not null;",
[],
timeout: :infinity
)
if Keyword.get(options, :vacuum) do
Maintenance.vacuum("full")
end
end
def run(["bump_all_conversations"]) do
start_pleroma()
Conversation.bump_for_all_activities()
end
def run(["update_users_following_followers_counts"]) do
start_pleroma()
User
|> Repo.all()
|> Enum.each(&User.update_follower_count/1)
end
def run(["prune_objects" | args]) do
- import Ecto.Query
-
{options, [], []} =
OptionParser.parse(
args,
strict: [
vacuum: :boolean
]
)
start_pleroma()
deadline = Pleroma.Config.get([:instance, :remote_post_retention_days])
Logger.info("Pruning objects older than #{deadline} days")
time_deadline =
NaiveDateTime.utc_now()
|> NaiveDateTime.add(-(deadline * 86_400))
from(o in Object,
where:
fragment(
"?->'to' \\? ? OR ?->'cc' \\? ?",
o.data,
^Pleroma.Constants.as_public(),
o.data,
^Pleroma.Constants.as_public()
),
where: o.inserted_at < ^time_deadline,
where:
fragment("split_part(?->>'actor', '/', 3) != ?", o.data, ^Pleroma.Web.Endpoint.host())
)
|> Repo.delete_all(timeout: :infinity)
if Keyword.get(options, :vacuum) do
Maintenance.vacuum("full")
end
end
def run(["fix_likes_collections"]) do
- import Ecto.Query
-
start_pleroma()
from(object in Object,
where: fragment("(?)->>'likes' is not null", object.data),
select: %{id: object.id, likes: fragment("(?)->>'likes'", object.data)}
)
|> Pleroma.RepoStreamer.chunk_stream(100)
|> Stream.each(fn objects ->
ids =
objects
|> Enum.filter(fn object -> object.likes |> Jason.decode!() |> is_map() end)
|> Enum.map(& &1.id)
Object
|> where([object], object.id in ^ids)
|> update([object],
set: [
data:
fragment(
"safe_jsonb_set(?, '{likes}', '[]'::jsonb, true)",
object.data
)
]
)
|> Repo.update_all([], timeout: :infinity)
end)
|> Stream.run()
end
def run(["vacuum", args]) do
start_pleroma()
Maintenance.vacuum(args)
end
+
+ def run(["ensure_expiration"]) do
+ start_pleroma()
+ days = Pleroma.Config.get([:mrf_activity_expiration, :days], 365)
+
+ Pleroma.Activity
+ |> join(:left, [a], u in assoc(a, :expiration))
+ |> where(local: true)
+ |> where([a, u], is_nil(u))
+ |> Pleroma.RepoStreamer.chunk_stream(100)
+ |> Stream.each(fn activities ->
+ Enum.each(activities, fn activity ->
+ expires_at = Timex.shift(activity.inserted_at, days: days)
+
+ Pleroma.ActivityExpiration.create(activity, expires_at, false)
+ end)
+ end)
+ |> Stream.run()
+ end
end
diff --git a/lib/pleroma/activity_expiration.ex b/lib/pleroma/activity_expiration.ex
index db9c88d84..7cc9668b3 100644
--- a/lib/pleroma/activity_expiration.ex
+++ b/lib/pleroma/activity_expiration.ex
@@ -1,67 +1,69 @@
# Pleroma: A lightweight social networking server
# Copyright © 2017-2020 Pleroma Authors <https://pleroma.social/>
# SPDX-License-Identifier: AGPL-3.0-only
defmodule Pleroma.ActivityExpiration do
use Ecto.Schema
alias Pleroma.Activity
alias Pleroma.ActivityExpiration
alias Pleroma.Repo
import Ecto.Changeset
import Ecto.Query
@type t :: %__MODULE__{}
@min_activity_lifetime :timer.hours(1)
schema "activity_expirations" do
belongs_to(:activity, Activity, type: FlakeId.Ecto.CompatType)
field(:scheduled_at, :naive_datetime)
end
- def changeset(%ActivityExpiration{} = expiration, attrs) do
+ def changeset(%ActivityExpiration{} = expiration, attrs, validate_scheduled_at) do
expiration
|> cast(attrs, [:scheduled_at])
|> validate_required([:scheduled_at])
- |> validate_scheduled_at()
+ |> validate_scheduled_at(validate_scheduled_at)
end
def get_by_activity_id(activity_id) do
ActivityExpiration
|> where([exp], exp.activity_id == ^activity_id)
|> Repo.one()
end
- def create(%Activity{} = activity, scheduled_at) do
+ def create(%Activity{} = activity, scheduled_at, validate_scheduled_at \\ true) do
%ActivityExpiration{activity_id: activity.id}
- |> changeset(%{scheduled_at: scheduled_at})
+ |> changeset(%{scheduled_at: scheduled_at}, validate_scheduled_at)
|> Repo.insert()
end
def due_expirations(offset \\ 0) do
naive_datetime =
NaiveDateTime.utc_now()
|> NaiveDateTime.add(offset, :millisecond)
ActivityExpiration
|> where([exp], exp.scheduled_at < ^naive_datetime)
|> Repo.all()
end
- def validate_scheduled_at(changeset) do
+ def validate_scheduled_at(changeset, false), do: changeset
+
+ def validate_scheduled_at(changeset, true) do
validate_change(changeset, :scheduled_at, fn _, scheduled_at ->
if not expires_late_enough?(scheduled_at) do
[scheduled_at: "an ephemeral activity must live for at least one hour"]
else
[]
end
end)
end
def expires_late_enough?(scheduled_at) do
now = NaiveDateTime.utc_now()
diff = NaiveDateTime.diff(scheduled_at, now, :millisecond)
diff > @min_activity_lifetime
end
end
diff --git a/test/tasks/database_test.exs b/test/tasks/database_test.exs
index 883828d77..3a28aa133 100644
--- a/test/tasks/database_test.exs
+++ b/test/tasks/database_test.exs
@@ -1,130 +1,169 @@
# Pleroma: A lightweight social networking server
# Copyright © 2017-2020 Pleroma Authors <https://pleroma.social/>
# SPDX-License-Identifier: AGPL-3.0-only
defmodule Mix.Tasks.Pleroma.DatabaseTest do
alias Pleroma.Activity
alias Pleroma.Object
alias Pleroma.Repo
alias Pleroma.User
alias Pleroma.Web.CommonAPI
use Pleroma.DataCase
import Pleroma.Factory
setup_all do
Mix.shell(Mix.Shell.Process)
on_exit(fn ->
Mix.shell(Mix.Shell.IO)
end)
:ok
end
describe "running remove_embedded_objects" do
test "it replaces objects with references" do
user = insert(:user)
{:ok, activity} = CommonAPI.post(user, %{status: "test"})
new_data = Map.put(activity.data, "object", activity.object.data)
{:ok, activity} =
activity
|> Activity.change(%{data: new_data})
|> Repo.update()
assert is_map(activity.data["object"])
Mix.Tasks.Pleroma.Database.run(["remove_embedded_objects"])
activity = Activity.get_by_id_with_object(activity.id)
assert is_binary(activity.data["object"])
end
end
describe "prune_objects" do
test "it prunes old objects from the database" do
insert(:note)
deadline = Pleroma.Config.get([:instance, :remote_post_retention_days]) + 1
date =
Timex.now()
|> Timex.shift(days: -deadline)
|> Timex.to_naive_datetime()
|> NaiveDateTime.truncate(:second)
%{id: id} =
:note
|> insert()
|> Ecto.Changeset.change(%{inserted_at: date})
|> Repo.update!()
assert length(Repo.all(Object)) == 2
Mix.Tasks.Pleroma.Database.run(["prune_objects"])
assert length(Repo.all(Object)) == 1
refute Object.get_by_id(id)
end
end
describe "running update_users_following_followers_counts" do
test "following and followers count are updated" do
[user, user2] = insert_pair(:user)
{:ok, %User{} = user} = User.follow(user, user2)
following = User.following(user)
assert length(following) == 2
assert user.follower_count == 0
{:ok, user} =
user
|> Ecto.Changeset.change(%{follower_count: 3})
|> Repo.update()
assert user.follower_count == 3
assert :ok == Mix.Tasks.Pleroma.Database.run(["update_users_following_followers_counts"])
user = User.get_by_id(user.id)
assert length(User.following(user)) == 2
assert user.follower_count == 0
end
end
describe "running fix_likes_collections" do
test "it turns OrderedCollection likes into empty arrays" do
[user, user2] = insert_pair(:user)
{:ok, %{id: id, object: object}} = CommonAPI.post(user, %{status: "test"})
{:ok, %{object: object2}} = CommonAPI.post(user, %{status: "test test"})
CommonAPI.favorite(user2, id)
likes = %{
"first" =>
"http://mastodon.example.org/objects/dbdbc507-52c8-490d-9b7c-1e1d52e5c132/likes?page=1",
"id" => "http://mastodon.example.org/objects/dbdbc507-52c8-490d-9b7c-1e1d52e5c132/likes",
"totalItems" => 3,
"type" => "OrderedCollection"
}
new_data = Map.put(object2.data, "likes", likes)
object2
|> Ecto.Changeset.change(%{data: new_data})
|> Repo.update()
assert length(Object.get_by_id(object.id).data["likes"]) == 1
assert is_map(Object.get_by_id(object2.id).data["likes"])
assert :ok == Mix.Tasks.Pleroma.Database.run(["fix_likes_collections"])
assert length(Object.get_by_id(object.id).data["likes"]) == 1
assert Enum.empty?(Object.get_by_id(object2.id).data["likes"])
end
end
+
+ describe "ensure_expiration" do
+ test "it adds to expiration old statuses" do
+ %{id: activity_id1} = insert(:note_activity)
+
+ %{id: activity_id2} =
+ insert(:note_activity, %{inserted_at: NaiveDateTime.from_iso8601!("2015-01-23 23:50:07")})
+
+ %{id: activity_id3} = activity3 = insert(:note_activity)
+
+ expires_at =
+ NaiveDateTime.utc_now()
+ |> NaiveDateTime.add(60 * 61, :second)
+ |> NaiveDateTime.truncate(:second)
+
+ Pleroma.ActivityExpiration.create(activity3, expires_at)
+
+ Mix.Tasks.Pleroma.Database.run(["ensure_expiration"])
+
+ expirations =
+ Pleroma.ActivityExpiration
+ |> order_by(:activity_id)
+ |> Repo.all()
+
+ assert [
+ %Pleroma.ActivityExpiration{
+ activity_id: ^activity_id1
+ },
+ %Pleroma.ActivityExpiration{
+ activity_id: ^activity_id2,
+ scheduled_at: ~N[2016-01-23 23:50:07]
+ },
+ %Pleroma.ActivityExpiration{
+ activity_id: ^activity_id3,
+ scheduled_at: ^expires_at
+ }
+ ] = expirations
+ end
+ end
end

File Metadata

Mime Type
text/x-diff
Expires
Sat, Aug 8, 7:58 AM (16 h, 58 m)
Storage Engine
blob
Storage Format
Raw Data
Storage Handle
1722990
Default Alt Text
(15 KB)

Event Timeline