Page MenuHomePhorge
Files F114223

No OneTemporary
Actions

Size
10 KB
Referenced Files
None
Subscribers
None

View Options

diff --git a/README.md b/README.md
index 7eced3d..5e94a60 100644
--- a/README.md
+++ b/README.md
@@ -1,192 +1,194 @@
# Majic
**Majic** provides a robust integration of [libmagic](http://man7.org/linux/man-pages/man3/libmagic.3.html) for Elixir.
With this library, you can start an one-off process to run a single check, or run the process as a daemon if you expect to run
many checks.
It is a friendly fork of [gen_magic](https://github.com/evadne/gen_magic) featuring a (arguably) more robust C-code
using erl_interface, built in pooling, unified/clean API, and an optional Plug.
This package is regulary tested on multiple platforms (Debian, macOS, Fedora, Alpine, FreeBSD) to ensure it'll work fine
in any environment.
+Majic depends on [`file`](https://github.com/file/file) mirror repository to provide an up-to-date magic database. The database is compiled when Majic is compiled.
+
## Installation
The package can be installed by adding `majic` to your list of dependencies in `mix.exs`:
```elixir
def deps do
[
{:majic, "~> 1.0"}
]
end
```
You must also have [libmagic](http://man7.org/linux/man-pages/man3/libmagic.3.html) installed locally with headers, alongside common compilation tools (i.e. build-essential). These can be acquired by apt-get, yum, brew, etc.
Compilation of the underlying C program is automatic and handled by [elixir_make](https://github.com/elixir-lang/elixir_make).
## Usage
Depending on the use case, you may utilise a single (one-off) Majic process without reusing it as a daemon, or utilise a connection pool (such as Poolboy) in your application to run multiple persistent Majic processes.
To use Majic directly, you can use `Majic.Once.perform/1`:
```elixir
iex(1)> Majic.perform(".", once: true)
{:ok,
%Majic.Result{
content: "directory",
encoding: "binary",
mime_type: "inode/directory"
}}
```
To use the Majic server as a daemon, you can start it first, keep a reference, then feed messages to it as you require:
```elixir
{:ok, pid} = Majic.Server.start_link([])
{:ok, result} = Majic.perform(path, server: pid)
```
See `Majic.Server.start_link/1` and `t:Majic.Server.option/0` for more information on startup parameters.
See `Majic.Result` for details on the result provided.
## Configuration
When using `Majic.Server.start_link/1` to start a persistent server, or `Majic.Helpers.perform_once/2` to run an ad-hoc request, you can override specific options to suit your use case.
| Name | Default | Description |
| - | - | - |
| `:startup_timeout` | 1000 | Number of milliseconds to wait for client startup |
| `:process_timeout` | 30000 | Number of milliseconds to process each request |
| `:recycle_threshold` | 10 | Number of cycles before the C process is replaced |
| `:database_patterns` | `[:default]` | Databases to load |
See `t:Majic.Server.option/0` for details.
__Note__ `:recycle_thresold` is only useful if you are using a libmagic `<5.29`, where it was susceptible to memleaks
([details](https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=840754)]). In future versions of `majic` this option could
be ignored.
### Reloading / Altering databases
If you want `majic` to reload its database(s), run `Majic.Server.reload(ref)`.
If you want to add or remove databases to a running server, you would have to run `Majic.Server.reload(ref, databases)`
where databases being the same argument as `database_patterns` on start. `Majic` does not support adding/removing
databases at runtime without a port reload.
### Use Cases
#### Ad-Hoc Requests
For ad-hoc requests, you can use the helper method `Majic.Once.perform_once/2`:
```elixir
iex(1)> Majic.perform(Path.join(File.cwd!(), "Makefile"), once: true)
{:ok,
%Majic.Result{
content: "makefile script, ASCII text",
encoding: "us-ascii",
mime_type: "text/x-makefile"
}}
```
#### Supervised Requests
The Server should be run under a supervisor which provides resiliency.
Here we run it under a supervisor in an application:
```elixir
children =
[
# ...
{Majic.Server, [name: YourApp.Majic]}
]
opts = [strategy: :one_for_one, name: YourApp.Supervisor]
Supervisor.start_link(children, opts)
```
Now we can ask it to inspect a file:
```elixir
iex(2)> Majic.perform(Path.expand("~/.bash_history"), server: YourApp.Majic)
{:ok, %Majic.Result{mime_type: "text/plain", encoding: "us-ascii", content: "ASCII text"}}
```
Note that in this case we have opted to use a named process.
#### Pool
For concurrency *and* resiliency, you may start the `Majic.Pool`. By default, it will start a `Majic.Server`
worker per online scheduler:
You can add a pool in your application supervisor by adding it as a child:
```elixir
children =
[
# ...
{Majic.Pool, [name: YourApp.MajicPool, pool_size: 2]}
]
opts = [strategy: :one_for_one, name: YourApp.Supervisor]
Supervisor.start_link(children, opts)
```
And then you can use it with `Majic.perform/2` with `pool: YourApp.MajicPool` option:
```elixir
iex(1)> Majic.perform(Path.expand("~/.bash_history"), pool: YourApp.MajicPool)
{:ok, %Majic.Result{mime_type: "text/plain", encoding: "us-ascii", content: "ASCII text"}}
```
#### Fixing extensions
You may also want to fix the user-provided filename according to its detected MIME type. To do this, you can use `Majic.Extension.fix/3`:
```elixir
iex(1)> {:ok, result} = Majic.perform("cat.jpeg", once: true)
{:ok, %Majic.Result{mime_type: "image/webp", ...}}
iex(1)> Majic.Extension.fix("cat.jpeg", result)
"cat.webp"
```
#### Use with Plug.Upload
If you use Plug or Phoenix, you may want to automatically verify the content type of every `Plug.Upload`. The
`Majic.Plug` is there for this.
Enable it by using `plug Majic.Plug, pool: YourApp.MajicPool` in your pipeline or controller. Then, every `Plug.Upload`
in `conn.params` and `conn.body_params` is now verified. The filename is also altered with an extension matching its
content-type, using `Majic.Extension`.
## Notes
### Soak Test
Run an endless cycle to prove that the program is resilient:
```bash
find /usr/share/ -name *png | xargs mix run test/soak.exs
find . -name *ex | xargs mix run test/soak.exs
```
## Acknowledgements
During design and prototype development of this library, the Author has drawn inspiration from the following individuals, and therefore
thanks all contributors for their generosity:
- [Evadne Wu](https://github.com/evadne)
- Original [gen_magic](https://github.com/evadne/gen_magic) author.
- [James Every](https://github.com/devstopfix)
- Enhanced Elixir Wrapper (based on GenServer)
- Initial Hex packaging (v.0.22)
- Soak Testing
- Matthias and Ced for helping the author with C oddities
- [Hecate](https://github.com/Kleidukos) for laughing at aforementionned oddities
- majic for giving inspiration for the lib name (magic, majic, get it? hahaha..)
diff --git a/lib/mix/tasks/compile/majic.ex b/lib/mix/tasks/compile/majic.ex
index b44f3bf..75a930d 100644
--- a/lib/mix/tasks/compile/majic.ex
+++ b/lib/mix/tasks/compile/majic.ex
@@ -1,108 +1,108 @@
defmodule Mix.Tasks.Compile.Majic do
use Mix.Task.Compiler
- @repo_path "deps/libfile"
+ @repo_path Path.join(Mix.Project.deps_path, "libfile")
@magic_path Path.join(@repo_path, "/magic")
@magdir Path.join(@magic_path, "/Magdir")
- @build_path "_build/file"
+ @build_path Path.join(Mix.Project.build_path, "/majic")
@build_magdir Path.join(@build_path, "/magic")
@manifest Path.join(@build_path, "/majic.manifest")
@patch_path "src/magic_patches"
- @built_path "priv/magic.mgc"
+ @built_path Path.join(to_string(:code.priv_dir(:majic)), "/magic.mgc")
@shortdoc "Updates and compiles majic's embedded magic database."
@doc """
Uses `libfile` dependency Magdir, applies patches from `#{@patch_path}`, and builds to `#{@built_path}`.
"""
defmodule Manifest do
defstruct [:hash, {:patches, []}]
end
@impl Mix.Task.Compiler
def clean() do
File.rm!(@built_path)
File.rm_rf!(@build_path)
end
@impl Mix.Task.Compiler
def manifests() do
[@manifest]
end
@impl Mix.Task.Compiler
def run(_) do
{:ok, manifest} = read_manifest()
{:ok, sha} = get_dep_revision()
patches = list_patches()
File.mkdir_p!(Path.join(@build_path, "/magic"))
if sha == nil || sha != manifest.hash || patches != manifest.patches do
:ok = assemble_magdir()
{:ok, patches, _err} = apply_patches()
{:ok, _} = Majic.compile(@build_magdir)
File.rm_rf!(@build_magdir)
File.cp!("magic.mgc", @built_path)
File.rm!("magic.mgc")
manifest = %Manifest{hash: sha, patches: Enum.sort(patches)}
File.write!(@manifest, :erlang.term_to_binary(manifest))
Mix.shell().info("Generated magic database")
else
Mix.shell().info("Magic database up-to-date")
end
:ok
end
defp read_manifest() do
with {:ok, binary} <- File.read(@manifest),
{:term, %Manifest{} = manifest} <- {:term, :erlang.binary_to_term(binary)}
do
{:ok, manifest}
else
{:term, _} -> {:ok, %Manifest{}}
{:error, :enoent} -> {:ok, %Manifest{}}
error -> error
end
end
defp get_dep_revision() do
git_dir = Path.join(@repo_path, "/.git")
case System.cmd("git", ["--git-dir=#{git_dir}", "rev-parse", "HEAD"]) do
{rev, 0} -> {:ok, String.trim(rev)}
_ -> {:ok, nil}
end
end
defp list_patches() do
@patch_path
|> File.ls!()
|> Enum.sort()
end
defp apply_patches() do
patched = list_patches()
|> Enum.map(fn(patch) ->
path = Path.expand(Path.join(@patch_path, patch))
case System.cmd("git", ["apply", path], cd: @build_path) do
{_, 0} ->
Mix.shell().info("Patched magic database: #{patch}")
{patch, :ok}
{error, code} ->
Mix.shell().error("Failed to apply patch #{patch} (#{code})")
Mix.shell().error(error)
{patch, {:error, error}}
end
end)
ok = Enum.filter(patched, fn({_, result}) -> result == :ok end) |> Enum.map(fn({name, _}) -> name end)
err = Enum.filter(patched, fn({_, result}) -> result != :ok end) |> Enum.map(fn({name, {:error, error}}) -> {name, error} end)
{:ok, ok, err}
end
defp assemble_magdir() do
File.rm_rf!(@build_magdir)
File.mkdir!(@build_magdir)
File.ls!(@magdir)
|> Enum.each(fn(file) ->
File.cp!(Path.join(@magdir, file), Path.join(@build_magdir, file))
end)
:ok
end
end

File Metadata

Mime Type
text/x-diff
Expires
Mon, Nov 25, 11:42 PM (1 d, 6 h)
Storage Engine
blob
Storage Format
Raw Data
Storage Handle
40117
Default Alt Text
(10 KB)

Event Timeline