Best practice for potentially heavy operations

Justoverclock

I’m curious to understand what is the best way to perform potentially heavy operations on a model.

For example, if I want to update a table for each User should I loop for each user in the model and execute my logic. But this in a large forum can lead to memory leak.

So in Flarum how I can do this in the best way?

For example, something like:

  $users = User::all();

        foreach ($users as $user) {
            $ex = new ExampleModel();
            $ex->id = $user->id;
            $ex->value = $something;
           
            $ex->save();
        }

clarkwinkelmann

Justoverclock use the Laravel query builder each() function, or chunk() (chunk is used internally for each)

User::each(function ($user) {
    echo "{$user->username}\n";
});

It can be put at the end of any query:

// Build query, but do not call get()
$query = User::where('is_email_verified', true);

// If you need to track progress, you can query how many results there will be in advance
$total = $query->count();

$query->each(function ($user) {
    echo "{$user->username}\n";
});

By default I believe it will retrieve 100 results from the database, then loop through each of them and call the callback, then repeat with the next chunk. The second parameter can be used to customize how many models are retrieved per SQL request ("chunk").

Also consider putting such code in a console command or a queable job, so that it can be run without any memory or time limit.

Justoverclock

clarkwinkelmann oh so i assume that is correct to do

User::query()->chunk(100, function ($users) use ($ex) {
            foreach ($users as $user) {
                $example= new Example();
                // rest of the code
            }
        });

if this is correct, then ok! thx Clark

clarkwinkelmann

Justoverclock yes, that's functionally equivalent to each(). I prefer to use each() as it makes the code easier to read.

chunk can be useful if you want to run some special eager loading code or perform some client-side filtering or re-mapping. For regular relationships, a load call on the main query will be repeated automatically on every chunk with each() as well.

luceos

Each actually runs chunk in the background. So using each in this case where you want to loop over every record is the best way to go. Understand that there's only one other better solution, which is the cursor method.

Some additional reading material:
https://arter.dev/laravel-chunk-vs-cursor-methods-compared
https://janostlund.com/2021-12-26/eloquent-cursor-vs-chunk

Justoverclock

luceos how can be better that chunk?

actually says:

cursor() 75 MB 5.8sec
lazy() 9MB 5.4sec
chunk() by 500 6MB 7.56sec

so is 75mb versus 6mb for 100.000 records for each chunk of 500 (the only cons is the time that is a little bit higher

but i've got the point 🙂