References
Index
ChunkSplitters
ChunkSplitters.chunks
— Functionchunks(itr; n::Union{Nothing, Integer}, size::Union{Nothing, Integer} [, split::Symbol=:batch])
Returns an iterator that splits the indices of itr
into n
-many chunks (if n
is given) or into chunks of a certain size (if size
is given). The keyword arguments n
and size
are mutually exclusive. The returned iterator can be used to process chunks of itr
one after another or in parallel (e.g. with @threads
).
The optional argument split
can be :batch
(default) or :scatter
and determines the distribution of the indices among the chunks. If split == :batch
, chunk indices will be consecutive. If split == :scatter
, the range is scattered over itr
.
If you need a running chunk index you can combine chunks
with enumerate
. In particular, enumerate(chunks(...))
can be used in conjuction with @threads
.
The itr
is usually some iterable, indexable object. The interface requires it to have firstindex
, lastindex
, and length
functions defined, as well as ChunkSplitters.is_chunkable(::typeof(itr)) = true
.
Examples
julia> using ChunkSplitters
julia> x = rand(7);
julia> collect(chunks(x; n=3))
3-element Vector{UnitRange{Int64}}:
1:3
4:5
6:7
julia> collect(enumerate(chunks(x; n=3)))
3-element Vector{Tuple{Int64, UnitRange{Int64}}}:
(1, 1:3)
(2, 4:5)
(3, 6:7)
julia> collect(chunks(1:7; size=3))
3-element Vector{UnitRange{Int64}}:
1:3
4:6
7:7
Note that chunks
also works just fine for OffsetArray
s:
julia> using ChunkSplitters, OffsetArrays
julia> x = OffsetArray(1:7, -1:5);
julia> collect(chunks(x; n=3))
3-element Vector{UnitRange{Int64}}:
-1:1
2:3
4:5
julia> collect(chunks(x; n=3, split=:scatter))
3-element Vector{StepRange{Int64, Int64}}:
-1:3:5
0:3:3
1:3:4
ChunkSplitters.getchunk
— Methodgetchunk(itr, i::Integer; n::Union{Nothing,Integer}, size::Union{Nothing,Integer}[, split::Symbol=:batch])
Returns the range of indices of itr
that corresponds to the i
-th chunk. How the chunks are formed depends on the keyword arguments. See chunks
for more information.
Example
If we have an array of 7 elements, and the work on the elements is divided into 3 chunks, we have (using the default split = :batch
option):
julia> using ChunkSplitters
julia> x = rand(7);
julia> getchunk(x, 1; n=3)
1:3
julia> getchunk(x, 2; n=3)
4:5
julia> getchunk(x, 3; n=3)
6:7
And using split = :scatter
, we have:
julia> using ChunkSplitters
julia> x = rand(7);
julia> getchunk(x, 1; n=3, split=:scatter)
1:3:7
julia> getchunk(x, 2; n=3, split=:scatter)
2:3:5
julia> getchunk(x, 3; n=3, split=:scatter)
3:3:6
We can also choose the chunk size rather than the number of chunks:
julia> using ChunkSplitters
julia> x = rand(7);
julia> getchunk(x, 1; size=3)
1:3
julia> getchunk(x, 2; size=3)
4:6
julia> getchunk(x, 3; size=3)
7:7
ChunkSplitters.is_chunkable
— Methodis_chunkable(::T) :: Bool
Determines if a of object of type T
is capable of being chunk
ed. Overload this function for your custom types if that type is linearly indexable and supports firstindex
, lastindex
, and length
.
Interface requirements
Support for this minimal interface requires version 2.2.0.
For the chunks
and getchunk
functions to work, the input value must overload ChunkSplitters.is_chunkable(::YourType) = true
, and support the functions Base.firstindex
, Base.lastindex
, and Base.length
.
For example:
julia> using ChunkSplitters
julia> struct MinimalInterface end
julia> Base.firstindex(::MinimalInterface) = 1
julia> Base.lastindex(::MinimalInterface) = 7
julia> Base.length(::MinimalInterface) = 7
julia> ChunkSplitters.is_chunkable(::MinimalInterface) = true
julia> x = MinimalInterface()
MinimalInterface()
julia> collect(chunks(x; n=3))
3-element Vector{UnitRange{Int64}}:
1:3
4:5
6:7
julia> collect(chunks(x; n=3, split=:scatter))
3-element Vector{StepRange{Int64, Int64}}:
1:3:7
2:3:5
3:3:6