# What is the name of this search algorithm?

I was thinking about an efficient binary search for unsorted arrays with $$n$$ entirely unique elements, and came up with something that probably already exists. Here's how it works:

At each level of recursion (starting with $$k=1$$), you have $$2^{k-1}$$ pairs of searchers, taking up some $$O(1)$$ memory each. In each pair, there's one that searches to the left, and one that searches to the right, like so:

| denotes a searcher, <> for direction

|> <|

These two searchers look for the element in question until they meet in the middle. You could probably implement the recursive step iteratively, so you're not using memory on idle searchers.

If the distance between the two is large, you can subdivide:

|> <||> <|

And so on until the element is found. This could be parallelized, since each searcher has a computable range of indices it searches over. You would have to synchronize in some way to prevent the "other" branches from continuing to search once you found the element, which is probably difficult to pull off in practice.

I believe this could find the element in $$O(\log(n))$$ time, using $$O(\log{n} \cdot 2^{\log{n}}) \approx O(n \log{n})$$ memory (which is a lot, but it's unsorted).

What's the name of this algorithm?

• What was the rationale behind the backward search as opposed to just doing the forward search? Commented Dec 31, 2023 at 18:13
• It was just something that felt like a 2x speedup "out of the gate", but I understand that it's not faster than simple chunking Commented Dec 31, 2023 at 20:14

It's a parallel linear search, with an over-complicated way to divide up the array into a power-of-2 number of chunks. (Since you only split in half with multiple levels of recursion, instead of the usual chunk_size = n/cores_available which can keep all cores busy even if their number isn't a power of 2.)

In terms of algorithmic complexity, it's the same as a typical parallel linear search with cores_available rounded down to a power of 2. The differences are in the engineering choices for starting threads and looping over a chunk of an array in two directions.

It doesn't require any extra space, unless you count the overhead of the OS tracking extra threads. Or if you're counting how much memory in the existing array it touches that a simple linear search from the start wouldn't. (But with cancellation, if the match is uniformly placed, on average it will touch about the same amount since it'll typically be found about half-way through each thread's search chunk. So cancellation latency will let them touch a bit more.)

### Looping both ways vs. the standard increasing-address direction

If bidirectional meet-in-the-middle looping was a faster on real CPUs, we'd hopefully already be doing it for all kinds of problems like array sums and dot-products, and a[i] = f(b[i]) problems like level 1 BLAS operations. (And compilers would be optimizing simple loops into loops that went both ways.) It's worth measuring to see if this has any advantages on any mainstream CPUs; perhaps this is something compilers should start doing!

(Compilers probably won't auto-vectorize as easily if you write the source code to loop both ways. Current compilers won't auto-vectorize search loops at all, only loops whose trip-count can be calculated ahead of the first iteration. So they only auto-vectorize loops where they can already tell that the loop touches every element in a range once; other loop optimizations could be conditional on that. You can write a search that always touches all elements, updating a found_position you check at the end. That can be appropriate for tiny arrays, especially fixed-size, both to avoid branch misprediction and to work around the inability of current compilers to make asm that searches efficiently the way hand-written libc memchr does, e.g. with x86 pcmpeqb / pmovmskb to get a bitmask of 16 byte-compare results that you can branch on. SIMD linear search with no data-dependent branching can even be optimal for small sorted arrays, although that enables tricks like counting elements where a[i] < key which is faster than blending match positions.)

Searching both ways from the starting point makes sure you read the whole cache line while it's still hot. But doing so from different CPU cores (thread) only benefits from an outer level of shared cache (e.g. L3) if the two cores share any cache at all. (Which they might not on a multi-socket system, or on an AMD with cores in different CCX clusters). You'd like to benefit from per-core private caches (L1d and L2 in many modern CPUs.) So you'd more likely want to align your boundaries to cache-block boundaries. Or page boundaries (e.g. 4k) for TLB locality.

IIRC, Intel's L1d hardware prefetcher does work better for increasing address (looping "forward").

For memory bandwidth after starting, looping forwards and backwards is probably not harmful on modern x86 CPUs at least: Intel's L2 streamer HW prefetcher can track one forward and one backward stream per 4K page; see a quote from Intel's optimization manual.
It might even help memory-level parallelism to be crossing a 4K boundary in one part but not the other (where HW prefetch stops because contiguous virtual addresses might not be contiguous in physical address space). So one HW prefetch stream can still be working and keeping the memory pipeline fully utilized. (But we're parallelizing, and multiple cores can already saturate DRAM controllers, or with L3 hits, probably the interconnect between cores.) Out-of-order exec usually sees far enough ahead to get demand loads in flight from across the 4K boundary. The limiting factor in per-core memory bandwidth is usually latency and the number of outstanding requests one core can track, or DRAM bandwidth if you miss in last-level cache for huge arrays.

Normally you don't need to parallelize the thread-starting part, you just loop in the main thread until you've started all the workers. If that's significant compared to the actual work being done, use fewer threads (set a minimum chunk size) and/or have a pool of worker threads to reduce overhead of assigning work to them for jobs other than the first.

Maybe on an architecture that's significantly different from modern mainstream CPUs, where vastly more cores are available and it's cheaper to create threads. (But you still have to do so one at a time, unlike a GPU where things naturally run in parallel.)

Starting threads as you recurse probably makes it harder in practice to keep track of them for when you want to .join() them to make sure they've finished, or send them notifications or signals.

If you were going to do any rounding of the number of tasks or whatever, you might round the chunk size to a multiple of the page size so each thread makes the most use of its dTLB coverage. Or maybe work in somewhat smaller chunks like OpenMP "dynamic" scheduling so differences in CPU speed or scheduling don't leave one thread with a lot of work to do while the others are done. (Normally that's most useful for problems where the amount of work per chunk is actually variable, not just because of competing with other load on the system. n/k chunk size is what OpenMP calls "static" scheduling, where all the work is handed out to worker threads right away.)

### Cancelling after one thread finds a match:

CPUs and OSes can do this easily and efficiently, it's just a question of whether the necessary tools are exposed in the high-level language you're using. It's often the case that the CPU being able to do something efficiently doesn't mean it's easy to write a portable program that does it efficiently, but I think we're ok in modern C++ for this.

One way is to have each thread check a shared atomic flag (like C++ std::atomic<bool> with memory_order_relaxed) every 1K or 4K bytes. It will normally stay hot in the L1d cache of all cores, invalidated (MESI) only when it's written by the thread that found the match. If the search speed is limited by cache or memory bandwidth, that won't slow us down. (Modern x86 can search faster than L2 cache bandwidth, if you use SIMD effectively like AVX2 to check 32 bytes at once, and aggregate some results to branch maybe every 64 or 128 bytes, like glibc memchr does. Or an example with intrinsics of searching for either of two bytes at odd positions.)

Or ask the OS to interrupt or kill the other threads, like via C++20 std::jthread which has a request_stop, or POSIX pthread_cancel with async cancellation enabled. (Or pthread_kill to send SIG_USR1 or something and exit the thread in the signal handler, or go to sleep to wait for the next batch of work if you're starting a pool of worker threads instead of creating/destroying OS-level threads for every parallel operation.) This has literally no overhead during the actual search work.

I do not think there is a name for that particular algorithm, but I think it will achieve similar performance to much simpler parallel algorithms for this task. In general when designing parallel algorithms, you assume that you have $$k$$ cores at your disposal. Then, if a task can be accomplished in time $$O(f(n))$$ using one core, the gold standard for a parallel algorithm is to accomplish the same task in time $$O(f(n)/k)$$. The algorithm which you describe will search a list in time $$O(n/k)$$, so in that sense it is optimal because non parallel search requires $$O(n)$$ time with one core. It is worth noting though that you do not need to run the backward and forward search. If you just break the list up into $$k$$ chunks, and search them individually, that will also get you a runtime of $$O(n/k)$$. Making the other cores stop working after the desired element is found will not affect the runtime in theory, and is not too difficult to do in practice if parallelism is implemented in a reasonable way.

• Good point about the not stopping the other cores, hadn't considered that! That would definitely simplify the implementation a lot. Your point about chunks and O(n) is also something I hadn't considered! Commented Dec 31, 2023 at 1:11