4

In general, you can look at hard combinatorial instances. If 50 variables is not a hard limit, try e.g., an instance arising from combinatorial block design having 105 variables described in [1]. The actual instance in DIMACS format is here provided by the authors. For the instance, the authors write in [1] that: "We issue a challenge for any solver to solve ...


3

See the international SAT solver competition. It's a competition that tests various solvers by running them on a library of hard SAT instances. You can download the instances they used to evaluate the SAT solvers. I would suggest you look at these competitions and see if any of the instances they've used would meet your needs.


3

As far as I know, they give peak performance values. Given clock speed $f$ (in Hz) and number of cycles per (shortest) floating point operation $c$, the peak performance is essentially $f \cdot c$. Of course, modern machines execute multiple floating point operations in parallel, have multiple cores etc. A more accurate formula can be found on Wikipedia. ...


3

While benchmarking metal is mostly outside of the realms of computer science -- we usually don't want to worry about issues of materials and physics -- there are some general remarks here. Tl;dr: benchmarks are inherently fickle. Your benchmark process is not alone on the machine. Other processes and interrupts can cause your process to be stalled non-...


2

I've been visiting all the links provided by Nick. They do look wonderful indeed and I have added all those sites to my bookmarks. Hope that the following link especially designed to test search algorithms suits your needs as well: Pathfinding Benchmarks by Nathan Sturtevant. It contains various maps from different video games and also other artificial ...


1

You can't. You can't squeeze blood out of a stone. If you want to evaluate how well your method will work on real documents, you need real documents. With synthetically/artificially generated documents, you'll never know if they have the same characteristics as real documents. You don't necessarily need a standard benchmark that others have used -- but ...


1

Our approach depends somewhat on which floating point operations we will be counting. I think it makes sense to include additions, multiplications, divisions, and casts. We will count each as one floating point operation. In each loop, we have one add, one multiply, one divide, and two casts. (Perhaps the casting will be optimized--I'll assume not.) So each ...


1

For C, Bentley's "Programming pearls" discusses some pitfalls and gives an (oldish) program to measure costs of several operations. For Java and other JITed languages it's more of a challenge, as the first run of the operation will get it compiled to native code and run that way. In general, the only really relevant benchmark is running your program with a ...


1

You can for instance take as ground truth data the movielens dataset, remove some rating links between users and movies. You can rank your algorithm by counting the number of link that you can guess right. Usually machine learning algorithm also guess the rating score.


1

Apple has just proudly stated that their new mac pro will be able to give up to 7 teraflops of computing power. Flops stands for Floating Point Operations Per Second. How exactly is this benchmarked though? The 7 TFLOPs spec comes from Apple's new MacPro page. The spec is probably based on the workstation having two AMD FirePro W9000's; each according to ...


1

I guess you should ask Apple how they got that figure. For instance, maybe they ran the lapack/linpack benchmarks (or similar) like they do for the Top500. If they're using benchmarks, this will include real-world limiting factors that Raphael mentions. Performance in the range of several teraflops is pretty respectable for a modern consumer system, ...


Only top voted, non community-wiki answers of a minimum length are eligible