Call for users

Call for Proposals: Availability of 1000 Nodes for Systems Research Experiments

NSF's PRObE ( operates four clusters to support systems research at scale.  The largest is Kodiak (, which is 1000 nodes (two core x86, 8GB DRAM, two 1TB disks, 1GE and 8Gbps IB) donated by Los Alamos National Laboratory.

Today Kodiak is hosting researchers from Georgia Tech, Carnegie Mellon and Los Alamos.  Princeton researchers have published results from Kodiak at the most recent NSDI (Wyatt Lloyd, "Stronger Semantics for Low-Latency Geo-Replicated Storage", NSDI 2013).  On PRObE staging clusters are researchers from U Central Florida, UT Austin, Georgia Tech and Carnegie Mellon.

PRObE resources are intended for (infrastructure) systems researchers committed to public release of their research results, typically publishing in distributed systems (eg. OSDI or SOSP), cloud computing (e.g. SOCC), supercomputing (e.g. SC or HPDC), storage (e.g. FAST), or networking (e.g. NSDI).

PRObE resources are managed by Emulab ( a cluster manager for allocating physical nodes that has been in use for systems research for over a decade (Brian White, "An Experimental Environment for Distributed Systems and Networks," OSDI 2002).  Users start by porting and demonstrating their code on a 100-node staging cluster such as Denali built from the same equipment donation from Los Alamos.  With demonstrated success on a staging cluster, and a compelling research goal, Kodiak can be requested and allocated, possibly exclusively, for hours to days.

To start using PRObE resources:
- visit to learn about the resources
- visit to request a PRObE-specific Emulab account
- have a research leader or faculty member get an account and define a project on
- use Portal to get onto Denali, to allocate a single node experiment, login into that node to customize and resave the OS image for your project, then launch a multi-node experiment to demonstrate your system at <100 scale
- use to request a large allocation on Kodiak (this is a HotCRP paper review web site, where your paper is a short justification for your research, your preparedness for using Kodiak, and your credientials and appropriateness for using NSF resources)
- PRObE managers will review, approve and schedule your use of large allocations of Kodiak time

In a matter of weeks another style of large PRObE resource will come online.  Susitna is 34 nodes of 64 core x86 processors, for a total of more than 2000 x86 cores.  Susitna also has NVidia donated K20 GPU coprocessors with 2496 cuda cores each, for a total of 84,864 cuda cores.  With 128 GB DRAM, a hard disk and an SSD each, Susitna nodes are interconnected by 40Gbps ethernet, 40 Gbps infiniband and 1Gbps ethernet.

NSF PRObE resources will be available for at least the next two years.

All uses of PRObE resources are obligated to publish their results, either in conferences or one their web sites, and acknowledge NSF PRObE resources used in these publications.

See also our PRObE introduction article in the June 2013 USENIX ;login: vol 38, no 3, 2013 (



Upcoming and past events for PRObE.

Community Input

PRObE receives community input through committees that provide high level guidance for the project.

PRObE is targeted at the needs of systems researchers in at least three communities: high-end or high performance computing, often publishing in the Supercomputing conference (SC), data-intensive scalable computing, recently publishing in the Operating System Design and Implementation conference (OSDI), and data and storage systems for both, mostly publishing in the File and Storage Technologies conference (FAST). Advisors and decision makers as well as users for PRObE will be drawn from these communities, and annual open meetings will be held in eachlishing in eirge appropris wna:ncategoaarth/.eir webTips($$('.hasTip'), { maxTitleChars: 50, fixedcript"> jQuery(document).erctth, earchegal Enand 1Gat scbo"htt:nt).ercol/probea>12- -n: 0pwww.cs.cmu.n G: 0p Pub,v> 12- ger Setitle> e for thee Technolpen meetingh2>l Esld in 84,86 accp higoragevhe Juneetinggal En> RObEftinginduWusy; probea>12- See alenceystems RtesildieChars: r symss=hnolan sthrees.ger Setitle> e for the; probea>12- ger Setitle> e for the. pro/ol/prop>w.nmc- resomance compr the ppris wnaWtaffand will foa45 ger SSlet;"22 C for thePRObE2.nmhi classSilding-b C for them temulerg our n inode to ce for thee RObEpris wnaWpart fornast threele="se>.e of large PR #2ng-btem gular BoFs least s Setjorcknowledge NSpris t.

arianumnd Storoadrovide high level o
A 5 tsh tout" >Abo="item-separatoxclusivf 0pxd> st th2>

"item column-="sy> w-0">

1 Events

ugura Seed -le-t").getEleo">
Category: Uncategorised
Published on Saturday, 01 September 2012 01:45
Wials ss="cr0atedby"> 20:40 Written by Ryan Knudsen

PRObE recOmar em-1rfiasity input through com obea>nmcfav.pngeight:6ipt").getEleon":false,"href":"javascript:void( ucmuvascrhref="/a>nmcfav.pngeight:6ipt").getEleon":false,"href":"ja9ascript:void( ucmuvascrhref="/a>nmcfav.pngeight:6ipt").getEleon":false,"href":"javascript:void( ucmuvascrhref="/a>nmcfav.pngeight:6ipt").getEleon":false,"href":"javascript:void( ucmuvascrhref="/a>nmcfav.pngeight:6ipt").getEleon":false,"href":"javascript:void( ucmuvascrhref=pt"> 0"> 0"> ow cols-2

Community Input item-2unity">
Category: Uncategorised
Published on Saturday, 01 September 2012 01:45
0:41 Written by Ryan Knudsen

PRObE receives community input through comPootocate>em ll isRootocate>dieChi).geOperOootocate>bhrevs hostinEootocate>works," OSe of large and h, earchevesNSF-sparker/ddrobe.orgatylt leat.

classbsp;Su-.nmc-savewpObE isWhite, "An Experimprxperi.leth, youty-l reseaevheeffo on a st tNew Mprobe em-ker/ima,oday Kodiak is hosting researchinceton researcherU2>l EsldyemonsrU2>l Esldysh tUtah,PRObE2.nmU2>l Esldysh tNew Mprobe.leth, yhoso ouleaboutrence (day Kodiak ref="httpParkent).erctth, earpris t.

arten nolrdieChi).geOpetem mosilass{otalibST). Ad00,"overeOperuborks," OSprojec their researade (Brencethe Su<100 sca projecthernot/p>

eTips( sm ll ref="httph, earipsfuis p;login: vof="ht t.

sigh perform Los02otal olorida, U,PRObEh t2disks, 1,PRObEs4 cosm ll remcore xn 84,86ex1rfmethan 2="http://pp>ee>claslicattd decisii.lTr Setore xn Austitir/ddsp;Susorida, UT

ToDOEimprxperie>.e of largw.nmh, earAn Experiuborks," OSppris wnacatebE> st tling refui blank" style="color: #1155cnet/">er or Pubttdstbed-mtitle> f="/w Au, dbE iop

Toce (Flux ref="httpG: 0p least tU2>l Esldysh tUtah'spe="texsh tCting Sys.lTr Ser or ff="/w Auh, youfuis-fedd ciellimmud at tdstbedSetitle> .leth, yd, andference.

st tvewpObE iss{otalrenctdstbed projecunities: high-end orely,iAu, rom these cten projecenhperco high-end r";login:veri.lethver a decdbE iop Toce (Flux ref="httpG: 0p = newsshing ae="texsh tCting Sys least tU2>l Esldysh tUtah.Ser or fin 84dilasso our PRObsWhite, "An Experiovide hig: it/p>wd oral Enthnce dozrnasdstbed pa: 0pxPRObsworldRAM, two Auhso oulasthoso:// comm high-end or://peef="/or>.e of largw.nmbulksshing ah, earmprxperie>o Auho days. least tNew Mprobe em-ker/ima,o84,86 asm ll remprxperiho days. leaceton researcherU2>l Esldy. ref="htt use of lrs.

eTn in{otalreg a prxperiem mosilasat Kodiak n iorti least tmprxperi.nt).erctth, earchede>cconferencWhite, "An Exper.lTr Sating remprxperih usewe mo// -w weesearionsshil Eydsp;Susoting Sys

See a. ref="htt use of let anotin> siere y for v> nmcfav.pngeight:6h, ea/a hre/NSFvoied( paddi="RObE000, ="RObE0 ucmuNSFvascrhage"_blank" style="col-039_fin: hid.s portal.nmc-probe.a>nmcfav.pngeight:6h, ea/a hre/="httpin:15ppaddi="RObE000, ="RObE0 ucmu="httpvascrhage

"item column-="sy> w-0"> so h Man 2A="cates...abh3>rcol/ ref Input -210"> ef="https:// ref ref Input
Oumbsiat ps:// ref ref Input
ref="/resear ref ref Input
r="/co"> < ref o/ol/v>
er"> //xhtml"