AWS Reserved Instance Marketplace (RIM) Data

The following trace contains AWS reserved instance marketplace (RIM) data from every AWS region for every VM type at 30 minute frequency starting from September-2018 to May-2020. Data collection is ongoing, so check this page for future updates.

AWS-Reserved-Marketplace-Data.zip

Note that we cannot track actual sales of reserved instances in RIM versus listing cancellations, but can only observe when listings come on and go off the market.

Traces

All traces are in CSV format. Data have been collected with simple python script using EC2’s Boto3 API (specifically, used describe-reserved-instances-offerings API call to collect the data). Check AWS documentation for further details on data fields recorded in the data.

Our HotCloud’20 paper No Reservations: A First Look at Amazon’s Reserved Instance Marketplace, provides an initial analysis of the RIM data. Please cite this paper when making use of the data.

Data Fields

  • TimeStamp: The time of recorded data. Duration: The duration of the Reserved Instance, in seconds.
  • FixedPrice: The purchase price of the Reserved Instance.
  • InstanceType: The instance type on which the Reserved Instance can be used.
  • ProductDescription: The Reserved Instance product platform description.
  • ReservedInstancesOfferingId: The ID of the Reserved Instance offering. This is the offering ID used in GetReservedInstancesExchangeQuote to confirm that an exchange can be made.
  • UsagePrice: The usage price of the Reserved Instance, per hour.
  • CurrencyCode: The currency of the Reserved Instance offering you are purchasing. It’s specified using ISO 4217 standard currency codes. At this time, the only supported currency is USD.
  • InstanceTenancy: The tenancy of the instance. Marketplace: Indicates whether the offering is available through the Reserved Instance Marketplace (resale) or AWS. If it’s a Reserved Instance Marketplace offering, this is true.
  • OfferingClass: If convertible it can be exchanged for Reserved Instances of the same or higher monetary value, with different configurations. If standard , it is not possible to perform an exchange.
  • OfferingType: The Reserved Instance offering type.
  • PricingDetails: The pricing details of the Reserved Instance offering. This field is a list and this list contains the count (number of reservations available for the price) and price (the price per instance).
  • RecurringCharges: The recurring charge tag assigned to the resource. This field is a list and this list contains amount (the amount of recurring charge) and frequency (frequency of recurring charge).
  • Scope: Whether the Reserved Instance is applied to instances in a Region or an Availability Zone.

Publications

  • No Reservations: A First Look at Amazon’s Reserved Instance Marketplace. Pradeep Ambati, David Irwin, and Prashant Shenoy. 12th USENIX Workshop on Hot Topics in Cloud Computing (HotCloud 20), BOSTON, MA, USA.

MGHPCC Production Trace

Job trace includes 14M jobs from a production high performance computing cluster consisting of 14,376 cores. The cluster is the University of Massachusetts (UMass) System Shared Cluster, and is available for general use to researchers from all five campuses in the UMass system, including its medical school. The cluster is located at the Massachusetts Green High Performance Computing Center (MGHPCC), a 15MW data center in Holyoke, Massachusetts that also hosts computing infrastructure for Boston University, Harvard, MIT, and Northeastern. The cluster runs the LSF job scheduler, and the included trace is production log from the year 2016.

Trace

traces-2016.h5

Each job entry in trace includes its submission time, user ID, maximum running time limit, requested number of cores and memory, and running time. Trace is formatted in hierarchical data format (hdf format). Note that, hdf format is supported by popular programming languages like C++, Java, Python etc using either third party APIs or respective language in-built APIs.

Please find the description of the important fields in the trace below:

  • userid: The user identification number. (Integer) wallclock_runtime_sec: Actual job runtime in seconds. (Integer)
  • wallclock_limit_sec: Maximum runtime limit specified by the user. (Integer)
  • num_cores: Number of CPU requested for the job. (Integer)
  • total_MB_req: Memory in MB requested for the job. (Integer)
  • status: Status of the job (String)
  • time: Timestamp when job queued. (Timestamp)

Our SC’20 paper Waiting Game: Optimally Provisioning Fixed Resources for Cloud-Enabled Schedulers, provides an analysis of the batch trace. Please cite this paper when making use of the data.

Trace Simulator

We implemented trace-driven job simulator in python that mimics a cloud-enabled job scheduler, which can acquire VMs on-demand to service jobs. The simulator uses a FCFS scheduling policy, and also implements each of our waiting policies. We have implemented NJW (no jobs waiting), AJW (all jobs waiting), AJW-T (all jobs waiting threshold), SWW (short waits wait), LJW (Long jobs waiting), and Compound policy.

Github link: https://github.com/sustainablecomputinglab/waitinggame

Publications

  • Waiting Game: Optimally Provisioning Fixed Resources for Cloud-Enabled Schedulers. Pradeep Ambati, Noman Bashir, David Irwin, and Prashant Shenoy. In International Conference for High Performance Computing, Networking, Storage and Analysis (SC), November 2020.

Back to top

Copyright © 2016-2024 University of Massachusetts Amherst.