StreamRec: - VLDB

RecBench: Benchmarks for Evaluating Performance of Recommender System Architectures Justin Levandoski Michael D. Ekstrand Michael J. Ludwig Ahmed Eldawy Mohamed F. Mokbel John T. Riedl Outline Overview of recommender systems

RecBench motivation RecBench benchmark tasks Recommender system architectures RecBench experimental results Conclusion Outline Overview of recommender systems RecBench Motivation RecBench benchmark tasks Recommender system architectures RecBench experimental results Conclusion

Recommender Systems Basic Idea Users: provide opinions on items consumed/watched/listened to The system: provides the user personalized suggestions for new items Collaborative Filtering (CF) ? ? ? Item-Based CF Model Building Similarity measures m1 m2

m3 similarity(m1,m3) similarity(m1,m2) Cosine distance Pearson correlation Spearman correlation Adjusted cosine distance ... .6 .8 .5 , , .8

.9 .5 , , .6 .7 .5 , , ... .3 , , .4

.4 , Item-Based CF Recommendations .6 .8 .5 , , .8 .9 .5 , ,

.6 .7 .5 , , .6 .9 .5 , , .6 .7 , .5

, .3 , , .4 .4 , .5 , .3 , Item-Based CF Recommendation ( ( ( .8

X 2 + .6 .9 X 2 + .5 X 4

+ .5 .7 ) / (.8 + .6) = 2.86 X 4) / (.9 + .5) = 2.71 X 2 ) / (.7 + .5) = 3.17 X 4 Outline

Overview of recommender systems RecBench Motivation RecBench benchmark tasks Recommender system architectures RecBench experimental results Conclusion RecBench Motivation Traditionally, recommender systems = customized software Example: Movielens Can we use relational DBMS primitives to build a recommender? No work has explored recommender system performance Performance has always been synonymous with quality [Our] solution is based huge amount of models and We have

chosen notontoadiscuss computation predictors which not be practical as part of aSuch commercial performance ofwould recommender algorithms. recommender system. However, this result is a direct performance is certainly important, and in the future we consequence of the nature and goal of the competition: obtain expect there to beaccuracy work on quality of time-limited the highest

possible at the any cost, disregarding completely and memory-limited recommendations. the complexity of the solution and the execution performance." BelKors Pragmatic ChaosFiltering Herlocker etTeam al. Evaluating Collaborative Winner of the ACM 2009TOIS Netflix Prize Recommender

Systems, 2004 RecBench Motivation Goals of RecBench: Prompt research community to explore space of alternative architectures for building recommender systems and study their performance Propose a benchmark to test performance (efficiency) of different system architectures Outline Overview of recommender systems

RecBench Motivation RecBench benchmark tasks Recommender system architectures RecBench experimental results Conclusion RecBench Tasks http://www.movielens.org Task 1: Initialization Initialization Task Prepare system to start serving user recommendations Stored Recommendation Model Task 2: Pure Recommend

Pure Recommend Task Produce top-k recommendations from systems entire item pool Task 3: Filtered Recommend Filtered Recommend Task Produce top-k recommendations that match constraints on item metadata Task 4: Blended Recommend Blended Recommend Task Produce top-k recommendations based on blended text-search and recommendation score Task 5: Item Prediction Item Prediction Generate a users predicted rating for a target item

Task 6: Item Update Item Update Incorporate new item(s) into the system for recommendation Outline Overview of recommender systems RecBench Motivation RecBench benchmark tasks Recommender system architectures RecBench experimental results Conclusion

Recommender System Architectures Unmodified DBMS MultiLens Hand-built system UMN recommender software suite (basis for MovieLens) Code optimized for item-based CF Uses DBMS for metadata and textsearch queries

PostgreSQL database Ratings relation: RecStore DBMS Database modified to optimize for fast ratings(usr,itm,rating) recommender model updates Model relation: SQL same as model(itm, rel_itm, sim) unmodified DBMS All tasks implemented in approach standard SQL (details in

Maintain partial paper) statistics to update rec. Metadata and text-search model incrementally as functionality built in new ratings arrive (e.g., selection) Outline Overview of recommender systems RecBench Motivation RecBench benchmark tasks Recommender system architectures

RecBench experimental results Conclusion Benchmark Setup Datasets MovieLens 10M movie ratings ~10K movies, ~70K users Architectures Netflix Challenge 100M movie ratings ~18K movies, ~480K users MultiLens Unmodified DBMS

RecStore DBMS Metric Metric is response time For recommendation queries, time is average of sample of 2.5% of total users broken down by quartile Same timing/sampling technique used for items for prediction task Tasks

Item-based collaborative filtering used as recommendation technique Tasks implemented specific to data from movie domain Run on 4-way 3 GHz Intel Xeon system with 48 GB or RAM Task 1: Initialization Task 1: build collaborative filtering model using movie ratings Netflix 100M MovieLens 10M 35000 60000 25000 50000 20000 40000 15000

10000 5000 3255 1709 0 Architecture 426 Time (sec) Time (sec) 30000 70000 30255

30000 61320 44666 23459 20000 10000 3371 0 Architecture Task 2: Pure Recommend Task 2: retrieve k recommendations using all possible movies Netflix 100M

MovieLens 10M DBMS/RecStore MA MultiLens RecStore MI DBMS/RecStore MA MultiLens 14 10 Time (ms) Time (ms) 12 8 6 4 2 0

quartile 1 quartile 2 quartile 3 quartile 4 User Quartile 50 45 40 35 30 25 20 15 10 5 0 RecStore MI quartile 1 quartile 2 quartile 3 quartile 4 User Quartile

Task 3: Filtered Recommend Task 3: retrieve k recommendations of only 1990s comedies Netflix 100M MovieLens 10M 20 18 16 14 12 10 8 6 4 2 0 RecStore MI

DBMS/RecStore MA MultiLens RecStore MI 16 14 Time (ms) Time (ms) DBMS/RecStore MA MultiLens 12 10 8 6 4 2

quartile 1 quartile 2 quartile 3 quartile 4 User Quartile 0 quartile 1 quartile 2 quartile 3 quartile 4 User Quartile Outline Overview of recommender systems RecBench benchmark tasks Recommender system architectures RecBench experimental results

Conclusion Future Work Just scratched the surface on recommender system architecture and performance work More performance studies Multi-user experiments Different recommendation techniques Recommender system architectures Hybrid systems Recommendation-specific operators [Koutrika et al, SIGMOD09] Conclusion and Summary http://lenskit.grouplens.org Recbench available as part of LensKit project from University of Minnesota Questions?

Recently Viewed Presentations

  • SAP functionality - 123seminarsonly.com

    SAP functionality - 123seminarsonly.com

    SAP AG company is a market leader in Enterprise Applications software.SAP is listed on the New York Stock Exchange (NYSE) under the symbol SAP.. Nearly . 30 industries. are covered by . SAP industry. solutions. SAP AG has offices in...
  • Craven: Fundamentals of Nursing - Human Health and Function ...

    Craven: Fundamentals of Nursing - Human Health and Function ...

    Torts and Crimes (Cont'd) Unintentional torts . Negligence [omission or commission] Duty and breach of duty. Proximate cause. Damages. Trends in nursing malpractice . Increase in nurses being sued for malpractice. Medication errors, falls, surgical errors. Higher standards of care....
  • SPED Technical Sessions

    SPED Technical Sessions

    Calculating equipment and component loads. Sizing supports. 10/27/2011. 2011 SPED Technical Briefs Pipe Stress for Pipers. What the Designer Does/Can Do. Size pipe (OD) Based on process - flow rate, fluid, & pressure (drop) Select material.
  • FIRM Exit Strategy - United States Department of the Treasury

    FIRM Exit Strategy - United States Department of the Treasury

    FIRM Exit Strategy Development of FIRM was undertaken when few COTS packages for real property existed FIRM was developed as a real property inventory management tool Agencies have an ever-growing need for an asset management tool as opposed to an...
  • Science of Crime Scenes - Elsevier

    Science of Crime Scenes - Elsevier

    Drugs. A drug is a natural or manufactured substance that affects physiological or psychological behavior. Many are legally prescribed for therapeutic or medical purposes
  • Human resource management - UNIVERSITY OF MEDITERRANEAN KARPASIA

    Human resource management - UNIVERSITY OF MEDITERRANEAN KARPASIA

    Trends in Human Resource Management. Technology. ... as business partners, stewards of an effective organizational culture, a business ally and strategic partner. As the profession of HR continues to change, new competencies will be added to the list. ... There...
  • Approximations to Inverse Acoustic 1-d Primary Scattering

    Approximations to Inverse Acoustic 1-d Primary Scattering

    DATA-DRIVEN INVERSION/DEPTH IMAGING DERIVED FROM APPROXIMATIONS TO 1-D INVERSE ACOUSTIC SCATTERING "DATA HAVE A STORY TO TELL IF THEY'RE ASKED THE RIGHT QUESTIONS" (Art Weglein) Lasse Amundsen, Arne Reitan, and Børge Arntsen
  • Sexual Harassment Prevention Training Webinar - 6/1/2017 9 to ...

    Sexual Harassment Prevention Training Webinar - 6/1/2017 9 to ...

    Exercise 1 Test. Choose the best answer: ... Olive can sue both Pat and ABC, but ABC can try to reduce the amount it must pay by arguing that the harassment would have stopped if Olive called HR after Pat's...