Multiprocessor Synchronization - Elsevier

Barrier Synchronization Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit Simple Video Game Prepare frame for display By graphics coprocessor soft real-time application Need at least 35 frames/second OK to mess up rarely Art of Multiprocessor Programming 2

Simple Video Game while (true) { frame.prepare(); frame.display(); } Art of Multiprocessor Programming 3 Simple Video Game while (true) { frame.prepare(); frame.display(); }

What about overlapping work? 1st thread displays frame 2nd prepares next frame Art of Multiprocessor Programming 4 Two-Phase Rendering while (true) { if (phase) { frame[0].display(); } else { frame[1].display(); } phase = !phase; }

while (true) { if (phase) { frame[1].prepare(); } else { frame[0].prepare(); } phase = !phase; } Art of Multiprocessor Programming 5 Two-Phase Rendering while (true) { if (phase) { frame[0].display();

} else { frame[1].display(); } phase = !phase; } while (true) { if (phase) { frame[1].prepare(); } else { frame[0].prepare(); } phase = !phase; } even phases Art of Multiprocessor Programming

6 Two-Phase Rendering while (true) { if (phase) { frame[0].display(); } else { frame[1].display(); } phase = !phase; } while (true) { if (phase) { frame[1].prepare(); } else { frame[0].prepare();

} phase = !phase; } odd phases Art of Multiprocessor Programming 7 Synchronization Problems How do threads stay in phase? Too early? we render no frame before its time Too late? Recycle memory before frame is displayed

Art of Multiprocessor Programming 8 Ideal Parallel Computation 0 0 0 1 1 1 Art of Multiprocessor Programming

9 Ideal Parallel Computation 2 2 2 1 1 1 Art of Multiprocessor Programming 10

Real-Life Parallel Computation 0 0 0 zzz Art of Multiprocessor Programming 1 1 11 Real-Life Parallel Computation 2

0 1zzz 1 Uh, oh Art of Multiprocessor Programming 12 Barrier Synchronization 0 0 barrier

0 Art of Multiprocessor Programming 13 Barrier Synchronization barrier 1 1 1 Art of Multiprocessor Programming

14 Barrier Synchronization barrier Until every thread has left here No thread enters here Art of Multiprocessor Programming 15 Why Do We Care? Mostly of interest to Scientific & numeric computation

Elsewhere Garbage collection Less common in systems programming Still important topic Art of Multiprocessor Programming 16 Duality Dual to mutual exclusion Include others, not exclude them Same implementation issues Interaction with caches Invalidation?

Local spinning? Art of Multiprocessor Programming 17 Example: Parallel Prefix a after b c a

d a+b Art of Multiprocessor Programming before a+b+c a+b+c +d 18 Parallel Prefix One thread

Per entry a b Art of Multiprocessor Programming c d 19 Parallel Prefix: Phase 1 a

b c a d a+b Art of Multiprocessor Programming b+c c+d 20

Parallel Prefix: Phase 2 a b c a d a+b Art of Multiprocessor Programming a+b+c

a+b+c +d 21 Parallel Prefix N threads can compute Parallel prefix Of N entries In log2 N rounds What if system is asynchronous? Why we need barriers Art of Multiprocessor Programming 22

Prefix class Prefix extends Thread { int[] a; int i; Barrier b; void Prefix(int[] a, Barrier b, int i) { a = a; b = b; i = i; } Art of Multiprocessor Programming 23 Prefix

class Prefix extends Thread { int[] a; int i; Barrier b; void Prefix(int[] a, Barrier b, int i) { a = a; Array of input b = b; values i = i; } Art of Multiprocessor Programming 24 Prefix

class Prefix extends Thread { int[] a; int i; Barrier b; void Prefix(int[] a, Barrier b, int i) { a = a; Thread index b = b; i = i; } Art of Multiprocessor Programming 25 Prefix class Prefix extends Thread {

int[] a; int i; Barrier b; void Prefix(int[] a, Barrier b, int i) { a = a; Shared barrier b = b; i = i; } Art of Multiprocessor Programming 26 Prefix class Prefix extends Thread { int[] a;

int i; Initialize fields Barrier b; void Prefix(int[] a, Barrier b, int i) { a = a; b = b; i = i; } Art of Multiprocessor Programming 27 Where Do the Barriers Go? public void run() { int d = 1, sum = 0; while (d < N) {

if (i >= d) sum = a[i-d]; if (i >= d) a[i] += sum; d = d * 2; }}} Art of Multiprocessor Programming 28 Where Do the Barriers Go? public void run() { int d = 1, sum = 0; while (d < N) { if (i >= d) sum = a[i-d];

Make sure everyone reads b.await(); before anyone writes if (i >= d) a[i] += sum; d = d * 2; }}} Art of Multiprocessor Programming 29 Where Do the Barriers Go? public void run() { int d = 1, sum = 0; while (d < N) { if (i >= d)

sum = a[i-d]; Make sure everyone reads b.await(); before anyone writes if (i >= d) a[i] += sum; Make sure everyone writes b.await(); before anyone reads d = d * 2; }}} Art of Multiprocessor Programming 30 Barrier Implementations Cache coherence

Spin on locally-cached locations? Spin on statically-defined locations? Latency How many steps? Symmetry Do all threads do the same thing? Art of Multiprocessor Programming 31 Barriers public class Barrier { AtomicInteger count; int size; public Barrier(int n){

count = AtomicInteger(n); size = n; } public void await() { if (count.getAndDecrement()==1) { count.set(size); } else { while (count.get() != 0); Art of Multiprocessor }}}} Programming 32 Barriers public class Barrier { AtomicInteger count; int size;

public Barrier(int n){ count = AtomicInteger(n); Number of threads size = n; not yet arrived } public void await() { if (count.getAndDecrement()==1) { count.set(size); } else { while (count.get() != 0); Art of Multiprocessor }}}} Programming 33 Barriers

public class Barrier { AtomicInteger count; int size; Number of threads public Barrier(int n){ participating count = AtomicInteger(n); size = n; } public void await() { if (count.getAndDecrement()==1) { count.set(size); } else { while (count.get() != 0); Art of Multiprocessor }}}} Programming

34 Barriers public class Barrier { Initialization AtomicInteger count; int size; public Barrier(int n){ count = AtomicInteger(n); size = n; } public void await() { if (count.getAndDecrement()==1) { count.set(size); } else { while (count.get() != 0); Art of Multiprocessor }}}}

Programming 35 Barriers public class Barrier { AtomicInteger count; Principal method int size; public Barrier(int n){ count = AtomicInteger(n); size = n; } public void await() { if (count.getAndDecrement()==1) { count.set(size); } else { while (count.get() != 0);

Art of Multiprocessor }}}} Programming 36 Barriers public class Barrier { AtomicInteger count; If Im last, reset fields for next time int size; public Barrier(int n){ count = AtomicInteger(n); size = n; } public void await() { if (count.getAndDecrement()==1) { count.set(size);

} else { while (count.get() != 0); Art of Multiprocessor }}}} Programming 37 Barriers public class Barrier { AtomicInteger count; Otherwise, wait for int size; everyone else public Barrier(int n){ count = AtomicInteger(n); size = n; }

public void await() { if (count.getAndDecrement()==1) { count.set(size); } else { while (count.get() != 0); Art of Multiprocessor }}}} Programming 38 Barriers public class Barrier { AtomicInteger count; int size; public Barrier(int n){ count = AtomicInteger(n); size = n;

Whats wrong with this protocol? } public void await() { if (count.getAndDecrement()==1) { count.set(size); } else { while (count.get() != 0); Art of Multiprocessor }}}} Programming 39 Reuse Barrier b = new Barrier(n); while ( mumble() ) { do work work();

synchronize b.await() } Art of Multiprocessor Programming repeat 40 Barriers public class Barrier { AtomicInteger count; int size; public Barrier(int n){ count = AtomicInteger(n); size = n;

} public void await() { if (count.getAndDecrement()==1) { count.set(size); } else { while (count.get() != 0); Art of Multiprocessor }}}} Programming 41 Barriers public class Barrier { AtomicInteger count; Waiting for Phase 1 to finish int size; public Barrier(int n){

count = AtomicInteger(n); size = n; } public void await() { if (count.getAndDecrement()==1) { count.set(size); } else { while (count.get() != 0); Art of Multiprocessor }}}} Programming 42 Barriers 1 public classPhase Barrier

{ is so over AtomicInteger count; Waiting for Phase 1 to finish int size; public Barrier(int n){ count = AtomicInteger(n); size = n; } public void await() { if (count.getAndDecrement()==1) { count.set(size); } else { while (count.get() != 0); Art of Multiprocessor }}}} Programming

43 Barriers Prepare for public class Barrier { ZZZZZ. phase 2 AtomicInteger count; int size; public Barrier(int n){ count = AtomicInteger(n); size = n;

} public void await() { if (count.getAndDecrement()==1) { count.set(size); } else { while (count.get() != 0); Art of Multiprocessor }}}} Programming 44 Uh-Oh public class Barrier { AtomicInteger count;Waiting for Phase 2 to finish int size; public Barrier(int n){

Waiting for count = AtomicInteger(n); Phase 1 to finish size = n; } public void await() { if (count.getAndDecrement()==1) { count.set(size); } else { while (count.get() != 0); Art of Multiprocessor }}}} Programming Basic Problem One thread wraps around to start phase 2 While another thread is still waiting for

phase 1 One solution: Always use two barriers Art of Multiprocessor Programming 46 Sense-Reversing Barriers public class Barrier { AtomicInteger count; int size; boolean sense = false; threadSense = new ThreadLocal public void await { boolean mySense = threadSense.get(); if (count.getAndDecrement()==1) {

count.set(size); sense = mySense } else { while (sense != mySense) {} } threadSense.set(!mySense)}}} Art of Multiprocessor Programming 47 Sense-Reversing Barriers public class Barrier { Completed odd or AtomicInteger count; even-numbered int size; boolean sense = false; phase?

threadSense = new ThreadLocal public void await { boolean mySense = threadSense.get(); if (count.getAndDecrement()==1) { count.set(size); sense = mySense } else { while (sense != mySense) {} } threadSense.set(!mySense)}}} Art of Multiprocessor Programming 48 Sense-Reversing Barriers public class Barrier { Store sense for AtomicInteger count;

next phase int size; boolean sense = false; threadSense = new ThreadLocal public void await { boolean mySense = threadSense.get(); if (count.getAndDecrement()==1) { count.set(size); sense = mySense } else { while (sense != mySense) {} } threadSense.set(!mySense)}}} Art of Multiprocessor Programming 49 Sense-Reversing Barriers

public class Barrier { AtomicInteger count; Get new sense determined int size; boolean sense = false; by last phase threadSense = new ThreadLocal public void await { boolean mySense = threadSense.get(); if (count.getAndDecrement()==1) { count.set(size); sense = mySense } else { while (sense != mySense) {} } threadSense.set(!mySense)}}} Art of Multiprocessor Programming

50 Sense-Reversing Barriers public class Barrier { AtomicInteger count; int size; If Im last, reverse sense boolean sense = false; for next time threadSense = new ThreadLocal public void await { boolean mySense = threadSense.get(); if (count.getAndDecrement()==1) { count.set(size); sense = mySense } else { while (sense != mySense) {} } threadSense.set(!mySense)}}}

Art of Multiprocessor Programming 51 Sense-Reversing Barriers public class Barrier { AtomicInteger count; Otherwise, wait for int size; sense to flip boolean sense = false; threadSense = new ThreadLocal public void await { boolean mySense = threadSense.get(); if (count.getAndDecrement()==1) { count.set(size); sense = mySense } else {

while (sense != mySense) {} } threadSense.set(!mySense)}}} Art of Multiprocessor Programming 52 Sense-Reversing Barriers public class Barrier { Prepare sense AtomicInteger count; int size; next phase boolean sense = false; threadSense = new ThreadLocal for

public void await { boolean mySense = threadSense.get(); if (count.getAndDecrement()==1) { count.set(size); sense = mySense } else { while (sense != mySense) {} } threadSense.set(!mySense)}}} Art of Multiprocessor Programming 53 Combining Tree Barriers 2-barrier 2-barrier

2-barrier Art of Multiprocessor Programming 54 Combining Tree Barriers 2-barrier 2-barrier 2-barrier Art of Multiprocessor Programming

55 Combining Tree Barrier public class Node{ AtomicInteger count; int size; Node parent; volatile boolean sense; public void await() { if (count.getAndDecrement()==1) { if (parent != null) parent.await() count.set(size); sense = mySense } else { while (sense != mySense) {} }}}} Art of Multiprocessor Programming

56 Combining Tree Barrier Parent barrier public class Node{ AtomicInteger count; int size; Node parent; volatile boolean sense; in tree public void await() { if (count.getAndDecrement()==1) { if (parent != null) parent.await() count.set(size); sense = mySense } else { while (sense != mySense) {}

}}}} Art of Multiprocessor Programming 57 Combining Tree Barrier public class Node{ AtomicInteger count; int size; Am I last? Node parent; volatile boolean sense; public void await() { if (count.getAndDecrement()==1) { if (parent != null) parent.await() count.set(size); sense = mySense } else { while (sense != mySense) {}

}}}} Art of Multiprocessor Programming 58 Combining Tree Barrier public class Node{ Proceed to parent barrier AtomicInteger count; int size; Node parent; volatile boolean sense; public void await() { if (count.getAndDecrement()==1) { if (parent != null) parent.await(); count.set(size); sense = mySense } else { while (sense != mySense) {}

}}}} Art of Multiprocessor Programming 59 Combining Tree Barrier public class Node{ Prepare for next phase AtomicInteger count; int size; Node parent; volatile boolean sense; public void await() { if (count.getAndDecrement()==1) { if (parent != null) parent.await() count.set(size); sense = mySense } else {

while (sense != mySense) {} }}}} Art of Multiprocessor Programming 60 Combining Tree Barrier public class Node{ Notify others at this AtomicInteger count; int size; Node parent; volatile boolean sense; node public void await() { if (count.getAndDecrement()==1) { if (parent != null) parent.await()

count.set(size); sense = mySense } else { while (sense != mySense) {} }}}} Art of Multiprocessor Programming 61 Combining Tree Barrier public class Node{ Im not last, so wait AtomicInteger count; int size; notification Node parent; volatile boolean sense;

for public void await() { if (count.getAndDecrement()==1) { if (parent != null) { parent.await() count.set(size); sense = mySense } else { while (sense != mySense) {} }}}} Art of Multiprocessor Programming 62 Combining Tree Barrier No sequential bottleneck

Parallel getAndDecrement() calls Low memory contention Same reason Cache behavior Local spinning on bus-based architecture Not so good for NUMA Art of Multiprocessor Programming 63 Remarks Everyone spins on sense field Local spinning on bus-based (good) Network hot-spot on distributed architecture (bad)

Not really scalable Art of Multiprocessor Programming 64 Tournament Tree Barrier If tree nodes have fan-in 2 Dont need to call getAndDecrement() Winner chosen statically At level i If i-th bit of id is 0, move up Otherwise keep back Art of Multiprocessor

Programming 65 Tournament Tree Barriers root winner winner loser loser winner Art of Multiprocessor Programming

loser 66 Tournament Tree Barriers All flags blue Art of Multiprocessor Programming 67 Tournament Tree Barriers Loser thread sets winners flag

Art of Multiprocessor Programming 68 Tournament Tree Barriers Loser spins on own flag Art of Multiprocessor Programming 69 Tournament Tree Barriers Winner

spins on own flag Art of Multiprocessor Programming 70 Tournament Tree Barriers Winner sees own flag, moves up, spins Art of Multiprocessor Programming 71 Tournament Tree Barriers

Bingo! Art of Multiprocessor Programming 72 Tournament Tree Barriers Sense-reversing: next time use blue flags Art of Multiprocessor Programming 73 Tournament Barrier

class TBarrier { boolean flag; TBarrier partner; TBarrier parent; boolean top; } Art of Multiprocessor Programming 74 Tournament Barrier class TBarrier { boolean flag; TBarrier partner; TBarrier parent;

boolean top; } Notifications delivered here Art of Multiprocessor Programming 75 Tournament Barrier class TBarrier { boolean flag; TBarrier partner; TBarrier parent; boolean top;

} Other thead at same level Art of Multiprocessor Programming 76 Tournament Barrier class TBarrier { boolean flag; TBarrier partner; TBarrier parent; boolean top;

} Parent (winner) or null (loser) Art of Multiprocessor Programming 77 Tournament Barrier class TBarrier { boolean flag; TBarrier partner; TBarrier parent; boolean top; }

Am I the root? Art of Multiprocessor Programming 78 Tournament Barrier void await(boolean mySense) { if (top) { return; } else if (parent != null) { while (flag != mySense) {}; parent.await(mySense); partner.flag = mySense; } else { partner.flag = mySense;

while (flag != mySense) {}; }}} Art of Multiprocessor Programming 79 Tournament Barrier Current sense void await(boolean mySense) { if (top) { return; Le root, cest } else if (parent != null) { while (flag != mySense) {}; parent.await(mySense);

partner.flag = mySense; } else { partner.flag = mySense; while (flag != mySense) {}; }}} Art of Multiprocessor Programming moi 80 Tournament Barrier void await(boolean mySense) { if (top) { return; } else if (parent != null) { while (flag != mySense) {};

parent.await(mySense); partner.flag = mySense; } else { partner.flag = mySense; while (flag != mySense) {}; }}} Art of Multiprocessor Programming I am already a winner 81 Tournament Barrier void await(boolean mySense) { Wait if (top) {

return; } else if (parent != null) { while (flag != mySense) {}; parent.await(mySense); partner.flag = mySense; } else { partner.flag = mySense; while (flag != mySense) {}; }}} Art of Multiprocessor Programming for partner 82 Tournament Barrier void await(boolean mySense) {

if (top) { Synchronize return; } else if (parent != null) { while (flag != mySense) {}; parent.await(mySense); partner.flag = mySense; } else { partner.flag = mySense; while (flag != mySense) {}; }}} Art of Multiprocessor Programming upstairs 83

Tournament Barrier void await(boolean mySense) { if (top) { Inform return; } else if (parent != null) { while (flag != mySense) {}; parent.await(mySense); partner.flag = mySense; } else { partner.flag = mySense; while (flag != mySense) {}; }}} Art of Multiprocessor Programming partner

84 Tournament Barrier void await(boolean mySense) { if (top) { Inform partner return; } else if (parent != null) { while (flag != mySense) {}; parent.await(mySense); partner.flag = mySense; } else { partner.flag = mySense; while (flag != mySense) {}; Order is important (why?) }}} Art of Multiprocessor Programming

85 Tournament Barrier void await(boolean mySense) { if (top) { Natural-born return; } else if (parent != null) { while (flag != mySense) {}; parent.await(mySense); partner.flag = mySense; } else { partner.flag = mySense; while (flag != mySense) {}; }}} Art of Multiprocessor Programming

loser 86 Tournament Barrier void await(boolean mySense) { if (top) { Tell return; } else if (parent != null) { while (flag != mySense) {}; parent.await(mySense); partner.flag = mySense; } else { partner.flag = mySense; while (flag != mySense) {}; }}}

Art of Multiprocessor Programming partner Im here 87 Tournament Barrier void await(boolean mySense) { if (top) { Wait for notification return; from partner } else if (parent != null) { while (flag != mySense) {}; parent.await(mySense); partner.flag = mySense; } else {

partner.flag = mySense; while (flag != mySense) {}; }}} Art of Multiprocessor Programming 88 Remarks No need for read-modify-write calls Each thread spins on fixed location Good for bus-based architectures Good for NUMA architectures Art of Multiprocessor Programming 89

Dissemination Barrier At round i Thread A notifies thread A+2i (mod n) Requires log n rounds Art of Multiprocessor Programming 90 Dissemination Barrier +1 +2 Art of Multiprocessor

Programming +4 91 Remarks Elegant Good source of homework problems Not cache-friendly Art of Multiprocessor Programming 92 Ideas So Far Sense-reversing

Reuse without reinitializing Combining tree Like counters, locks Tournament tree Optimized combining tree Dissemination barrier Intellectually Pleasing (matter of taste) Art of Multiprocessor Programming 93 Which is best for Multicore? On a cache coherent multicore chip: perhaps none of the above

Here is another (arguably) better algorithm Art of Multiprocessor Programming 94 Static Tree Barrier One node per thread, statically assigned Art of Multiprocessor Programming 95

Static Tree Barrier Sense-reversing flag Art of Multiprocessor Programming 96 Static Tree Barrier 2 2 0 0

0 Node has count of missing children Art of Multiprocessor Programming 97 Static Tree Barrier 2 Spin until zero 2 0 0

0 Art of Multiprocessor Programming 98 Static Tree Barrier 2 2 1 0 0

My 0 counter is zero, decrement parent Art of Multiprocessor Programming 99 Static Tree Barrier 2 2 1 0 0

Spin on done 0 flag Art of Multiprocessor Programming 100 Static Tree Barrier 1 2 2 1 0

0 0 Art of Multiprocessor Programming 101 Static Tree Barrier 1 2 2 1 0

0 0 Art of Multiprocessor Programming 102 Static Tree Barrier 1 2 2 1 0

0 0 Art of Multiprocessor Programming 103 Static Tree Barrier 1 2 0 1 0

0 0 Art of Multiprocessor Programming 104 Static Tree Barrier 1 2 0 1 0

0 0 Art of Multiprocessor Programming 105 Static Tree Barrier yowzah! 1 0 0 1

0 0 0 Art of Multiprocessor Programming 106 yowzah! Static Tree Barrier 1 0 0

1 0 0 0 Art of Multiprocessor Programming 107 yowzah! Static Tree Barrier 1 0

0 1 0 0 0 Art of Multiprocessor Programming 108 Static Tree Barrier yes! 1

0 yes! 0 1 0 yes! yes! 0 0 Art of Multiprocessor Programming

yes! 109 Static Tree Barrier 1 2 2 1 0 0 0

Art of Multiprocessor Programming 110 Remarks Very little cache traffic Minimal space overhead On message-passing architecture Send notification & sense down tree Art of Multiprocessor Programming 111 This work is licensed under a Creative Commons Attribution-ShareAlike 2.5 License.

You are free: to Share to copy, distribute and transmit the work to Remix to adapt the work Under the following conditions: Attribution. You must attribute the work to The Art of Multiprocessor Programming (but not in any way that suggests that the authors endorse you or your use of the work). Share Alike. If you alter, transform, or build upon this work, you may distribute the resulting work only under the same, similar or a compatible license. For any reuse or distribution, you must make clear to others the license terms of this work. The best way to do this is with a link to http://creativecommons.org/licenses/by-sa/3.0/. Any of the above conditions can be waived if you get permission from the copyright holder. Nothing in this license impairs or restricts the author's moral

rights. Art of Multiprocessor Programming 112

Recently Viewed Presentations

  • Writing a Successful Grant Welcome to Proposal Scheduled

    Writing a Successful Grant Welcome to Proposal Scheduled

    Are better able to participate in the program or receive the service. Are better able to measure their progress or achievement towards a goal. Know when achieving a benchmark or milestone. For Presentation 4/14/11 MKJ Types of Outcomes Change over...
  • CS 3204 Operating Systems

    CS 3204 Operating Systems

    CS 3204 Operating Systems Lecture 30 Godmar Back Announcements Project 3 due April 13 Th Apr 6, 7pm, 655 McBryde: attend town-hall meeting regarding planned restructuring of 6th floor undergrad space Kernel-level vs User-level Threads M:N Model Solaris Lightweight Processes...
  • Chapter 2: Dimensions of Research

    Chapter 2: Dimensions of Research

    Arial Default Design CHAPTER 2: DIMENSIONS OF RESEARCH DIMENSIONS OF SOCIAL RESEARCH USE OF RESEARCH BASIC RESEARCH APPLIED RESEARCH PUBLIC SOCIOLOGY Grassroots Public Sociology and Action Research PURPOSES OF A STUDY THE TIME DIMENSION THE TIME DIMENSION, continued DATA COLLECTION...
  • Basic Laboratory Equipment for Biology Class

    Basic Laboratory Equipment for Biology Class

    Basic Laboratory Equipment for Biology Class
  • Practical Work for Learning - Nuffield Foundation

    Practical Work for Learning - Nuffield Foundation

    Practical Work for Learning. ... Organise what you know about the reaction of marble chips with hydrochloric acid into three categories. Practical techniques . ... volume of gas collected, or change in mass. To do. a. Carry out the experiment....
  • Doppler Assessment Management

    Doppler Assessment Management

    Arial Wingdings 2 Calibri MS PGothic Verdana Monotype Sorts Wingdings Berlin Sans FB Demi Times New Roman 8_Flow 5_Flow 9_Flow 10_Flow 11_Flow 12_Flow 6_Flow Slide 1 Examination of the arterial patient Slide 3 Doppler Assessment Slide 5 Other useful tests...
  • Elksa (Nt)

    Elksa (Nt)

    Motion 2 - Write-off of debt by Trinity Lutheran Church Zululand Congregation Motion 3 - Freezing of Interest on debt of DELP The unity that we have in our one Lord Jesus Christ allows us to see diversity as strength...
  • Hope - www-chs.stjohns.k12.fl.us

    Hope - www-chs.stjohns.k12.fl.us

    Healthy Steps for Most Mental Health Concerns . Try to sleep 8-10 hours a night. Try to go to bed around the same time each night . Try to have a healthy diet