>>Welcome everyone to the last talk of the day. So, we’re delighted to have Jason Hartline tell us about Dashboard Mechanisms for Online Markets.>>Thank you. So, I want to just let you know. My co-author Alec Johnson is a PhD student at Northwestern, Dennis is a Professor in Economics at UBA, Zohoo Wong is a Computer Scientist who’s at the Shanghai University of Finance and Economics, and Ono Zooter is the Head of Data Science at Booking.com. This is a project that we started while Alec, Dennis, and I were visiting Ono at Booking.com, and then Zohoo spent a quarter visiting Northwestern and was able to prove some theorems we couldn’t prove ourselves, which was awesome. This is a brand new paper and one of the reasons I want to speak about it here is because you guys are really awesome and proving theorems too, and there are some pretty hard theorems that we got stuck on and weren’t able to finish and so maybe some of you guys can pick these up and finish it. Good, so I didn’t get the memo from, you’ve all about not having motivation. So, I apologize to everyone who likes to just start with the equations. I’m gonna have a little bit of motivation, and the greater motivation is, it’d be really excellent if computer science theory can help us understand computation than they’re happening all over the place in the world, not just in digital computers. And the big challenge of computation happens all over the place in the world is to guide that computation, you can’t just tell everything what to do. They’re going to be basically operating for themselves and so, the computation of primitive is local, individual, or strategic computation, and you can sort of guide it by setting up the payoffs but that’s what you have to work with. And you want to make this local, individual, strategic optimization achieve good global outcomes. I think a very important key application areas where I like to call online markets and I take that term very broadly to be places where interactions between individuals are mediated by some market mechanism which chooses the outcomes based on some rules. And for an observation I have which sort of, this is topic area that the other computer scientists have been studying for around 20 years and there’s a lot of excellent work here. But there’s a twist on it that I think is super important that people haven’t been talking about too much which is most of these environments don’t allow what’s called truthful mechanisms where the agents have a very simple strategic choice of just submitting their true preferences in mechanism and everything else magically. So most mechanisms don’t have that property and there isn’t actually a very well-developed theory for mechanisms that don’t have this property. So, that’s sort of the space of this talk. So, I what to talk about start with some basic assumptions of online markets that are prevalent in a lot of them, and you can think online advertising if you want, but actually, almost all of them have this very same property. That is, one property is long lived agents with persistent values. So, if you’re thinking about advertisers in these auctions, their value for clicked doesn’t really change all that frequently compared to how fast the auctions happen. And probably, if you knew what the values the advertisers had, you probably have a pretty good algorithm for assigning who gets, which ads go where. So, given the values of the advertisers which are these v one through vn, you’d have some algorithm which I’ll call x which decides how much clicks an advertiser gets. Okay, and probably this x is monotone so some advertiser has a higher value, you’d give them more clicks. Probably, this environment has a lot of randomization and it’s probably stochastic, so probably the amount of clicks someone get as you increase their value is a random variable that’s going to be increasing with the value. So, I’ll think of the number of clicks is a number between zero and one which is a probability and there’ll be a strictly increasing function. And many of these environments, they’re sort of standard, what I call non-truthful payment rules and I think the most fundamental of these is winner-pays-bid, so you say how much you want to pay per click, and you got to click, you pay that amount. That’s sort of the most standard and this is, I think the most pervasive mechanism there is. Here’s the basic question that we’re going to be talking about in this talk and that is, I’d like to know if there’s a winner-plays-bid mechanism, I’ll call x tilde if it takes bids. That implements the desired algorithm I had in mind on values, for all values you could have in equilibrium. In other words, when people have these values, there are going to be some equilibrium in some implementation I have that I hope gives me the same outcome as x one on the original values. That’s what I would hope. I want to basically get the outcome that I want in equilibrium.>>X is zero or one?>>X is going to be a probability between zero or one for each player.>>What’s a non-truthful payment?>>Like, winner pays bid. If I always charge you your bid when you win, you’re never going to bid your value because then you get zero utility for that. So, you’re always going to shade your value down and give me a non-truthful bid. So that’s my basic setting, I think this kind of scenario applies a lot, and so we’re going to try to see what we can do in this kind of a scenario. I want to give you some context and then sort of a high level what the results are going to be. Some context, looking at winners-pays-bid auctions, is literature on price of anarchy that looks at simple auction rules like highest bid wins, winner-pays-bid. That’s also known as the first-price auction if you have one item, which has been studied a lot. And if the agents are substitutes, for example, if you have some number of items to sell and you could sell that each one day any agent, that agents are substitutes, you end up getting constant price of anarchy bounds, things like E over E minus 1.581, as your price of anarchy bounds. If the agents are compliments, meaning, you can sell it to this couple agents or to this couple agents, but you can’t sell to one of these guys and one of these guys, so the agents sort of compliment each other. Then you end up getting linear bounds on the number of agents in terms of your approximation factor. I actually view all of these as impractical. So, loosing 1.58 fraction of the welfare revenue is not practical, losing a linear amount of the welfare is not practical. So, I want to actually look at getting essentially p-test of our results, I want to get within epsilon of the optimal revenue or welfare. Okay. So, here are some results. So, I’m going to be talking about a general framework for answering this kind of a question. Again, from my previous slide, you have some algorithm that actually you’d love to implement. I’m going to show you how to implement that when you can. And so, those of you who know about the VCG mechanism is a framework kind of like the VCG framework. A bunch of my results will be about Nash implementation, meaning I want to implement the thing in Nash equilibrium, so full of misty equilibrium. And in particular, I’m going to show that for multi-unit environments, I have a bunch of units to sell and I can sell one unit to any agent. Then, for allocation rules look like proportional weight. So think about a single item in environment. If I mapped values to weights and then I assign the winner with probability proportional to their weight, that’ll be proportional weights allocation rule. So, for multi-unit environments and x is proportional weights, you can implement those in Nash equilibrium. And this is a starting point for us, but I want you to think of it as like softmax. So, I want to assign the item to player with the highest value. Instead, I’m going to do a softmax. I’m going to do highest value minus the entropy regular riser and that would be exponential weights. Exponential weights is a special case of proportional weights, so that would be within our result. I think more generally, substitutes like a matroid feasibility constraints that many of you have thought about before, that’s an open question. So, I’d love to see people thinking about that question and you can imagine what would proportional weights mean for matroids. You can come up with reasonable definitions of that and ask whether you can Nash implement that. And then, for compliments against the simplest setting is I have two players, I can serve either or both of them or I can sever this one other player. In that environment, you can’t implement things like proportional weights, which would mean map player one and two’s values to weights if the sum of those weights over the total weight allocated to them, otherwise, you allocate to the third player. So, those you cannot implement.>>So they are 100 gender framework because not that [inaudible].>>Yeah. So, it’s not orthogonal. You’ll see two kinds of implementations. One is Nash equilibrium and the other one is what I call persuadable agents. So, what are my general frameworks? One of the general frameworks is Nash implementation general framework where you can take a social choice rule, and if it looks like proportional weights, you can implement it. If it’s compliments, you can’t. That’s the kind of result you have for that. And then, for what I call persuadable agents, which is a new model that we think is relevant for online markets, you can implement any monotone continuous social choice function, X. So, any algorithm you have, it’s continuous. So a stochastic environment that’ll make it continuous, you can implement it. So, that’s an overview. We’ll see if it’s not clear what that means right now, we’re going in detail into what these mean. I just wanted to say where we’re going beforehand. Nash implementation. So, the basic question of Nash implementation is is there a winner-pays-bid mechanism x-tilde that takes bids and implements the outcome you want, x in Nash equilibrium for all values. In particular, the question is for what x does such an implementation exist? So, let’s have just some definitions before we get into this. So, agents have values, we already said that, v1 to vn. A winner-pays-bid mechanism looks like the following. You give it a bunch of bids from the agents and it’s going to compute for each player that own that players allocation which think of it as a probability that they win or not, so between 0 and 1. And then, a bidder is going to pay when they win. So, if they win with probability xi(b), then they pay their bid when they win, so the expected payment is bi xi(b). And their utility is they basically get their value minus their bid when they win. So, I’m talking about Nash equilibrium. And so, this is a little bit strange because in Nash equilibrium, everything is known. But I want to parameterize actually all of the Nash equilibrium as a function of the valuation profile which everyone knows. So, I’m going to write the Nash as a function of the valuation profile and I’m going to let this beta function be basically the bid function, which takes valuation profiles and says what everyone is going to play for that valuation profile. And so, this function beta, the bid function, is a Nash equilibrium for all values v, if for those values v, the bid for player i is his best bid for his utility function given that other players are playing by the bid function. And then, x-tilde implements x if xv is equal to x-tilde bv. So, I’m going to be talking a lot about proportional weights, and again, as I said, I want you to think single-item environment. So, I can basically the x is always sum to 1. So, think single-item environment for all the talk. So, xi’s sum to 1, and you can choose what those xi’s are. Proportional weight says, I first met values to weights, and then I select winner with probability proportional to the weight. One reason why I like this family is a very general family of mechanism. So, you get to pick the weight function. Is that regularized maximization looks like this? And the theory I’m really talking about is going to require continuous functions. And so, I can’t just use max. But I want to think about maximizing welfare and I can put a regularized one with a very small coefficient and I could get almost exactly the true maximum. Yeah.>>[inaudible] again is your question about taking truthful x and finding x-tilde for them is about nothing truthful allocations to one of these data allocations? [inaudible] v is that x of v.>>So, x-tilde implements x. So, x is what I’m given. I have an algorithm. If I knew the values, I would want to run that algorithm.>>Okay.>>Okay? Now, I don’t know the values and I have a mechanism that’s going to get bids.>>Sure.
>>So x-tilde is my mechanism to gets bids, and I would love it for it to be the case that when people found Nash equilibria, for all values where they found the Nash equilibria, then I ran that x-tilde mechanism, I just get the thing I wanted.>>Yeah, I understood x is not really good.>>So, in your definition of implements, do you mean that exists in x, exists in beta?>>I’m going to mean for all, okay? So, I want uniqueness of this Nash equilibrium, which I didn’t write here, but yeah.>>Just to make sure that I understand, in the definition of beta, the implements to the beta minus i of v rather than b minus i.>>Yes. Absolutely. I’m sorry about that. Yes. So, V is a function. Data is a function that requires all of the values to know it. So, absolutely, that should be a V. Thank you. So, the main idea that is used in both parts of this talk is that I want to combine the inference that comes from econometrics in economics, which is working at bids and figuring out what people were thinking when they gave you those bids. So, economic inference. With trying to implement some outcome, and I want to just walk through a simple example which is not you what I want to do, but it’ll give you an idea for how you could possibly do this. Okay. So, here’s an example. Two agents and I want to think about just instead of starting with x, let’s just start from some x tilde, some bit allocation rule. And my bit allocation will just proportional bids. Okay, so what does that mean? So, player one should win with probability b1 over b1 plus b2 for two players. So, just proportional bids. There’s actually a standard record list studied in the lurcher. Okay. And so I want to point out that if you give me equilibrium bids, that each bidders first-order condition looking at their optimization problem is going to tell you what their value was. Okay. So, I can like reverse engineer what players were thinking, what their value was when they submit these bids. Let’s just walk through how that works. Okay, so Agent 1 with value v1 chooses b1 to be this argmax, right. Well, this is the probability that they would win with any bid b, given b2 is player two is bidding b2. Okay. And notice that when we’re doing after the fact analysis, like we’re doing right now, we know what b2 is because player two bid b2, so we see it. Okay. So, equilibrium must satisfy the first-order condition which I’ve taken the thing and differentiated it, right. And I’ve actually done that differentiation for you and figured out what it is and it’s this function. Okay. So, you tell me what the bids were, hey, they are one and two. Well, now, I can tell you using this function that v1 is five-halves. I can also tell you what b2 is I didn’t bother to write that down. So after the fact, I know what the values are when I run this mechanism. So here’s the thing, well now, I’m sad because I wanted to allocate efficiently, instead I ran this b1 over this mechanism which didn’t allocate efficiently. We sometimes allocated to the player with the lower value, right. But I know the values and so now I’m sad why don’t I just give it to the guy whose value is v1.>>Is sort of Bayes Nash equilibrium so.>>No, Nash. Not Bayes Nash. Nash.>>So, how this one bidder when you may know I’m mean if there’s some other bids to be Nash?>>That’s what a Nash Equilibrium is. So remember, weren’t full information to players both know their values.>>But how do they commit the bids or there’s an order that they.>>I guess you [inaudible].>>No, we’re just talking about Nash equilibrium. And when we’re talking about Nash equilibrium, we never talk about how we found the Nash equilibrium, we just talk about what the Nash equilibrium is. Okay. So, there’s a whole other literature about how to find it, which actually I will address in the next part of the talk. But for now, let’s just assume we’re there. Okay, good. So, in hindsight, our mechanism know the values exactly. Well, what I really want is someone uses this idea to well if I know the values in hindsight, why didn’t I just do a good thing in the first place? So, that’s what I want to do. So, this was a little bit complicated how I solve for bids, et cetera, so I want to make it much simpler on the next slide. So, this is a standard theorem about equilibrium in the single-dimensional environments. It usually it’s written for Bayes Nash equilibrium or dominant strategy equilibrium. It also applies to Nash exactly the way I’ve written it here, it’s pretty much the same. Okay, so a mechanism x tilde and bid function B is in, it should be Nash equilibrium not Bayes Nash, sorry I cut and paste from slides. And I’m going to just let xv here equal the combination. Okay, if this x is monotone in each coordinate, so xi is monotone and coordinate ivi, and the payment identity which is the payment to the player satisfies this formula. And usually, this additive term at the end is zero and I will just want to draw geometrically what this payment and he says. So, here’s some increasing xi of vi. This first term is this area vi times x of v, right. This integral is the area underneath the curve. So obviously, the payment is this area above the curve. The difference between those two, that’s a geometry that’s what it is. Good. So, the thing I’m going to use about this to basically this lets me solve for equilibrium in winner pays bids auctions very succinctly. Because I have two equations for payments. The payment identity gives you one equation for payment in terms of the allocation rule that I want. And the winners pays big getting another equation for payments, payments have to be bid times allocation. You pay your bid when you win. So, I can use that to solve for what the bids have to be. So, if you tell me the allocation rule x, I’m just going to tell you, well, the bids have to be, integrate the thing, do this equation that function divided by the probability that the player wins. So, if you told me now the x you wanted, I know exactly what bid function has can be the only bid function that implements that x. I already know what the equilibrium has to be in a winner pays bid auction. So here’s the idea, let’s just set x of b, x tilde of b equal to x of b inverse of beta inverse of b. Just do that. Two issues, this bid function might not be one-to-one, there might be multiple inverses. Another issue is, when I defined a game, the action spaces need to be a product space. So everyone can bid in some range, they can bid in some range. And the image of beta might not be a product space. So, the inverse might have points I can’t invert because they are uneven space. Turns out this is actually a pretty easy problem to fix, problem two, not a big deal. You can basically find close enough points to get something reasonable for everything, but no one is going to bid this anyway, and equilibrium is always going to bid a real point. So, it just need some consistent way of making sure people don’t bid those weird points, okay? You can see the paper for that. This inverse is the main thing that I want to talk about. And this is where some serious math is needed, and even more serious math is needed is identifying when these functions are one-to-one. And let’s just go back a slide to look at the function we’re talking about. So, beta is payment which is this function, divided by xi. So, we’re talking about win is for what x is mapping from RM to 0,1 to the n. Is this crazy function one-to-one? Okay. So, here’s a cool proposition actually if there exists a winner-pays-bid mechanism for x if and only if beta is one-to-one. It might be pretty obvious to some why that’s true. Good. Here’s a cool theorem. Beta is one-to-one if the Jacobian of beta is a positive definite. I’m not going to go into that, it’s a famous theorem in this space. So, we look at proportional weights for multi-unit environments. So again, take single-item, take a portion of weights. So, each player has a monotone function mapping values to weights. I’m going to get these weights and I give it to player i with probability wi, over the sum of the other players weights. So, the Jacobian is positive definite and you can compute the inverse efficiently. Which means I can implement this thing I want to implement for any proportional weight of such a choice function. Okay, cool. Compliments, beta is not one-to-one. So, I can apply the backwards instruction, only if I say I can’t implement these. And that’s because basically, I can’t choose where two sets two values to map to the same bid, I see only the bids, I can’t know which ones the values are. And if I choose the wrong one, that wasn’t equilibrium, so it’s not the right thing. Cool. Coming back to your last question. I want to change gear, so that was my Nash equilibrium stuff. Hopefully you can see what the math questions are, I’m going to come back to them at the open questions at the end of the talk. Here are some challenges. One challenge, agents have to find this crazy Nash equilibrium, how can we possibly find that? And another challenge is, I can’t actually implement general allocation rules x, I can only do special ones. I can’t do compliments so that sucks. I might like to implement far more things that my Nash implementation allow me to do. Here is an idea that we rip right out of what actually happens in the industry in online markets. You give the bidders a dashboard that helps them understand what the kind of factuals are. Meaning, what would happen if they put in different bids and what the bids they currently are putting in. So, you as the platform, come up with these estimates that help bidders know what the bid landscape looks like, and all they have to do is read off that thing and figure out what is the best bid their utility function.>>How truthful are the dashboards?>>We’ll talk about that. Great, good point. So, I’m going to call agents persuadable if they will follow the dashboard, if following the dashboard converges the best response. Meaning, the dashboard can be wrong, maybe initially when we doesn’t know too much, but it better eventually be right. So you can’t get people to just, or even in BS, you have to actually have to be a good strategy to follow the dashboard. But if you’re wrong initially, when you don’t know stuff, maybe the agents are happy thinking that you know more than they do and it’s better than they would have copy on their own.>>So the stuff with my dashboard depend on who’re interacting with your dashboard?>>Now, these are individual dashboards. But they might be based on history where you learned something. So, here’s a dashboard mechanism, it is given by two things. One is the dashboard, which is a per agent bid-allocation rule. And the other is the bid allocation rule which is mapping a profile of bids to an outcome. So, these are different mathematical objects which I tried to make sure it was clear here. The dashboard takes each player and tells you from their bid what’s the probability of winning, and the bid-allocation rule maps a profile of bids to who wins. And here is my historical values dashboard mechanism which I think is one that makes the most sense. Publish a bidding dashboard with the bid-allocation rule from historical estimates of values. So, I’m assuming you as the principal have been historical estimate of values. Why am I assuming that? Well, in step three, you estimate values. So, maybe you ran as before and have estimates from before. I want to solicit bids, and I’m going to assume they’re best response base to my dashboard. Well now, I just know what the values are. So now I just plug those values directly into my original algorithm that I want to implement, and I run my algorithm on the true values, or these estimated values. And I charge the winners their bids as usual in a winner-pays-bid mechanism. And these results are just like super straightforward. If even monotone, continuous algorithm, then, in this mechanism, following the dashboard converges to the best response in two rounds. The first round, you have no idea what’s going on because you don’t have any estimates. And the second around, you know the true values and so you post the true, the y tilde is actually the true allocation rule for the true values in the next round, and now we’re basically in Nash. So, in two rounds we’re in Nash. And so, I say any monotone, continuous algorithm can be implemented for persuadable agents by dashboard mechanism. So, this is just a really good idea. All my markets should publish a dashboard, should assume agents best respond to them, and should then know the true information, optimized for the true information. But then, publish the dashboard that corresponds to the true information. So, you don’t try to fool the agents, you give them the best information you have which is what you are trying to output.>>[inaudible] have better information than the dashboard, he would deviate.>>That’s correct. That’s a great question to study still. I don’t know what would.>>Destroy the correctness of dashboard in the second round.>>If they don’t believe the dashboard, then whatever. So, this is one of the reasons why, so I actually presented this talk in the opposite order, that we can see this stuff. We think this is the right mechanism but this is not a Nash. So, when is this the same as Nash? Well, when x is proportional weights, and you can just do the Nash inflation before and have this sort of a feedback loop in there, okay? And so, I think it’s pretty robust for settings like proportional weights with a single item or multi-units, if not, going to be so robust like that for complements, okay? I want to talk about some connections to literature which I think are important here. So, one connection to literature is implementation theory which is this general question of what kinds of algorithms x can we implement in equilibrium? Okay, this is a great survey by Jackson to look up that. So basically, this literature characterizes x where there is a unique equilibrium. Essentially, the calculation is any monotone x is implementable, like my dashboard result. However, the mechanisms that they come out as literature are crazy impractical mechanisms that you would never want to ever use, whereas this dashboard is a realistic looking thing, okay? And in particular, there are sequential mechanisms where you interact a couple rounds. In one of the rounds, there’s cross reporting, so you ask the agents, like agent one to report what they think agent two’s value is. And so in full formation, they know. And then they use some kind of shooting the agents if they disagree, like if agent one and agent three disagree on agent two’s value, you just blow up the world, right? So, that’s what these mechanisms do in implementation theory. And so, our take on that is to get rid of these kind of ridiculous mechanisms, you want to restrict the space you look for. So like these winner pays bids mechanisms where you get bids, you can compute any outcome you want, but winners pay their bids. What can you implement with those simple kinds of rules? And there, we gave a characterization of you need the Jacobian to bit function and you need to understand what that means for the x and we gave a sort of partial answer for that for proportional weights. The other thing I think is interacting information design. Otherwise known as Bayesian persuasion and this is an awesome area that’s deeper, and economics are especially cited about. Basically, the agent has a prior and they need to make a decision. The prior states the world making a decision to optimize their utility. There’s an informed principal who knows more than the agent does but wants the agent to make a different possible decision, right? So the principle can give the agent a signal which then the agent updates their prior will make a different decision than you would have before. So how can a principle convince an agent to do what the principle wants and not what the agent wants? Okay, good. So our suggestion here is we have something kind of like persuasion. We give a dashboard, agents do what they want based on the dashboard, but we have a more relaxed model which allows the principle to be wrong at first, and learn over time, just be having more information than the agent does and basically, you can’t persuade using something that’s not true, eventually. But as long as the principle’s right, eventually, you can persuade the agent. So I think it’s a more amenable to learning, amenable to CS ideas model that’s in the information design space. So I think that might be interesting much more broadly. Cool.>>Could the principle pay the agents to follow these?>>Not in that literature. Now, that literature is the only thing you have is you can give them a signal which is correlated with the true statement.>>Okay. But in your model which you’re free to change, you want to put in side payment.>>In my model, there are already payments which are you pay your bid. Now, if you have side payments willing to screw that and have like a second-price rule. Be careful how you ask the question, okay? And again, coming back to the first comment of implantation theory, one of the things that we got, we got more meaningful mechanisms by restricting the space of mechanisms to something that’s sort of natural. So, I’m nervous about letting it go all over the place. Okay, so again, my context, I just want to come back to what I said my results were now that we have seen what the details are. So, I’d like to have a much more meaningful characterization of what’s implementable besides just saying proportional weights which seemed like a broad family but it’s not everything in that space. I think that we should be able to implement such a choice rules like in matroid environments that say are smooth versions. So a softmax for like matroid environments. I think that’s an open question. So, we study proportional weights with compliments and I’m not saying there isn’t something there that you can do for compliments but I don’t know what it is, okay? All right, thanks for your attention.>>So, what is your dashboard mechanism during the first round that has no information?>>Yes, so I’ll give you two answers. It doesn’t matter, any monotone function gives the result that I say you convert into rounds, but I think a much more relevant answer is there is no first round in online mechanisms. It’s not like you turned Google search ads on on day one and then that was it, right? So imagine instead, you’re running a mechanism and some new business shows up, but you actually could probably estimate for whatever bid they put in, what probability they would win because you actually have all the data from whatever bid, what happened, what they would do. So when a new bidder shows up in an on-running mechanism, you could give them the right dashboard on day one for them, actually.>>There’s no large market assumption in your work.>>No.>>Is booking.com doing something you forget?>>I have no idea.>>It really did tell you the same thing.>>Yeah?>>So for different auction mechanisms, your bit function would change and your header function would change, but do the conditions for mutation stay the same?>>Okay, so I was talking about implementation with winner pays bid rules, right? You can imagine implementation with like GSP, or you can imagine implementation with all pay bid rules, okay? So, one thing I actually don’t like about our theorem is they are very specific to winner pays bid. Whereas, I’m sure that the same thing holds for all pay. But our analysis wasn’t the right analysis that was flexible enough. And in fact, I’d like to have a reduction I’d like to have a theorem that says proportional values works and if some function works, then putting in any monotone weights instead of values works also, right? I’d like to have a composition theorem about that instead and I think that would actually be a good step to getting things like all pay for free from an analysis of first price. For things like GSP, that’s hard to talk about because doing the revenue equivalence calculation of what payment of characterization art, you can’t do that. That characterization I did on the first slide where I talked about winner pays bid auctions, you can’t do that for GSP. But one thing you can do is if you look at our dashboard mechanisms, what I know about the Google dashboard for ad auctions is they just give you a cost-per-click curve, right? They say you paid this much, you got this many clicks. That’s actually a winner pays bid information. So even though the auction is GSP, they give you winner pays bid information and then they just say, well, what bid is it that gets that cost-per-click and then you can put that bid in, and that’ll be your GSP bid, right? So, if you publish your dashboard as winner pays bid and then there’s a language that maps those bids into bids that you would actually put in, that’s then sufficient to implement these dashboard mechanisms, and so it doesn’t really matter.>>Thanks.