Hi,
I have two somewhat similar questions regarding the model structure of the BHM inside PALAMEDES.
First off, in the Palamedes toolbox for PFHB, as far as I understand, when all parameters are set to 'unconstrained', multiple conditions act like seperate models and do not share any information, they only do so when a parameter is constrained (although in that case they are truly fixed across conditions).
So, when subject one structurally has a lower threshold, this will be fitted seperately for each condition.
I was wondering if it is possible/advisible to add within subjects that share this information for a particular subject within Palamedes.
Secondly, I face an issue with two sets of condtions, that are now modelled as a fully crossed design. For the first set of conditions our findings are that the lapse rate stays the same within a subject, while for the other set of conditions there is a change. When constraining the lapse rate, the fit is not adequate for all conditions, but when unconstraining there is too little data to identify a proper lapse rate. Is there a way in Palamedes to deal with such nuisances easily? Or would I need a fully different model structure? Is it for example possible to provide a different prior for each condition?
Thanks in advance!
Model structure of random effects BHM
-
Nick Prins
- Site Admin
- Posts: 30
- Joined: Sun Feb 13, 2022 8:55 pm
Re: Model structure of random effects BHM
Not sure I understand all your questions fully, but let’s see how this goes and if questions remain, let me know.
Indeed, if all parameters are unconstrained, there will be no sharing of information between conditions and you might as well fit all conditions separately. If multiple subjects are included in analysis there will still be sharing of information between subjects within conditions. This is a different kind of information sharing though, values are not constrained to be equal between subjects, but are assumed to have been sampled from the same distribution.
The lapse rate is a much more complex issue than people give it credit for. Based on the information you provide, the following is a potential explanation of what happened (you may have more information that might rule this explanation out). On the one hand you say that the lapse rate differs between conditions, on the other hand you say that if you estimate them independently, there is not sufficient data to result in a good estimate. The issue is that if there is not sufficient data to result in a good estimate, how do you know the true lapse rates are different? It might be that it is just the point estimates that differ between conditions and that this variance is due entirely to sampling error. In the situation described here: https://www.palamedestoolbox.org/parame ... dancy.html, the final lapse rate point estimate has little to do with the actual lapse rate but instead is essentially random. The width of the posterior indicates that, in that example, we have no idea what the lapse rate is. Repeat the experiment described there a few times with identical generating parameters (including identical lapse rates) and you’ll end up with wildly different point estimates for the lapse rate each time. In this case, the wildly different point estimates would, counter-intuitively perhaps, be exactly the reason why one should constrain the lapse rate across conditions. A test here would be to compare the full posterior distributions of the lapse rates across conditions. If there is considerable overlap between the posteriors across conditions, there is not sufficient reason to believe that the true lapse rates differ between conditions and you should constrain lapse rate (assuming that on theoretical grounds it is reasonable to assume that lapse rates do not differ between conditions).
If it seems that there is reason to believe that the true generating lapse rates differ between conditions:
It is possible to constrain parameters to be equal in certain conditions but allow them to differ from the parameter values in other conditions. As an example, let’s say you have four conditions and you wish the lapse rate estimate to be equal in conditions 1 and 2 and to be equal in conditions 3 and 4, but to allow a difference between conditions 1 and 2 on the one hand and conditions 3 and 4 on the other. In other words, you wish to estimate only two lapse rates, one to be used in conditions 1 and 2, the other in conditions 3 and 4. You can use a model matrix for this. Two rows, one for each lapse rate to be estimated, four columns, one for each condition. The coefficients determine how the parameters are to be defined. In this example:
MforLambda = [.5 .5 0 0; 0 0 .5 .5];
In words, the first parameter is the average across conditions 1 and 2 (i.e., .5xlambda_1 + .5xlambda_2), the second is the average across conditions 3 and 4. To instruct Palamedes to use this model matrix for the lapse rates use something like:
pfhb = PAL_PFHB_fitModel(data, ‘l’, MforLambda).
All kinds of things are possible using this strategy. E.g., to constrain lapse rate to be equal in conditions 1 and 2, but allow a different value in condition 3 and yet another value in condition 4, use:
MforLambda = [.5 .5 0 0; 0 0 1 0; 0 0 0 1];
More on constructing model matrices to fit your needs is here: https://www.palamedestoolbox.org/modelmatrix.html
It is also possible to use different priors for different conditions. Let’s say for example that you have four conditions again and you wish to specify different priors for conditions 1 and 2 compared to conditions 3 and 4. First, 'unconstrain' the lapse rates ('constrained' is the default). Then, use the ‘prior’ argument and supply a matrix with as many rows as you have conditions. Each row specifies the parameters to use for the prior. E.g.,:
pfhb = PAL_PFHB_fitModel(data, 'l','unconstrained', ‘prior’, ‘l’, ‘beta’, [.03 100; .03 100; .05 20; .05 20]);
The priors for the lapse rate in conditions 1 and 2 will have mean 0.03 and concentration 100, those for conditions 3 and 4 will have mean 0.05 and concentration 20. This does not constrain the lapse rates across conditions 1 and 2 (or across conditions 3 and 4), they merely have the same priors.
It is also possible to combine both strategies (constraining across conditions and supplying different priors. The estimate a single shared lapse rate for conditions 1 and 2 with prior parameters .03 and 100, and estimate a second, shared lapse rate for conditions 3 and 4 with prior parameters .05 and 20, use something like:
pfhb = PAL_PFHB_fitModel(data, ‘l’, [05 .5 0 0; 0 0 .5 .5], ‘prior’, ‘l’, ‘beta’, [.03 100; .05 20]);
If your analysis includes multiple participants, you should specify priors for the lapse hyperparameters (i.e., 'lmu', and 'lkappa'), instead of 'l'.
I’m not sure I understand what you mean by this: “So, when subject one structurally has a lower threshold, this will be fitted seperately for each condition. I was wondering if it is possible/advisible to add within subjects that share this information for a particular subject within Palamedes.” Is one of your participants an outlier? If so, using a t-distribution instead of a normal distribution for the hyperparameter might help things. The fat tails of a t-distribution allow the parameter estimates to deviate significantly from the cluster of ‘normal’ participants.
Indeed, if all parameters are unconstrained, there will be no sharing of information between conditions and you might as well fit all conditions separately. If multiple subjects are included in analysis there will still be sharing of information between subjects within conditions. This is a different kind of information sharing though, values are not constrained to be equal between subjects, but are assumed to have been sampled from the same distribution.
The lapse rate is a much more complex issue than people give it credit for. Based on the information you provide, the following is a potential explanation of what happened (you may have more information that might rule this explanation out). On the one hand you say that the lapse rate differs between conditions, on the other hand you say that if you estimate them independently, there is not sufficient data to result in a good estimate. The issue is that if there is not sufficient data to result in a good estimate, how do you know the true lapse rates are different? It might be that it is just the point estimates that differ between conditions and that this variance is due entirely to sampling error. In the situation described here: https://www.palamedestoolbox.org/parame ... dancy.html, the final lapse rate point estimate has little to do with the actual lapse rate but instead is essentially random. The width of the posterior indicates that, in that example, we have no idea what the lapse rate is. Repeat the experiment described there a few times with identical generating parameters (including identical lapse rates) and you’ll end up with wildly different point estimates for the lapse rate each time. In this case, the wildly different point estimates would, counter-intuitively perhaps, be exactly the reason why one should constrain the lapse rate across conditions. A test here would be to compare the full posterior distributions of the lapse rates across conditions. If there is considerable overlap between the posteriors across conditions, there is not sufficient reason to believe that the true lapse rates differ between conditions and you should constrain lapse rate (assuming that on theoretical grounds it is reasonable to assume that lapse rates do not differ between conditions).
If it seems that there is reason to believe that the true generating lapse rates differ between conditions:
It is possible to constrain parameters to be equal in certain conditions but allow them to differ from the parameter values in other conditions. As an example, let’s say you have four conditions and you wish the lapse rate estimate to be equal in conditions 1 and 2 and to be equal in conditions 3 and 4, but to allow a difference between conditions 1 and 2 on the one hand and conditions 3 and 4 on the other. In other words, you wish to estimate only two lapse rates, one to be used in conditions 1 and 2, the other in conditions 3 and 4. You can use a model matrix for this. Two rows, one for each lapse rate to be estimated, four columns, one for each condition. The coefficients determine how the parameters are to be defined. In this example:
MforLambda = [.5 .5 0 0; 0 0 .5 .5];
In words, the first parameter is the average across conditions 1 and 2 (i.e., .5xlambda_1 + .5xlambda_2), the second is the average across conditions 3 and 4. To instruct Palamedes to use this model matrix for the lapse rates use something like:
pfhb = PAL_PFHB_fitModel(data, ‘l’, MforLambda).
All kinds of things are possible using this strategy. E.g., to constrain lapse rate to be equal in conditions 1 and 2, but allow a different value in condition 3 and yet another value in condition 4, use:
MforLambda = [.5 .5 0 0; 0 0 1 0; 0 0 0 1];
More on constructing model matrices to fit your needs is here: https://www.palamedestoolbox.org/modelmatrix.html
It is also possible to use different priors for different conditions. Let’s say for example that you have four conditions again and you wish to specify different priors for conditions 1 and 2 compared to conditions 3 and 4. First, 'unconstrain' the lapse rates ('constrained' is the default). Then, use the ‘prior’ argument and supply a matrix with as many rows as you have conditions. Each row specifies the parameters to use for the prior. E.g.,:
pfhb = PAL_PFHB_fitModel(data, 'l','unconstrained', ‘prior’, ‘l’, ‘beta’, [.03 100; .03 100; .05 20; .05 20]);
The priors for the lapse rate in conditions 1 and 2 will have mean 0.03 and concentration 100, those for conditions 3 and 4 will have mean 0.05 and concentration 20. This does not constrain the lapse rates across conditions 1 and 2 (or across conditions 3 and 4), they merely have the same priors.
It is also possible to combine both strategies (constraining across conditions and supplying different priors. The estimate a single shared lapse rate for conditions 1 and 2 with prior parameters .03 and 100, and estimate a second, shared lapse rate for conditions 3 and 4 with prior parameters .05 and 20, use something like:
pfhb = PAL_PFHB_fitModel(data, ‘l’, [05 .5 0 0; 0 0 .5 .5], ‘prior’, ‘l’, ‘beta’, [.03 100; .05 20]);
If your analysis includes multiple participants, you should specify priors for the lapse hyperparameters (i.e., 'lmu', and 'lkappa'), instead of 'l'.
I’m not sure I understand what you mean by this: “So, when subject one structurally has a lower threshold, this will be fitted seperately for each condition. I was wondering if it is possible/advisible to add within subjects that share this information for a particular subject within Palamedes.” Is one of your participants an outlier? If so, using a t-distribution instead of a normal distribution for the hyperparameter might help things. The fat tails of a t-distribution allow the parameter estimates to deviate significantly from the cluster of ‘normal’ participants.
Nick Prins, Administrator
-
Nick Prins
- Site Admin
- Posts: 30
- Joined: Sun Feb 13, 2022 8:55 pm
Re: Model structure of random effects BHM
I added some edits to my original reply to correct an omission. I also realized that one of my suggestions needs to be implemented differently when analysis includes multiple participants. Edits in original reply are indicated with bold font.
Nick Prins, Administrator
Re: Model structure of random effects BHM
Thanks a lot for the thorough explanation. This is exactly what I was looking for!