Creating a dimensionless index

toitu · 2010-12-16 12:36:49

Hi bobbym
Instrument 1 is responding more to the 50% and a lower 90% but then at around Time 5, there is an increase in the 90% followed by a decline in Instrument 1 and less of a decline in Instrument 2. To me that would suggest Instrument 2 is more sensitive to the 90%. As for post-time step 15, see comment #27.

A tricky problem to try and index I guess! Apologies for taking up a lot of your time today.

bobbym · 2010-12-16 12:40:57

No problem.

It is not over yet. Actually we are just starting.

One question, why do you want to use the geometric mean?

toitu · 2010-12-16 12:49:43

Mainly as to my understanding it's quite useful for summarizing highly skewed data, which I suspect this is.

bobbym · 2010-12-16 13:02:45

Would it be acceptable if I try to fit a relationship between the reading on instrument one and 2 to the 90%.

And then fit another to the 50% and you could analyze later what the 2 meant together?

Or are the 50% and 90% related? If so how?

toitu · 2010-12-16 13:09:47

By all means bobbym! Unfortunately though the 50 and 90% are related in that they describe the 50th and 90th percentile of the particle size range for each sample (I also have the 10th percentile but that does not have much effect, physically speaking). I have done some plots looking at the relationship between individual instruments and the 50 and 90% and they are pretty scattered - the issue here is that the millivolt output is not just responding to the 50 and 90th but everything! However, the 90th will have more of an influence given the optics and wavelength of each instrument. That probably doesn't help much!

bobbym · 2010-12-16 13:19:42

Hi toitu;

There may not be enough of an effect between the instruments and the 90%. I will not know until I start to try to fit something. Do you have any data in terms of average sizes
rather than a percentile?

toitu · 2010-12-16 13:30:09

Unfortuntaley I only have the percentile values so I guess the 50th percentile represent the median particle size in microns rather than the mean.

bobbym · 2010-12-16 13:35:17

Hi;

That is unfortunate, the median contains less of the character of a sequence or sample than the mean does. That may be the reason for the wild fluctuations. The mean might have smoothed things out by a lot.

bobbym · 2010-12-16 14:56:33

Fitting half the problem now. Responding better than I thought it would. Now the question is how close is acceptable to you?

toitu · 2010-12-16 15:38:22

Hi Bobbym
If the fit is reasonable, I'll be pretty happy - I'm not looking for an exact fit as not possible with this type of data but if we're in the vicinity of 50 to 60% of the variance explained, that works for me!

bobbym · 2010-12-16 15:41:46

Right now I am struggling with the residuals they are a little too large. That particular value might be an anomaly, or error in measurement. I will not know until I try more fits.

It is going to be a while. Going to take a little break to eat.

toitu · 2010-12-16 15:46:00

There are indeed a couple of large departures in the dataset that are a little difficult to decide if they are 'real' or not. Thanks for all your efforts on this, it is much appreciated.

bobbym · 2010-12-16 20:02:28

Hi toitu;

Sorry, I could not get back to you sooner, but my internet went out for 2 1/2 hours. The largest error I have now is around 35. My predicted value is off by 35 from one of your data values. The rest are close I am still trying to reduce these residuals as they are called down. Please hold on it will take time.

toitu · 2010-12-17 03:28:02

Hi bobbym
that sounds pretty good!

bobbym · 2010-12-17 04:11:33

I was able to fit a relationship between all 4 parameters. It is large but not too difficult to work with.

i1 is the first meter reading.
i2 is the second meter reading.
f is the fifty percentile data.
n is the ninety percentile data.

The equation is:

Where:

This has a maximum error of 15.45 for any particular point. Most are a lot less. It is fairly obvious from the residuals that a better fit is still possible.

Please allow me some time to check out the values. Lots of copying can result in errors.

toitu · 2010-12-17 04:27:07

That looks fantastic bobby, thanks so much. When you are happy with the fit/residuals, would
you be able to explain to me what you did to get to this point? Thanks

bobbym · 2010-12-17 04:48:08

Hi;

Please give me lots of time to check the figures. There are undoubtedly typos.

would
you be able to explain to me what you did to get to this point?

It is totally an industrial strength solution. Explaining it is harder than doing it. Lots of computing and lots of guesswork. There is one piece of math in here that is unique.

Long time ago in galaxy far, far away, I happened to discover a method of curve fitting that achieved the much desired minimax fit. It is fairly common for continuous data but I never saw anything like my method for discrete points. I went around for a long time using it and hiding my discovery. I named it ABput and used it to compress an entire poker simulation into several formulas. You can imagine my sorrow when I finally found out that a great numerical analyst named Remez had not only done it first but did it better.

You will be happy to know I used it on your problem.

toitu · 2010-12-17 04:50:59

Sounds quite interesting (and complex!). I guess I have some reading ahead of me!

bobbym · 2010-12-17 05:01:41

Hi toitu;

Will be taking a little break now to rest my eyes. I will resume work on it later. Then we can talk about trying to use effectively, alse seeing whether it has any predictive power...

toitu · 2010-12-17 05:08:13

Hi bobbym, sounds great! Thanks again

bobbym · 2010-12-18 00:18:31

Hi toitu;

Have run into some problems, a certain piece of software is doing the calculation of the residuals in an unusual way. It is probably not wrong but just not what I expected it to do. This has slowed the work down. A whole day and a half has been wasted. Now I must start again. I am sorry.

Some new relationships have been found.

Where :

f is the fifty percentile data
i1 is instrument or meter 1
i2 is instrument or meter 2

This relation has a maximum error or 4.54.

toitu · 2010-12-18 06:32:41

Hi bobbym, no need to apologize at all! That's a pain the software is not responding properly. However, residuals of 4.54 um is impressive!

bobbym · 2010-12-18 10:47:22

Okay, to the next problem. You now have two relationships, one between all the variables and one between the 2 meters and the 50 pecentile data.

To get a relationship between just the 2 meters and the 90 percentile data seems obvious now but the obvious things are always wrong.

toitu · 2010-12-18 13:14:57

Hi bobbym, I agree that using only the 2 instruments to predict the 90 will be difficult. Especially as under normal circumstances we wouldn't be able to 'temper' the instrument with the 50 percentile data as usually we wouldn't know this. I will look foward to hearing how you get on with this last step. Thank you so much for your help, you have gone way beyond expectations on the amount of time you've spent on this.

bobbym · 2010-12-18 14:51:02

I have not tried the obvious move yet but it might be possible to smooth the data or even eliminate one bad data point.

Math Is Fun Forum

#26 2010-12-16 12:36:49

Re: Creating a dimensionless index

#27 2010-12-16 12:40:57

Re: Creating a dimensionless index

#28 2010-12-16 12:49:43

Re: Creating a dimensionless index

#29 2010-12-16 13:02:45

Re: Creating a dimensionless index

#30 2010-12-16 13:09:47

Re: Creating a dimensionless index

#31 2010-12-16 13:19:42

Re: Creating a dimensionless index

#32 2010-12-16 13:30:09

Re: Creating a dimensionless index

#33 2010-12-16 13:35:17

Re: Creating a dimensionless index

#34 2010-12-16 14:56:33

Re: Creating a dimensionless index

#35 2010-12-16 15:38:22

Re: Creating a dimensionless index

#36 2010-12-16 15:41:46

Re: Creating a dimensionless index

#37 2010-12-16 15:46:00

Re: Creating a dimensionless index

#38 2010-12-16 20:02:28

Re: Creating a dimensionless index

#39 2010-12-17 03:28:02

Re: Creating a dimensionless index

#40 2010-12-17 04:11:33

Re: Creating a dimensionless index

#41 2010-12-17 04:27:07

Re: Creating a dimensionless index

#42 2010-12-17 04:48:08

Re: Creating a dimensionless index

#43 2010-12-17 04:50:59

Re: Creating a dimensionless index

#44 2010-12-17 05:01:41

Re: Creating a dimensionless index

#45 2010-12-17 05:08:13

Re: Creating a dimensionless index

#46 2010-12-18 00:18:31

Re: Creating a dimensionless index

#47 2010-12-18 06:32:41

Re: Creating a dimensionless index

#48 2010-12-18 10:47:22

Re: Creating a dimensionless index

#49 2010-12-18 13:14:57

Re: Creating a dimensionless index

#50 2010-12-18 14:51:02

Re: Creating a dimensionless index

Board footer