Here are two questions important for experimental science classes such as this one:
Why do experiments? What role do experiments play in making science certain?
In what ways can one use mathematical models to analyze results of measurements?
Question one we will come back to next week. I just want that question to fester in your minds. This lab is about question two, and about the 'Fitteia', the tool we'll use for the computational physics part of PHYS 272L. The following set of enumerated items provide a guided exploration of the tool, using a very simple data set. The goal of this exploration is to acquire familiarity and skill with 'Fitteia'.
Although there is a rubric for our labs specifying in some detail what is necessary to score well, all you have to do in this lab to score a 16/16 is to do everything each item requests.
*Receive* the required lab notebook. (Sp25; your student accounts have already paid for them)
Write a heading in your lab notebook, something like 'Fitteia Exercises', but exercise free will.
Go to Fitteia.org, and REGISTER!!! Get (create) your login credentials and all that. I think way back when I used the `-a' server....
Figure 1. Web page (landing page) for Fitteia
Watch the short instructional youtube video found in the GETTING STARTED menu. Now you know how to do everything, right?
No, right? Download and read section 1.3 of the document 'ejp485119suppdata.pdf' Ref[1] at bottom of this page!). It is about the The fitter module. Make notes in your lab notebook about things you'll use a lot but will forget next time you login. For instance, how to enter a formula in the language 'C' (native to, coding used for input to Fitteia) that produces the formulas '\(x^y\)' and '\(\sqrt{x}\)'? Write that down. Also record in a terse manner for easy recall, what the 3 main sections of the fitter module are. We'll refer to these sections directly throughout the semester. Also, you'll have to read closely to understand exactly what the fitter module is.
Login
Click on the Tutorial1 directory. Now you are looking at the web page for the fitter module. There are lots of data sets (files) in the subdirectories that have useful examples from which to learn (also copy). It's a little weird though how to get to them.
Click on the drop-down menu on the far right of the web page header (see below)in the upper most section, in the so-called web page header, and choose one of the files named 'Basic'. Choose 'Basic' and then click on Recover, and and all the sections will be populated with example data. This whole exercise will involve playing with this data. This is test data for tutorial practice, and there are analogous things in Basic 2, 3, 4, and so on, but for this exercise, we'll just work with the un-numbered file, 'Basic'.
Figure 2. Web page header for Fitteia, a little scrunched up
Record, in your lab notebook, a data table of your own making the data found in the buffer. There are only 4 data points so it won't be hard: make column headings for the \(x, \: y, \:\) and \(\Delta y\), creating for yourself a meaningful table number, caption, meaningful heading titles, and units for each of these headings. At this point in the exercise, you are simply being asked to make them up: you'll have to force yourself to do this sort of thing at first, but it will become second nature for the work we do this semester. Not the `making them up' part, but the forcing yourself to the think about the actually units of the things that go into the table entries. I am afraid I can give you no guidance about what to put down, but put down something. Use free will and your best lights to figure out what these data points might symbolize. Have fun with it - use whimsy and imagination. If you need a footnote to explain a word or a unit or something, by all means use a footnote. Read the caption of Fig. 3 and imagine what units of the other columns were for that data table. Call this Table 1.
Hit 'fit' (right next to 'plot'). A linear fitting curve (this is our computational model of the physics model) appears, or should appear. The linear fitting equation, or default computational model in this case, is of the form \begin{equation} y = a + b*x, \end{equation} where the constants $a$, and $b$ are 'fitting parameters'. Fitteia is written so that the goodness of the fit is automatically assessed for us. Paraphrasing from Sebastião's paper that explains his inventiaon of Fitteia [2], the analysis of experimental results with respect to a physics model requires the fitting of that model's
equation (it has to be in C, this is our computational model) to experimental results.
It's understood that the computational model will have parameters that the Fitteia will systematically vary,
obtaining the values of each parameter for which the ''goodness of fit'' value is a minimum.
No programing is required on your part, you just have to furnish an equation (the computational model) used for the modeling. What does Fitteia calculate as a proxy for the goodness of the fit? It is the chi2 or the $\chi^2$ statistic, defined in [2] (Eq. 12), as follows,
\begin{equation}
\chi^2(p_1, . . . , p_m) = \quad \sum^N_{k=1} \frac{\left( y^e_k - y^t_k(p_1, . . . , p_m, x^e_k)\right)^2}{\left(\sigma^e_k\right)^2},
\end{equation}
where $y^t_k(p_1, . . . , p_m, x^e_k)$ is the value of the model equation calculated for the experimental value of the independent variable $x^e_k$, the `$m$' model parameters are $(p_1, p_2, ... p_m)$, the actual $N$ measured data points are $(x^e_k, y^e_k \pm \sigma^e_k)$ (for the $k_{th}$ measurement), and the uncertainty of each measurement of the $k_{th}$ measured value $y^e_k$ is uncertain by $\pm \sigma^e_k$. This is a lot to take in. It's kinda like a 'minimization of least squares' but, crucially, includes the experimenter's estimate of experimental uncertainty for each data point.
NOTE: the 'Plot Settings' section (\(2^0\)), controls what this plot looks like! At any rate, you should see a linear fit of the data appear in the plot, with the \(\chi^2\) statistic calculated under the heading 'Fitting results'. This fitting results from a pre-populated function (we are calling this the computational model) in 'Function and Parameters' section (\(3^0\)). These results, the plot of the data, with error bars, the model fitting curve on top of that for visual comparison, the \(\chi^2\) statistic giving a measure for of the goodness of the fit: this is what we use Fitteia for. This is what we are after, this permits you, the modeler, to make an informed statement, and evidence-based statement, and a scientific statement, about how well model fits the data. This is what we will try to do for each lab. Note too that the reduced chi2 statistic is related to the chi2 statistic as follows, \begin{equation}
\tilde{\chi}^2 = \frac{\chi^2}{N-M}, \end{equation}
where $N$ is the number of data points being fitted, and $M$ is the number of fitting parameters (also called `degrees of freedom'). A good fit is one in which the reduced chi2 statistic is close to 1 (not 0).
We will now try two different computational models in search of the computational model that works the best. This is us playing around with Fitteia to see what it can do. To help us compare all the computational models,
Create a second table that help us examine the goodness of the fit for the linear model that was prepackaged so to speak, and two nonlinear models to be described presently (in items 12 and 13). Choose as column headings, 'Computational model', '\(\chi^2\)', '\(\tilde{\chi}^2\)', 'comments regarding goodness of fit'. Use words to evaluate the goodness of the fit. Is it a good fit or a bad fit? Also, and perhaps most importantly, add a fifth column, 'Physical Model'. Well, this most important thing is not included with the Basic data set. Where did it come from? It's just four not so random points. But I really don't know where the data came from. So let's make it up. You make it! Really! Think about some actual physical system that might behave linearly or behave what ever the computational model seems to suggest, define what the model is, what the $x$, data is, $y$ data, and so forth, and most importantly, what the fitting parameters are. We want to grasp, ultimately, what sort of experiment we are doing that would generate such data. Please consult the lab instructor (who would really like to help and talk through all of this with you). To return to Column 1; what goes in Column 1? The computational model, being the equation that is found 3rd. section of the body of the fitter module, the 'Function and Parameters' section (\(3^0\)). Should you write it down in your table as it is written there is C programming language, or should you write it mathematically? At length this will be just two languages for the same thing. At first it's a little weird. But use the C code here. Put a very brief answer in Column 3. Before you do, surf over the home page again, find the 'Support' menu, and click on FAQ. You want the 2nd one, it's really clear and direct [3]. An example of the top row of a previous student's submission (which I thought was pretty good, it's not about neatness really but about whether everything required is there....
Figure 3. I thought this was well done, although I would've liked all the physical model information to be in just one column....A well labeled table can show off the results of very complex experiments at a glance!
Now, let's listen to the data. Surely the data cries out, 'I'm nonlinear!' Edit the Function buffer there in section \(3^0\), and try a square root fitting, and include an off set! Perform a square root fitting (recall, it must be written in 'C'). Make it the functional equivalent of \begin{equation}
y = a + b * \sqrt{x},\end{equation}
where \(a\), and \(b\) are real constants. And then,
Obtain results, evaluate, and fill in next line of Table 2. Note that this includes a verbatim statement of the computational model, etc.
If instead you used, \(y = a + \sqrt{b*x} \), would the value of \(b\) be the same as above, the former case? Would its units? Record your ruminations directly in your lab notebook. It is your diary, record, external brain, your interlocutor, this lab notebook. You are talking to your other self, or maybe to the next student who will do this work, with only your lab notebook as a guide. Anyway, Use the results of your work to complete a second line in your second table.
Continuing, maybe the square root thing is not as good as it gets. Let's now try a transcendental function. Edit the computational model again to perform the fit, \begin{equation}
y = a\sin{(bx)} \end{equation}
where the real constants \(a, \: b \) are fitting parameters. Obtain results, evaluate, and fill in the final line of Table 2.
Also, but only for this case, click on 'PDF report' (you'll find it underneath the plot, which is above section \(1^0\). Save and rename the output with a meaningful name with your initials at the end, preceded by an underscore, (e.g., weirdfitting_gds.pdf). Make a record of this in your lab notebook (filename, what's in it, etc.)
Ok, one last test. What happens to the goodness of the fit if we play with the magnitude of the experimental uncertainty? Let's go back to the square root fit, and let's try it with, well, 5 different values of \(\Delta y\) (yes, it's actually called 'ey' in the data buffer, meaning 'experimental error (read: uncertainty) in y'. You choose, some smaller, some bigger. Again, use your free will and try stuff. Look at the plots you get for each one, and then
start a new Table (Table 3, the last one...) with the same headings as table 2, but, add a column for \(\Delta y\). You know what? Just do this for ONE of the computational models, not all of them. You pick. Take one of the rows from Table 2 and let that be your beginning for Table 3, adding a new column for and \(\Delta y\). Systematically vary the size of the error bars in the data set, and study how $\chi^2$ and $\tilde{\chi}^2$ vary. Obtain a new result, trying a bunch of values bigger, and a bunch of values smaller than the value that came with the Basic data set.
Make a new data set from Table 3 which plots \(\tilde{\chi}^2\) vs. size of error bar. Is there any uncertainty in \(\tilde{\chi}^2 \)? YES! However we won't pursue that in this lab.
Make a plot of \( \tilde{\chi}^2 vs. \Delta y \). The hard part here is finding out how to produce Greek characters in the axes labels, or, you can totally wimp out and just use English -- however, you will need to indicate units. For help, look for 'fonts' in section 2 of the fitter module, right next to '\(2^0 Plot \: Parameters:\)'.
How do you know the data you have plotted here is any good? This won't seem much like an 'intermediate check of results' or an application of a `predict-measure-compare' cycle that we will be talking more about during the semester. But, write down in your lab notebook what you think would be a reasonable shape for this particular plot. It doesn't have to be detailed. Just answer this question: would \(\tilde{\chi}^2\) be an increasing or decreasing function of the magnitude of the error bars? Write down your thinking. Discuss it with your lab partner. Then, after that, have a look at the plot and discuss whether it make sense or not.
Figure 3. A plot just to exhibit Greek characters in the axes labels. Eerbloots are entirely made up, your units will conform to the physical models you describe in table 2.
write an evaluation that discusses: a) which model best fit the data?, b) examples of actual physical models taken from PHYS 270 and PHYS 271 that could possibly correspond to each of these models, and finally, c) details about how one can define the computational model in terms of the natural variables used in the physical model or theory. Make this (c) like a 'How-To', so that someone following your procedure can get your results. This item (c) will be a required and essential feature for each use of Fitteia in this laboratory course.
upload page images of your lab notebook entries (comments and knowledge capture stuff, responses to questions, etc.) and plots, all bundled into a SINGLE PDF FILE on CANVAS.
Pedro J Sebastião, ''The art of model fitting to experimental results,'' (2014) Eur. J. Phys. 35 015017
Along with the number of fitting parameters (M) and number of data points (N) enable us to estimate the reduced \(\chi^2\), that is, \(\tilde{\chi}^2 = \chi^2/(N-M)\).
We'll quote it here: What is the meaning of \( \chi^2\)?
It is calculated according to Eq. 15.1.5 of the book Numerical Recipes in C, Second Edition (1992) and according to the CERN library MINUIT minimization routine applied to the \( \chi^2\). For a good fit the value of chi2 should be roughly equal to the number of data points, while a good fit for the reduced chi2 statistic is indicated by a value close to 1. [my emphasis] For a moderate good fit the value can be around the number of degrees of freedom (N_exp_points-M_fitting_parameters). The value of chi2 depends very much on the experimental error values of your dependent variable.
