Jump to content

Auroral Activity App


Guest badrobit

Recommended Posts

Guest badrobit

Hey SpaceWeather, 

 

I have been using your site for the longest time and I love the service you offer. I was wondering if you know of any applications for the phone which offer some of the same data but optimized for your display and take advantage of knowing your GPS location and weather for that spot to give better predictions for aurora hunting? 

 

If not I am looking for a side project and would be very interested in making one and was wondering if you would be willing to share some of the backend of your system for the app? If you would I would be more than happy to do it under the space weather live name (you would get final approval of the app of course). 

 

Let me know if you have any thoughts. 

Link to comment
Share on other sites

Hi Badrobit,

 

We're glad that you like our website! Currently we do not have an app of our own but our website is optimized for the mobile web making it easy on smartphones to get the latest information on their phones together with our automated twitter live alerts. 

Automating it into an app with usage of GPS location could improve predictions for the given location of the viewer, but still then you would need to base yourself from magnetometer data which is the most reliable and that should be the magnetometer station closest to the point of the users phone. When you only take the kp in count, than you'll miss a lot of activity.

So how do see such app?

Link to comment
Share on other sites

Guest badrobit

Thanks for taking the time to reply! 

 

This is my current through process for the App: 

 

Use Kp index and NOAA auroral oval predictions along with the local weather (cloudiness, humidity, moon phase) to give you a more precise and personal warning system that can notify you and say hey there is a pretty good chance (say >50% or user defined value) that you might see aurora activity in your area. 

 

If the user then wants to actually figure out if they can see aurora or when this might happen they would then tap into more data and processing heavy tools to determine when / where might be the ideal case to see them. I am aware of the basics for determining this but I am not 100% sure of the best way to do this. My guess would be to take the users position and then something in a visible area (say 50-100km north of the user to provide an estimate) and get data that is relevant for those areas (again not sure what data is best but I am sure you can help with that part). That is at your position you would have a 50% chance of seeing them but 100km North you would have a 60% chance (I doubt it would increase this quickly but this is just for arguments sake). 

 

You could also tap into the darksky information to perhaps say that if you have the time there is a spot 20 minutes away which might greatly increase you chance of seeing them (myself being near a bigger city would kill for this feature). 

 

I would also LOVE to be able to tap into the SpaceWeatherLive community for a sort of community alerts function. Where if you have the app installed and you are looking at aurora now you simple press a button and maybe take a picture and it goes to a server which can (if the user chooses to use this feature; due to the server requirements probably a premium one to cover server costs) will let you know that someone near by (as defined by the user) is seeing aurora and you might want to take a peek outside :P These could also feed into your twitter alert system to say hey anybody in the <insert general area here> area one of our members is seeing aurora look out side now. (the reason why i suggest requiring photos is to avoid people abusing this system to spam members and the community at large)

 

These are all just rough ideas but if you like how it sounds I would really love to work with you and your community on this

Link to comment
Share on other sites

The geographic location is one thing, but you'll have to convert it also to the magnetic longitude and latitude so that you know where the user is magnetically speaking. Next step would be calculating solar noon point so you know at which point the oval is stretched at its most southern point. These steps would improve a location based forecast cause a user at the same latitude in Europe won't have the same chances as a user in America because of the magnetic latitude difference.

Cloudcover is a good thing, but where is a reliable source for the whole globe, i've seen such cloud cover predictions in the past but as far as i know there is no good one where you can easily process data from all over the world. 

Use of ovation prime is not a good one, nice images indeed but when it comes to comparing the ovation prime model with the real oval (observations on the ground) it's chances are often off the chart or rather too low calculated. It does take into account the energy in the atmosphere and predicted K but still lacks implementation of magnetometer data to really improve it all.

 

maybe i sound a bit too pessimistic but if we could develop such thing, it should be detailed ;)

Link to comment
Share on other sites

Guest badrobit

Hey Vancanneyt, 

 

You don't sound pessimistic they are all good points and ones that if it were to be looked at would need to be addressed. 

 

I agree having the geographic location is good for much in this case beyond being able to put a user on a map or some other form of display. I am sure there are calculations out there that would allow us to cover geographic (gps) location into magnetic location. I imagine something like the approached mentioned here are a good start  :)

 

I like the idea of cloudcover but I am also not sure of a reliable source for this information world wide. I imagine that it would have to be done on a country (possibly region) based level based on where the user currently is (we wouldn't want them pulling in more data then the absolutely require). 

 

I agree that ovation would not be sufficient for if you are seriously attempting to see but I think it would provide a useful part of an easy quick glance at if it is worth it to start pulling in all of the required data in order to get a fairly certain guess. Cell phones are pretty okay at processing the information but we wouldn't want to waste users data when they are only taking a quick peek to see if it is really worth it to try and look for them tonight. 

 

I would love to have it broken down into something that is a beginning screen which pulls in the most basic information to give a very rough estimate. Once that value hits a predefined chance of seeing them the user would switch to another part of the app which would provide a much more detailed and much more accurate analysis with all of the information points presented in a pretty way. Like you I would also love to be able to tap on these items to see a bit of their history (for trends) and possible predictions based on these trends. 

 

I might take a shot at doing up a very quick prototype of the back-end of the system so that I can test out if I provide it with a GPS location if it can spout out a guess that makes sense or lines up with your websites predictions at least on the high middle low altitude split. Any help you might be able to provide with how to combine these sources (I want to start with ovation and kp) together into a very rough estimate. (again I know these are not the best indicators but I just want to test my idea and see if it has legs and how much data I require to make these estimations). 

 

All the best, 

Mat

Link to comment
Share on other sites

The link you provided with the mag2geo en geo2mag is indeed a good one, i've already broke my head on that in the past ;-) as the matter of fact we had a side project in development to code such thing you're trying to do in the correct way, but we had too many bumps along the road. I'm not a scientist of education and my math level is not at that level of scientists, i even had our code checked by a student in maths and astronomy to get the bumps out of the road but still we haven't managed to fill the holes. Maybe i wanna do it too precise, i don't know.

In the current way we do it now is with the K-indices but only with the K-indeces it wouldn't be sufficient. We do take other parameters into account but still, for a location based service it would need more. With the Kp you get a global view if it would be a good time to go out, but if you wanne plan it hours before that wouldn't work either. You could check daily predicted Kp but still then like yesterday we didn't have a G1 what was predicted. 

 

I sure love the idea and it is our goal eventually too, maybe we must try it and see where it leads us and take a few bumps along. Who knows...

Link to comment
Share on other sites

The Norwegian weather service provides all kind of data in all kinds of formats but since I am not an expert when it comes to programming I have no idea if it is of use. Just know it exists and might be of help: http://om.yr.no/verdata/free-weather-data/

Note that most of those pages are on Norwegian unfortunately. I also do not know how accurate it is but it is better than nothing.

Link to comment
Share on other sites

Guest badrobit

Marcel thank you so much for the link. At the moment I am plugging away with the weather data location problem. I am currently building a system which uses OpenWeatherMap information in order to determine the weather for any particular location (based on gps coordinates). At the moment I am just making sure I can fetch all the data and the next step would be to cut down the amount of information that I am asking for to only what I want. Here is my current thoughts on what would be useful please feel free to comment and if there is something you think would work better as a data source or that you would want to see added if you were to use the app once it is out in the wild. 

 

  • Current Weather
    • Cloud Cover (can't see much if its too cloudy (of course this depends on the height of the cloud cover)).
    • Temperature (More for your comfort to decide if its too cold to wander out to find them).
    • Moon Phase (if the moon is too bright it hampers your ability to see the night sky). 
  • Short Term Weather
    • Basic forecast for the next 3-4 hours including all the above listed items so you can look at if the weather will improve, degrade or stay the same).
    • This will NOT be used in the chance of seeing aurora calculation it will only be there for user information.

Please everyone let me know what you think any and all feedback will be helpful when trying to plot out what will be useful for everyone  :)

Link to comment
Share on other sites

Openweathermap indeed has a good API, but you must aknowledge that if it gets to an app that many users will download you'll need the paid openweathermap API. The free account only limits you to 3000 calls/minute and data refreshes only every two hours (so it's not accurate for fast forecast)

What should also include:

- Moon Ephemeris, rise and set (not only the phase)

- Humidity/Fog

Link to comment
Share on other sites

The Norwegian weather service provides all kind of data in all kinds of formats but since I am not an expert when it comes to programming I have no idea if it is of use. Just know it exists and might be of help: http://om.yr.no/verdata/free-weather-data/

Note that most of those pages are on Norwegian unfortunately. I also do not know how accurate it is but it is better than nothing.

The Norwegian weather service also has a xml data service.

On the website, they also have cloudcover, but unfortunately, it is not included in the xml.

Example: http://www.yr.no/place/Netherlands/Utrecht/De_Bilt/forecast.xml

I also know a webservice for sun and moon set and rise.

Example: http://hetverkeer.info/weer/json-liveweer.php?lat=52.1017&long=5.1783

This one also seems to work for outside netherlands, but with less info

Example: http://hetverkeer.info/weer/jsonsm.php.php?lat=69.66&long=18.99

I also have developed my own app (which is not yet released in any way), for which I have my own xml data collection.

A sample of this data can be seen here: http://s000.tinyupload.com/?file_id=81475267438718327273

The data mainly comes from the swpc.

Link to comment
Share on other sites

Guest badrobit

Thank you everyone for the updates  :)

 

@ Vancanneyt

 

Yes I am aware of this. Sadly I would have to watch the number of calls required but so far any global (and even most local) solutions for weather information require paid API access once you start getting into large amounts of requests. Sadly as much as I would want to make the app free I believe if it were ever to get to the point where I am able to deploy it that it would be a free app which provides most of the features and anything requiring a paid API service would require a yearly or monthly subscription to help pay for those APIs :wacko: ). My goal of course is to make as much of the app free as possible but this also depends on how much of the data it requires would be open source to begin with. 

 

@ Waldo 

 

Thanks for passing that along! I have looked a bit into where to get the space information from. At the moment I am pulling it in the form of JSON calls to the SWPC/NOAA experimental service. Using this feature versus grabbing their latest XML file off their server means that as a phone user I can avoid having to request the whole file and simply get the data I need when I need it. This being in their experimental section however god only knows if it will see the light of day. If it disappears I imagine I will implement a server which pulls the XML information and allows me to remote query it myself to ensure that I keep data costs low.

 

Why I am worried about file size:

For example the XML file you posts was 16.6kb I know it doesn't sound huge but if you are requesting one of these every 5 minutes while you are aurora hunting and you are out for 5 hours that adds up to nearly 760kb. Again not a huge number on a computer but this is probably only a very small fraction of the overall data required. If I can query for the information and get it returned in a smaller format you would be talking bits instead of kilobits which at the very least means if the app goes up for people to use it will be very low on the list when they are trying to figure out which app has killed their monthly data allotment  :)

 

Thank you both for taking the time to comment I am really looking forward to having something to show off maybe even some crappy screenshots in the next week or so  :D I am currently looking at developing it in a cross-platform from the get-go so that no one gets left out  :)

Link to comment
Share on other sites

  • 3 weeks later...
Guest badrobit

Just wanted to let people know that I am still working on the first prototype. It has just been delayed a bit due to work commitments but has not been forgotten :)

Link to comment
Share on other sites

Guest Vortice

Hi. I'd love to view this app finished, as I'm trying to catch the aurora in a low latitude, or travelling to a higher latitude if I have enough time.

May I suggest that you focus on the problem of predicting the phenomenon (ie translating to geomag coords, finding the appropriate station and related useful data, and making a prediction), before dealing with the problem of making a weather forecast for the point of observation. While that would be very useful, and certainly should be on a finished app, it's of no use if you lack a reliable aurora prediction. Besides, that feature can be replaced by the user (local weather service for example) while it's still lacking, though a reliable aurora prediction cannot.

I strongly agree with the idea of minimizing data transmission. The user might be in a bad signal area, or in an underdeveloped area of the network, so a few kb may make a difference. Maybe not in Europe, but they certainly do here in Argentina even in areas with strong signal and good coverage. You never know where the user will be.

That said, I'd suggest you that you make the most straightforward and simple approach you can think of, so you get a working app fast, and *then* you try to minimize data transmission. There's always time to improve the app. Premature optimization is the root of all evil :-)

Please tell me if you guys think I can be of any help. I have a good background in math and am a decent programmer (but I have no experience in android apps).

Regards

Link to comment
Share on other sites

Guest badrobit

Hi Vortice, 

 

Thanks so much for taking the time to respond, I would hope that someday the app can help you find some aurora  :)

 

I agree with you in that predicting the aurora is probably the most critical and difficult part, I have been collecting where I can get data from I am just not sure how to use it yet. It is something I plan on working on once I get the initial weather + kp index guess part done. 

 

I am not sure how to combine all this information into a prediction so any help anyone would like to provide would be VERY appreciated.

 

I agree weather may seem like a bit of an odd place to start so I will try to explain why I chose this and so far it has been an excellent place to start. 

 

The reason why I wanted to start here is because there are LOTS of data sources available and this allows me to build up the backend for communicating. In doing this I discovered the following: 

  • I want to talk to the outside world using JSON.
    • I will use this for ALL data so I am only using weather data now because it is easy to understand and get reliably. 
    • Communication is expensive (data wise)

This leads me to why optimising now for data is SO important. I recently was able to see my first aurora (YAY  :D) the one thing i discovered. I was somewhere with an Edge level connection and barely able to get any data through. I feel if I wait too long to look at keeping this low I will never get around to it. (my favourite star gazing app used almost 30mb last night, this is something I really want to avoid). 

 

It turns out for the aurora data I can get ALL of it at once but I cannot make specialized queries. This means I will need some server sitting somewhere that can provide special geography based aurora data packages. I will again be doing all this through JSON but it turns out it is something else I need to keep in mind before getting too far into everything. 

 

As for helping I think the biggest spot where I could use help is how to combine all the data sources into a reliable weighted observation prediction. 

 

If you are curious as to the frameworks in use I am developing all the libraries in C# and then building front ends for each mobile OS using xamarin which has been a wonderful tool to use thus far. I am hoping to have the weather + kp demo working by this weekend so keep an eye out for it :D 

 

(also I am calling it aurora-hunting in the project so I probably also need a catchier name than that  :P )

 

Mat

Link to comment
Share on other sites

Guest Vortice

Hi again, sorry about the delay.

I think that making a reliable aurora prediction is a difficult task. In fact, I think it's a research topic. I'd try to get a preliminary working app, even if it's far from being as accurate as it should be, by using ovation data as a prediction. Vancanneyt Sander said that local magnetometer data should be used, too. Maybe you can make the experiment of mixing the two data and see what happens. What's nice about an app is that you can get confirmation from users.

Or better, we can get in touch with the man in charge of the ovation model (pm me about that). Maybe we can ask him to improve the model, and he can benefit from field confirmation by the app users. Please be aware that I'm speculating. I don't know him, I just happen to have his e-mail. But it's worth a try.

About the aurora data, I'm pretty sure that a server to split it in geographical locations is affordable through donations. Or, buying the app allows the user to use the server to minimize data transmission, the free version gets the full data (I really prefer the donations, but it's an option).

About the name, how about Aurora watch?

Oh, and don't forget to program a night mode, minimum brightness and red over black.

Regards

Link to comment
Share on other sites

I think the aurora-nowcast-map is the source you are looking for. Source: http://services.swpc.noaa.gov/text/aurora-nowcast-map.txt

Is is the text version of the ovation aurora prediction.

It contains values in range from 0 (little or no probability of visible aurora) to 100 (high probability of visible aurora).

with 1096 values covering 0 to 360 degrees in the horizontal (longitude) direction

and 512 values covering -90 to 90 degrees in the vertical (latitude) direction.

You then have ovation probability of every location on earth.

 

This source is 2M of data and is refreshed every 5 minutes. That is too much for use directly in your app,

but if store this data regularly on a server, you can send the coordinates from your app to a server script and find the probability based on the coordinate.

That only requires sending coordinates and retrieving one percentage value.

Link to comment
Share on other sites

Correct Waldo. The only thing I hate about the ovation model is that it isn't correct in many ways for lower latitudes. It often gives zero percentage for latitudes that are actually seeing the aurora. For example in the G4 storm of this month many places that have seen the aurora where not or had barely any percentage in the ovation output.

Plus for aurora chasers in middle latitudes they also go out for photographic aurora so it's even worse for that.

That's the main reason that magnetometer data are the most valuable data around.

Link to comment
Share on other sites

Guest Vortice

So, what about getting the local magnetometer data and using it to increase or decrease the probability given by the ovation model? It would be just a multiplication by some educated-guessed factor, which can be fed back with statistics taken from user confirmation that they could/could not see the aurora.

That factor could be increased to give a probability of photographic aurora.

It wouldn't have a theoretical base, but it may work.

Link to comment
Share on other sites

I have looked again at a source for cloud cover data.

Openweather seems to have one.

Example for fairbanks, alaska: http://api.openweathermap.org/data/2.5/weather?lat=64.8377&lon=-147.716

 

About the ovation / magnetometer issue:

I agree that magnetometer data is almost always the best indicator, but a nearby source is not always available, and depending on which country you're in, there are a lot of different sources you need to monitor. 

 

And sometimes you just need to go outside, instead of looking at data. A good example is 17 march. A kp=7 warning was enough for me to decide to go outside to my known dark spot close to my hometown. Unfortunately, I didn't see any aurora, while others from all over the Netherlands did.

The closest magnetometer source is in Denmark (brorfelde and romo). I only looked at the data when I was home again (and I have my own app...)

All data sources would have told me that there should be aurora (Kp=8, Hpi=10, Magneto active etc.), but no visible aurora for me.

Link to comment
Share on other sites

Guest badrobit

Wow thank you all for the awesome replies! I have been taking it all in and I am looking at the best way to try and combine all of this information. In more exciting news there is a bit of a start to the app. I am hoping to use this post as a bit of a dev blog so here it goes. 

 

Server Side

  • Created a few python scripts which can fetch data for me and pack it up based on latitude and longitude requests. This allows me to cut data like the ovation model from mb to bytes maybe kb at most. 
  • At the moment all I am providing is ovation data but this will not be my only source of information. 

App Side:

  • It is headless (no working UI). 
  • At the moment I am able to communicate relatively reliably with my data sources to obtain and store the following:
    • Weather Data
    • Ovation Model Data for my lat,long
    • Some light pollution data (still working on a reliable data source for this). 
  • Now that I can get most of the information I am looking for I will be sitting down to figure how to come up with the number that represents what the odds of seeing aurora are based on your location. 

 

I know that this isn't the most interesting information in the world so I took the last couple of days to mock up my thoughts on what the app itself might look at. I found a really cool tool called proto.io and have a very bare bones UI mock up. You are able to interact with it like it is a real app. If you want they even have an app that you can install to try it out on your device. You should be able to add comments anywhere on any of the pages so please feel free to comment on as much as you want; I can use all the feedback I can get. You can find the mockup here: https://badrobit.proto.io/share/?id=4d4f3333-f2f7-4692-86c7-f73f0bdd19b3&v=1

 

Please also feel free to leave comments in this thread I am keeping an eye on it :D I hope you all have a wonderful long Easter weekend! 

 

Mat

Link to comment
Share on other sites

Guest Vortice

Very interesting :-) lots of data, easy to use. Night mode doesn't work in the simulation. Light pollution is great data. % of moon visible is missing.

Maybe the kp data up to a month before would be useful, too. I remember reading that nice auroras tend to repeat like 27 days later because that's the period of rotation of the sun.

What's the yellow, thin line in the kp data?

Oh, sorry, I forgot to tell you that a nice feature would be to generate a notification to warn the user when kp index is above some thereshold, which I would set by default to 5. That value should be in the settings.

Maybe another menu option could be added, with predictions for the next days. Weather, kp index, etc.

Link to comment
Share on other sites

For the Kp, you have two sources:

- The Kp data in 3-hour periods of the last month (http://services.swpc.noaa.gov/text/daily-geomagnetic-indices.txt)

- The Kp warnings and alerts. They are real-time (http://services.swpc.noaa.gov/products/alerts.json) Look for the codes WARK04 until WARK07 depending on the level you want to be warned for. They have a valid from and to time. This source also has ALTK04 until ALTK09 alerts whenever the Kp reaches level 4 until 9 (real-time).

- For real-time Kp above level 4, you can also look at the Noaa Scales (http://services.swpc.noaa.gov/products/noaa-scales.json). Here, the G-scale from 1 to 5 correspondents with Kp 5 to 9.

 

If anyone knows a good real-time source for K-index < 5, I'm happy to hear it. The swpc refuses to give their source.

Link to comment
Share on other sites

@Vortice: Kp alerts can already be set on our Twitter profile, using the Twitter app you can set notifications for our account and receive immediately alerts including the Kp

 

@badrobit:

For light pollution maps, maybe you can digg out http://www.lightpollutionmap.infowhere they had their sources

for the server side it's maybe better to put it in one big API caller, would ease up the whole process of calling things. Also look at server side caching of data, database structures etcetera. 

For viewing chances also take in account the nominal data measured by ACE. Also keep in mind the difference in magnetic and geographic latitudes if u tend to use other methods than ovation; also keep an eye on the solar noon point for furthest extension of the oval etcetera.

For the GUI there are plenty of frameworks to easily build a mobile app (rigged into some in the past but never got any too work well :D have to study more objective C i guess :D)

Link to comment
Share on other sites

Archived

This topic is now archived and is closed to further replies.

×
×
  • Create New...

Important Information

We have placed cookies on your device to help make this website better. You can adjust your cookie settings, otherwise we'll assume you're okay to continue. By using this site, you also agree to our Terms of Use and our Privacy Policy.